[SPARK-23254][ML] Add user guide entry and example for DataFrame multivariate summary

[WeichenXu123]

What changes were proposed in this pull request?

Add user guide and scala/java/python examples for ml.stat.Summarizer

How was this patch tested?

Doc generated snapshot:

[WeichenXu123]

@MLnick @MrBago Thanks!

[SparkQA]

Test build #86856 has finished for PR 20446 at commit 307f75f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• public class JavaSummarizerExample

[MLnick]

Few minor comments

[MLnick]

Perhaps "contain" -> "are" or "include"?

[MLnick]

Perhaps "The following example demonstrates using Summarizer(...) to compute the mean and variance for the input dataframe, with and without a weight column"?

[MLnick]

Why not just df.select(...).show()?

[WeichenXu123]

Because spark user will usually want to get the summary result (multiple vectors), I want to show the simple way to extract these results from the returned dataframe which contains only one row. I think some user is possible to get stuck here.

[MLnick]

ok fair enough

[MLnick]

Why not just df.select(...).show()?

[MLnick]

Same applies here, why not just df.select(...).show()?

[MLnick]

Same applies here, why not just df.select(...).show()?

[MLnick]

nit, but Tuple1 not required here?

[WeichenXu123]

Do you mean us .as[((Vector, Vector))] ? It compile fails..
or Do you mean change to

val (meanVal, varianceVal) = df.select(metrics("mean", "variance")
      .summary($"features", $"weight"))
      .as[(Vector, Vector)].first()

? Seems also do not work because it is a "struct type" value in the returned row. So the first row format should match Row(Row(mean, variance))

[MLnick]

oh ok - perhaps select("summary.mean", "summary.variance") would work to extract into two columns?

[WeichenXu123]

Good idea. This way make code easier to read.
Done.

[SparkQA]

Test build #86968 has finished for PR 20446 at commit 2592bb9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[MLnick]

sorry, one more comment here

I think perhaps "... to compute the mean and variance for a vector column of the input dataframe ..."

(and same below)

[MLnick]

A couple minor nit-picking comments, otherwise LGTM

[SparkQA]

Test build #86975 has finished for PR 20446 at commit fc9622b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #86980 has finished for PR 20446 at commit f02172f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #89569 has finished for PR 20446 at commit f9eb02a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #89570 has finished for PR 20446 at commit ee9d368.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[WeichenXu123]

@MLnick @srowen

[SparkQA]

Test build #92539 has finished for PR 20446 at commit ee9d368.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

@WeichenXu123 looks like there was one more outstanding comment, about using .show()?

[WeichenXu123]

@srowen The reason I do not use .show I have already reply here #20446 (comment)
thanks!

[srowen]

I see, you replied on the first instance of that comment.

[srowen]

Merged to master