[SPARK-3964] [MLlib] [PySpark] add Hypothesis test Python API

[davies]

pyspark.mllib.stat.StatisticschiSqTest(observed, expected=None)
    :: Experimental ::

    If `observed` is Vector, conduct Pearson's chi-squared goodness
    of fit test of the observed data against the expected distribution,
    or againt the uniform distribution (by default), with each category
    having an expected frequency of `1 / len(observed)`.
    (Note: `observed` cannot contain negative values)

    If `observed` is matrix, conduct Pearson's independence test on the
    input contingency matrix, which cannot contain negative entries or
    columns or rows that sum up to 0.

    If `observed` is an RDD of LabeledPoint, conduct Pearson's independence
    test for every feature against the label across the input RDD.
    For each feature, the (feature, label) pairs are converted into a
    contingency matrix for which the chi-squared statistic is computed.
    All label and feature values must be categorical.

    :param observed: it could be a vector containing the observed categorical
                     counts/relative frequencies, or the contingency matrix
                     (containing either counts or relative frequencies),
                     or an RDD of LabeledPoint containing the labeled dataset
                     with categorical features. Real-valued features will be
                     treated as categorical for each distinct value.
    :param expected: Vector containing the expected categorical counts/relative
                     frequencies. `expected` is rescaled if the `expected` sum
                     differs from the `observed` sum.
    :return: ChiSquaredTest object containing the test statistic, degrees
             of freedom, p-value, the method used, and the null hypothesis.

[davies]

cc @mengxr

[SparkQA]

Test build #22882 has started for PR 3091 at commit 5097d54.

• This patch merges cleanly.

[SparkQA]

Test build #22882 has finished for PR 3091 at commit 5097d54.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class Matrices(object):
  • class ChiSqTestResult(JavaModelWrapper):

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22882/
Test FAILed.

[SparkQA]

Test build #22886 has started for PR 3091 at commit 0ab0764.

• This patch merges cleanly.

[SparkQA]

Test build #22886 has finished for PR 3091 at commit 0ab0764.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class Matrices(object):
  • class ChiSqTestResult(JavaModelWrapper):

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22886/
Test PASSed.

[mengxr]

What happens if r is JavaArray or JavaList but not pickleable? Are we expecting that downstream can handle it?

[davies]

The caller will handle it. The JavaArray/JavaList is iterable in Python, caller can access the internal objects in this array/list.

[SparkQA]

Test build #22913 has started for PR 3091 at commit 145d16c.

• This patch merges cleanly.

[SparkQA]

Test build #22913 has finished for PR 3091 at commit 145d16c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class Matrices(object):
  • class ChiSqTestResult(JavaModelWrapper):

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22913/
Test PASSed.

[mengxr]

LGTM. Merged into master and branch-1.2. Thanks @davies !