[SPARK-7245][SQL] Add Spearman correlation support for DataFrames

[viirya]

JIRA: https://issues.apache.org/jira/browse/SPARK-7245

Besides, CovarianceCounter.merge has a bug that sometimes other.count could be zero. It makes the results incorrect (NaN). This pr fixes it and a minor typo in CovarianceCounter too.

[SparkQA]

Test build #31829 has finished for PR 5902 at commit 32f511d.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #31844 has finished for PR 5902 at commit 92b4ab9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

@viirya can you submit a separate PR for the bug fix? The bug fix should go into 1.4, and the rest can go into 1.5.

[viirya]

ok.

[viirya]

Submitted the bug fixing of CovarianceCounter.merge to #5931.

[srowen]

@viirya you still have ~14 open PRs, some of which have been out there for months. This is an example of one that doesn't seem to have traction; can you close these out including the JIRA?

[rxin]

@viirya we should close this one for now, and re-open it later when we do this as an aggregate function.

[viirya]

Sure. No problem.

[viirya]

@srowen The JIRA is not open by me, is it ok to close it by me?

[rxin]

I think he meant closing the github pull request for now - not the JIRA ticket. Thanks!

[srowen]

Yes @viirya thanks for the PR; I really mean we should only keep PRs open while they're 'active' and close them if they're not obviously moving forward. They can come back later if needed. I just noticed this morning you still had a number that had been inactive for over a month.

In this case, I think the JIRA could be closed too, though it's more debatable. You don't need to close the JIRA if in doubt. One Of These Days we'll go through and start auto-closing long-inactive JIRAs too.

[viirya]

@srowen Thanks for explaining that. I already closed some other PRs together.