[SPARK-3417] -Use new-style classes in PySpark

[mrocklin]

Tiny PR making SQLContext a new-style class. This allows various type logic to work more effectively

In [1]: import pyspark

In [2]: pyspark.sql.SQLContext.mro()
Out[2]: [pyspark.sql.SQLContext, object]

[JoshRosen]

Good catch! While you're at it, are there any other old-style classes in PySpark that should be made into new-style ones?

[JoshRosen]

Also, do you mind opening a JIRA ticket on https://issues.apache.org/jira/browse/SPARK and editing the title of your pull request to reference it, e.g. [SPARK-xxxx] Use new-style classes in PySpark?

[mrocklin]

Sure. Next time I find a few free minutes.

On Fri, Sep 5, 2014 at 8:04 AM, Josh Rosen [email protected] wrote:

Also, do you mind opening a JIRA ticket on
https://issues.apache.org/jira/browse/SPARK and editing the title of your
pull request to reference it, e.g. [SPARK-xxxx] Use new-style classes in
PySpark?

—
Reply to this email directly or view it on GitHub
#2288 (comment).

[mrocklin]

mrocklin@notebook:~/workspace/spark$ git grep "^class \w*:"
mrocklin@notebook:~/workspace/spark$ 

[mrocklin]

Done

[davies]

The title is a bit confusing, would you mind to change it to "Use new-style classes in PySpark" ?

The patch looks good to me, thanks!

[mrocklin]

done

[SparkQA]

Can one of the admins verify this patch?

[JoshRosen]

Jenkins, this is ok to test.

[davies]

Jenkins, test this please.

[JoshRosen]

Jenkins, this is ok to test. Jenkins, retest this please.

[SparkQA]

QA tests have started for PR 2288 at commit 4aadab6.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2288 at commit 4aadab6.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class RandomRDDs(object):
  • class MLUtils(object):
  • class SQLContext(object):
  • class StorageLevel(object):

[JoshRosen]

Jenkins, retest this please.

(want to see how long it takes Jenkins to respond to me...)

[nchammas]

He's been pretty slow to respond lately. 10-15 min I'd say.

[marmbrus]

Looks like this only failed unrelated streaming tests (Exception encountered when attempting to run a suite with class name: org.apache.spark.streaming.kafka.KafkaStreamSuite) /cc @tdas.

I'm going to merge this to master.