SPARK-11741 Process doctests using TextTestRunner/XMLTestRunner

[gliptak]

SPARK-11741 Process doctests using TextTestRunner/XMLTestRunner

Doctests were moved under a class to correct processing

[gliptak]

#8323
WIP
Please review. Please note that results end up as target/test-reports/TEST-doctest.DocTestCase*.xml

[SparkQA]

Test build #45910 has finished for PR 9710 at commit 4c45290.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * public class JavaMultilayerPerceptronClassifierExample\n * public class JavaGradientBoostingClassificationExample\n * public class JavaGradientBoostingRegressionExample\n * public class JavaRandomForestClassificationExample\n * public class JavaRandomForestRegressionExample\n * >>> class VectorAccumulatorParam(AccumulatorParam):\n * case class AppendColumns[T, U](\n

[gliptak]

Working to improve tooling at xmlrunner/unittest-xml-reporting#100

[gliptak]

@davies @mengxr @JoshRosen Please review. Thanks

[gliptak]

@JoshRosen Please review.

[JoshRosen]

Same comment as on your other PR: can you please write a PR description for this which explains the change and summarizes what you did?

Also, what's the status of the xmlrunner changes? Is there a certain minimum supported version that we should be using in our CI environment? If so, I can help to make sure that it's installed.

[gliptak]

xmlrunner/unittest-xml-reporting#100 was committed but not yet released. The version 1.12.0 installed with https://issues.apache.org/jira/browse/SPARK-10830 works, we could update to 1.14.0 (when released).

[JoshRosen]

Will the code here need to update after a new version of that library is released or will we only have to bump in Jenkins? If we won't have to do any code changes, then I think we should go ahead and modify the other files to also use this code. Since the few lines of test runner configuration might be duplicated in a bunch of files, we could refactor them into a separate function which we call in each file.

[gliptak]

I'm only foreseeing a library version bump.

[JoshRosen]

Okay, sounds good. Do you want to go ahead and apply this change to the other PySpark files?

[SparkQA]

Test build #49432 has finished for PR 9710 at commit d8c8883.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • >>> class VectorAccumulatorParam(AccumulatorParam):

[SparkQA]

Test build #49512 has finished for PR 9710 at commit 781a309.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • >>> class VectorAccumulatorParam(AccumulatorParam):

[gliptak]

@mengxr Locally I see a number of these doctests fail. Should failing ones be commented and JIRAs opened against them? Thanks

[gliptak]

This patch adds the following public classes (experimental):

    >>> class VectorAccumulatorParam(AccumulatorParam):

What is the recommended way to tag this class private?

[SparkQA]

Test build #49536 has finished for PR 9710 at commit c5b16cf.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • >>> class VectorAccumulatorParam(AccumulatorParam):

[JoshRosen]

Upthread, I suggested taking the code which handles the xmlrunner imports and configuration and putting it into a reusable function which is defined in one place and called in the __main__ method of all test suites. Could you do this instead of copy-pasting the code into every file?

[gliptak]

@JoshRosen Please review accumulators.py

[JoshRosen]

Before merging, we're going to want to remove this print statement I think.

[JoshRosen]

Those changes to accumulators.py are a huge improvement, so I'd go ahead and apply that same pattern across the other files.

[SparkQA]

Test build #49965 has finished for PR 9710 at commit c56132e.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #50045 has finished for PR 9710 at commit 61a4c47.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gliptak]

@JoshRosen A number of the doctests do fail locally too. I'm also unsure why only a single test result shows ...

[gliptak]

@JoshRosen Could you offer ointers on how to move this forward? Thanks

[gliptak]

Rebased.

[SparkQA]

Test build #51137 has finished for PR 9710 at commit 1c4e070.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #51149 has finished for PR 9710 at commit 9e1b8b2.

• This patch fails PySpark unit tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #51158 has finished for PR 9710 at commit 49dc7a0.

• This patch fails PySpark unit tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[gliptak]

@JoshRosen Could you offer pointers on how to move this forward? Thanks

[gliptak]

@JoshRosen Are you still considering this for inclusion? Thanks

[SparkQA]

Test build #53638 has finished for PR 9710 at commit 4e34b24.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.