SPARK-7021 Add JUnit output for Python unit tests #8323

gliptak · 2015-08-19T22:52:27Z

WIP

JoshRosen · 2015-08-19T23:01:46Z

Jenkins, this is ok to test.

SparkQA · 2015-08-19T23:08:47Z

Test build #41290 has finished for PR 8323 at commit d93e7cd.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class _DelegateIO(object):
- class _TestInfo(object):
- class _XMLTestResult(_TextTestResult):
- class XMLTestRunner(TextTestRunner):

gliptak · 2015-08-19T23:31:41Z

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41290/console
Recording test results
ERROR: Publisher 'Publish JUnit test result report' failed: No test report files were found. Configuration error?
Test FAILed.

Can somebody comment on how is Jenkins configured to look for JUnit results (or did this build fail on RAT)? Thanks

JoshRosen · 2015-08-19T23:44:17Z

The build failed on RAT:

Could not find Apache license headers in the following files:
 !????? /home/jenkins/workspace/SparkPullRequestBuilder/dev/xmlrunner/__init__.py
 !????? /home/jenkins/workspace/SparkPullRequestBuilder/dev/xmlrunner/xmlrunner.py
[error] running /home/jenkins/workspace/SparkPullRequestBuilder/dev/check-license ; received return code 1

gliptak · 2015-08-19T23:45:30Z

I just added the headers, xmlrunner is LGPL (and only used to run the tests).

SparkQA · 2015-08-19T23:48:46Z

Test build #41291 has finished for PR 8323 at commit d4f48ba.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class _DelegateIO(object):
- class _TestInfo(object):
- class _XMLTestResult(_TextTestResult):
- class XMLTestRunner(TextTestRunner):

gliptak · 2015-08-20T13:33:38Z

Is this a no-go because of the license? Thanks

mengxr · 2015-09-22T01:44:12Z

@gliptak Is https://pypi.python.org/pypi/unittest-xml-reporting/1.12.0 an alternative? It is BSD license. We can install the package on Jenkins machines, instead of including the source code in the codebase.

gliptak · 2015-09-23T00:55:11Z

@mengxr Would installing on Jenkins be independent from the codebase (the import xmlrunner just be available)? Thanks

mengxr · 2015-09-23T04:19:18Z

If this new library works, I think we could install it on Spark Jenkins. Could you test it locally and see whether it works?

gliptak · 2015-09-23T22:40:22Z

I tried this locally and the files were generated. Somebody with visibility on Jenkins might need to comment on file locations ...

SparkQA · 2015-09-23T22:51:18Z

Test build #42930 has finished for PR 8323 at commit a6b3a5b.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-09-23T23:08:49Z

Test build #42933 has finished for PR 8323 at commit 05e5a37.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-09-23T23:19:31Z

Test build #42935 has finished for PR 8323 at commit 18f5901.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

mengxr · 2015-09-24T07:44:47Z

Please fix Python style issue:

./python/pyspark/streaming/tests.py:1303:13: E128 continuation line under-indented for visual indent

mengxr · 2015-09-24T07:45:21Z

python/pyspark/streaming/tests.py

this should be aligned with tests,. you can run dev/lint-python to check style locally.

gliptak · 2015-09-24T21:10:12Z

@mengxr Thanks for the local lint pointer

SparkQA · 2015-09-24T21:27:29Z

Test build #42980 has finished for PR 8323 at commit e9a711e.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

mengxr · 2015-09-24T21:49:52Z

@shaneknapp Could you take a look at this PR and see whether we can install https://pypi.python.org/pypi/unittest-xml-reporting/1.12.0 on Jenkins?

@gliptak We shouldn't make xmlrunner a requirement on local machines. If it is installed, we output XML. Otherwise, we should fallback to the default behavior. This could be done by catching exceptions at the import line.

shaneknapp · 2015-09-24T22:48:10Z

could you create a spark issue for the plugin and assign it to me?

also, i won't be able to do this until monday, as i'm about to leave town
for the weekend.

On Thu, Sep 24, 2015 at 2:50 PM, Xiangrui Meng [email protected]
wrote:

@shaneknapp https://github.com/shaneknapp Could you take a look at this
PR and see whether we can install
https://pypi.python.org/pypi/unittest-xml-reporting/1.12.0 on Jenkins?

@gliptak https://github.com/gliptak We shouldn't make xmlrunner a
requirement on local machines. If it is installed, we output XML.
Otherwise, we should fallback to the default behavior. This could be done
by catching exceptions at the import line.

—
Reply to this email directly or view it on GitHub
#8323 (comment).

gliptak · 2015-09-25T00:18:59Z

@mengxr Revised.

SparkQA · 2015-09-25T00:34:04Z

Test build #42996 has finished for PR 8323 at commit 66bdb2b.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

mengxr · 2015-09-25T07:50:43Z

@shaneknapp I created https://issues.apache.org/jira/browse/SPARK-10830 and assigned it to you.

mengxr · 2015-09-25T07:50:49Z

test this please

SparkQA · 2015-09-25T08:23:25Z

Test build #43015 has finished for PR 8323 at commit 66bdb2b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

mengxr · 2015-09-28T22:42:08Z

test this please

mengxr · 2015-09-28T22:42:58Z

unittest-xml-reporting is installed on AMPLab Jenkins. So let's call Jenkins again.

SparkQA · 2015-09-29T00:46:04Z

Test build #1821 has finished for PR 8323 at commit 66bdb2b.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gliptak · 2015-09-29T01:02:05Z

https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1810/testReport/

doesn't seem to list the Python tests.

Can somebody with access to Jenkins configuration review?

Thanks

shaneknapp · 2015-09-29T03:45:17Z

i'll take a look at the config tomorrow

On Mon, Sep 28, 2015 at 6:02 PM, gliptak [email protected] wrote:

https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1810/testReport/

doesn't seem to list the Python tests.

Can somebody with access to Jenkins configuration review?

Thanks

—
Reply to this email directly or view it on GitHub
#8323 (comment).

mengxr · 2015-09-30T01:04:23Z

test this please

SparkQA · 2015-09-30T01:28:50Z

Test build #43109 has finished for PR 8323 at commit cd654c7.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

mengxr · 2015-09-30T02:08:31Z

test this please

SparkQA · 2015-09-30T02:28:31Z

Test build #43113 has finished for PR 8323 at commit cd654c7.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

gliptak · 2015-09-30T21:57:04Z

@mengxr Thank you for guiding this

mengxr · 2015-10-02T06:47:44Z

test this please

mengxr · 2015-10-02T06:48:17Z

it seems that this PR has a higher-than-normal failure rate .. I don't know why. +@davies

SparkQA · 2015-10-02T07:13:52Z

Test build #43176 has finished for PR 8323 at commit cd654c7.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

mengxr · 2015-10-05T21:49:59Z

test this please

SparkQA · 2015-10-05T22:15:07Z

Test build #43252 has finished for PR 8323 at commit cd654c7.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2015-10-06T18:57:47Z

Huh, looks like the last two failures were for the same PySpark test:

======================================================================
FAIL [0.000s]: test_pickling_file_handles (__main__.SerializationTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/tests.py", line 259, in test_pickling_file_handles
    self.assertEqual(out1, out2)
AssertionError: <StringIO.StringIO instance at 0x2d65a28> != <StringIO.StringIO instance at 0x2ffdb90>

JoshRosen · 2015-10-06T19:00:05Z

I guess that this doesn't handle doctests, right? Is there any way to get XML for those, too? If not, could you edit the PR title / description to reflect this?

JoshRosen · 2015-10-06T19:01:23Z

@gliptak, can you try to reproduce the failure locally? See ./pyspark/run-tests --help for hints on how to run only this test suite under Python 2.6.

gliptak · 2015-10-07T23:48:39Z

@JoshRosen I do not have python2.6 installed (or available as a package) locally

gliptak · 2015-10-08T00:53:14Z

@JoshRosen How are the doctest-s triggered on the commandline and in Jenkins?

$ python/run-tests --help
No handlers could be found for logger "root"
Usage: run-tests [options]

Options:
  -h, --help            show this help message and exit
  --python-executables=PYTHON_EXECUTABLES
                        A comma-separated list of Python executables to test
                        against (default: python,python3.4)
  --modules=MODULES     A comma-separated list of Python modules to test
                        (default: pyspark-core,pyspark-ml,pyspark-mllib
                        ,pyspark-sql,pyspark-streaming)
  -p PARALLELISM, --parallelism=PARALLELISM
                        The number of suites to test in parallel (default 4)
  --verbose             Enable additional debug logging

gliptak · 2015-10-14T01:59:59Z

What is the next step on this pull request? Thanks

mengxr · 2015-10-16T21:26:43Z

@gliptak I had an offline discussion with @davies . Let's do the following:

Change L259 in pyspark/tests.py to:

  # Regression test for SPARK-3415
  def test_pickling_file_handles(self):
      # JIRA number here
      if xmlrunner is None:
          ser = CloudPickleSerializer()
          out1 = sys.stderr
          out2 = ser.loads(ser.dumps(out1))
          self.assertEqual(out1, out2)

Create a JIRA for this so we remember to fix it. The issue is CloudPickeSerializer wraps stderr, which conflicts with xmlrunner. But it might take some time to fix.

Change the PR title to `Python unit tests" since we do not report doctests.

Then we can run Jenkins again and see whether it works.

SparkQA · 2015-10-16T22:47:35Z

Test build #43863 has finished for PR 8323 at commit 177fef9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

davies · 2015-10-16T22:56:19Z

Right now, all the Python unit tests are in the package of (root), is there a way to set a package for it (such as pyspark)?

gliptak · 2015-10-16T23:32:15Z

@davies Not that I see. I opened xmlrunner/unittest-xml-reporting#95 to discuss.

gliptak · 2015-10-16T23:35:05Z

@JoshRosen I opened xmlrunner/unittest-xml-reporting#94 to discuss doctests processing.

mengxr · 2015-10-22T21:44:12Z

This looks good to me. I think we can address the test namespace issue in a later PR. @davies @JoshRosen ?

davies · 2015-10-22T22:26:53Z

LGTM, merging this into master, thanks!

gliptak · 2015-10-22T23:37:09Z

As per xmlrunner/unittest-xml-reporting#95 the Python tests do not have packages (or possibly not being recognized by https://github.com/xmlrunner/unittest-xml-reporting/

gliptak · 2015-10-24T16:49:17Z

Adding packages to XMLs at #9263

mengxr reviewed Sep 24, 2015
View reviewed changes

gliptak changed the title ~~SPARK-7021 Add JUnit output for Python tests~~ SPARK-7021 Add JUnit output for Python unit tests Oct 16, 2015

SPARK-7021 Add JUnit output for Python unit tests

177fef9

asfgit closed this in 163d53e Oct 22, 2015

gliptak mentioned this pull request Nov 14, 2015

SPARK-11741 Process doctests using TextTestRunner/XMLTestRunner #9710

Closed

SPARK-7021 Add JUnit output for Python unit tests #8323

SPARK-7021 Add JUnit output for Python unit tests #8323

Uh oh!

Conversation

gliptak commented Aug 19, 2015

Uh oh!

JoshRosen commented Aug 19, 2015

Uh oh!

SparkQA commented Aug 19, 2015

Uh oh!

gliptak commented Aug 19, 2015

Uh oh!

JoshRosen commented Aug 19, 2015

Uh oh!

gliptak commented Aug 19, 2015

Uh oh!

SparkQA commented Aug 19, 2015

Uh oh!

gliptak commented Aug 20, 2015

Uh oh!

mengxr commented Sep 22, 2015

Uh oh!

gliptak commented Sep 23, 2015

Uh oh!

mengxr commented Sep 23, 2015

Uh oh!

gliptak commented Sep 23, 2015

Uh oh!

SparkQA commented Sep 23, 2015

Uh oh!

SparkQA commented Sep 23, 2015

Uh oh!

SparkQA commented Sep 23, 2015

Uh oh!

mengxr commented Sep 24, 2015

Uh oh!

mengxr Sep 24, 2015

Choose a reason for hiding this comment

Uh oh!

gliptak commented Sep 24, 2015

Uh oh!

SparkQA commented Sep 24, 2015

Uh oh!

mengxr commented Sep 24, 2015

Uh oh!

shaneknapp commented Sep 24, 2015

Uh oh!

gliptak commented Sep 25, 2015

Uh oh!

SparkQA commented Sep 25, 2015

Uh oh!

mengxr commented Sep 25, 2015

Uh oh!

mengxr commented Sep 25, 2015

Uh oh!

SparkQA commented Sep 25, 2015

Uh oh!

mengxr commented Sep 28, 2015

Uh oh!

mengxr commented Sep 28, 2015

Uh oh!

SparkQA commented Sep 29, 2015

Uh oh!

gliptak commented Sep 29, 2015

Uh oh!

shaneknapp commented Sep 29, 2015

Uh oh!

mengxr commented Sep 30, 2015

Uh oh!

SparkQA commented Sep 30, 2015

Uh oh!

mengxr commented Sep 30, 2015

Uh oh!

SparkQA commented Sep 30, 2015

Uh oh!

gliptak commented Sep 30, 2015

Uh oh!

mengxr commented Oct 2, 2015

Uh oh!

mengxr commented Oct 2, 2015