SPARK-1588 Re-add support for SPARK_YARN_USER_ENV and SPARK_JAVA_OPTS #514
Conversation
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14391/ |
|
Unrelated failures imo ? |
|
Jenkins, retest this please. |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14397/ |
|
again unrelated failures imo ... CC @pwendell ... any idea why these tests keep failing ? |
|
@mridulm this has been there for a long while, I just filed an issue about this https://issues.apache.org/jira/browse/SPARK-1600 I once reported it in the dev mail list http://apache-spark-developers-list.1001551.n3.nabble.com/a-weird-test-case-in-Streaming-td6116.html |
|
ah ok, that sucks ... |
|
Jenkins, retest this please. |
|
Merged build triggered. |
|
Merged build started. |
|
Hey @mridulm thanks for reporting this - I need to go and audit the spark-submit changes and make sure we properly backwards support all these options on YARN. I'm going to look at it this weekend. I think the issue with the test is currently unknown... however might be worth digging into |
|
I made this JIRA for it. https://issues.apache.org/jira/browse/SPARK-1625 |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
Thanks patrick - looks like the tests passed now ! Note that we used to propagate all SPARK_* env variables from Client to app master and so on - which got removed; which caused part of the problem (I selectively re-added only for these two variables, but it is possible other variables might be broken too ?). |
|
Closing this assuming it has been resolved through other PR's. Please let me know in case this is not the case. |
* first stage of PR apache#514 of just logic * fixing build and unit test issues * fixed integration tests * fixed issue with executorPodFactory unit tests * first series of PR comments * handle most PR comments * third round of PR comments * initial round of comments and initial unit tests for deploy * handled most of the comments and added test cases for pods * resolve conflicts * merge conflicts * adding thread sleeping for RSS issues as a test * resolving comments and unit testing * regarding comments on PR
…adaptive execution (auto setting the number of reducer)" (apache#514) This reverts https://github.com/palantir/spark/pull/443/files#diff-3cd46a3f60c5352282bd3f2c9efff7fc fysa @justinuang This reverts commit 8a4a29b, reversing changes made to 8bfddaf. ## Upstream SPARK-XXXXX ticket and PR link (if not applicable, explain) ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review http://spark.apache.org/contributing.html before opening a pull request.
…ork for adaptive execution (auto setting the number of reducer)" (apache#514)" This reverts commit a763413.
apache#514) * Add job for running tensorflow benchmarks with kubeflow/volcano on k8s This change added a job for running the tensorflow benchmarks on kubeflow + k8s 1.14 or volcano 0.5.0 +k8s 1.14 implement: theopenlab/openlab#256
4 participants
This is what I did to unblock our jobs - please feel free to modify or create a new PR based on this in case this wont suffice.
Unfortunately, I wont be able to shepard this patch actively ...