[SPARK-20393][WEBU UI][1.6] Strengthen Spark to prevent XSS vulnerabilities #19528

ambauma · 2017-10-18T19:49:32Z

What changes were proposed in this pull request?

This is the fix for the master branch applied to the 1.6 branch. My (unnamed) company will be using Spark 1.6 probably for another year. We have been blocked from having Spark 1.6 on our workstations until CVE-2017-7678 is patched, which SPARK-20393 does. I realize there will not be an official Spark 1.6.4 release, but it still seems wise to keep the code there patched for those who are stuck on that version. Otherwise I imagine several forks duplicating 1.6 compliance and security fixes.

How was this patch tested?

The patch came with unit tests. The test build passed. Manual testing on one of the effected screens showed the newline character removed. Screen display was the same regardless (html ignores newline characters).

Please review http://spark.apache.org/contributing.html before opening a pull request.

The patch itself is from previous pull requests associated to SPARK-20939. My original "work" was actions on what to apply to branch 1.6. and I license the work to the project under the project’s open source license.

…6 branch from the merge of SPARK-20393

dongjoon-hyun

Hi, @ambauma .
Thank you for contributing.

SPARK-20393 should be backported branch-2.0 first.
PR titles should be the same with the original one except the version numbers.

[SPARK-20393][WEBU UI][2.0] Strengthen Spark to prevent XSS vulnerabilities
[SPARK-20393][WEBU UI][1.6] Strengthen Spark to prevent XSS vulnerabilities

ambauma · 2017-10-18T21:04:12Z

Understood. Working on porting to 2.0...

srowen · 2017-10-19T07:24:15Z

I don't think there are any more 1.x releases coming, and doubt there are more 2.0.x releases. Do you really need this in Spark or is it something you can apply to your own release branch?

ambauma · 2017-10-19T17:47:22Z

I have a release in my fork for my immediate needs. However, Spark 1.6 is still included in Hortonworks and is default in Cloudera. This patch addresses CVE-2017-7678. Some companies in strict regulatory environments may fail audits and be forced to remove Spark 1.6 if it is not patched. Rather than keeping security patches in forks, I think it makes sense to merge them back into the official branch for branches that are still in active use. That way if I get hit by a bus and CVE-2018-XXXX comes out, CVE-2017-7678 will already be covered and the work will not need to be duplicated.

srowen · 2017-10-19T17:49:15Z

(Spark 1.x is legacy in Cloudera, but, it has its own 1.x branch anyway)
I think it's not a big deal to backport if it goes into later branches first, sure. But I doubt there is another 1.x release here.

ambauma · 2017-10-19T17:59:32Z

I just posted the 2.0 pull request. #19538

SparkQA · 2017-10-19T20:17:30Z

Test build #3955 has finished for PR 19528 at commit ffe3e98.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

ambauma · 2017-10-20T02:19:16Z

Working on duplicating PySpark failures...

ambauma · 2017-10-20T03:38:12Z

Able to duplicate. Working theory is that this is related to numpy 1.12.1. Here is my conda env:

ca-certificates 2017.08.26 h1d4fec5_0
certifi 2016.2.28 py34_0
intel-openmp 2018.0.0 h15fc484_7
libedit 3.1 heed3624_0
libffi 3.2.1 h4deb6c0_3
libgcc-ng 7.2.0 h7cc24e2_2
libgfortran 1.0 0
libstdcxx-ng 7.2.0 h7a57d05_2
mkl 2017.0.3 0
ncurses 6.0 h06874d7_1
numpy 1.12.1 py34_0
openblas 0.2.19 0
openssl 1.0.2l h077ae2c_5
pip 9.0.1 py34_1
python 3.4.5 0
readline 6.2 2
setuptools 27.2.0 py34_0
sqlite 3.13.0 0
tk 8.5.18 0
wheel 0.29.0 py34_0
xz 5.2.3 h2bcbf08_1
zlib 1.2.11 hfbfcf68_1

…n LogisticRegressionModel ## What changes were proposed in this pull request? Fixed TypeError with python3 and numpy 1.12.1. Numpy's `reshape` no longer takes floats as arguments as of 1.12. Also, python3 uses float division for `/`, we should be using `//` to ensure that `_dataWithBiasSize` doesn't get set to a float. ## How was this patch tested? Existing tests run using python3 and numpy 1.12. Author: Bago Amirbekian <[email protected]> Closes apache#18081 from MrBago/BF-py3floatbug.

ambauma · 2017-10-20T03:54:02Z

Believed fixed. Hard to say for sure without knowing the precise python and numpy versions the build is using.

srowen · 2017-10-20T07:36:16Z

core/src/main/scala/org/apache/spark/ui/jobs/JobsTab.scala


 package org.apache.spark.ui.jobs
-
+import javax.servlet.http.HttpServletRequest


Hm, I'm not sure if this back-port is correct. This file's change doesn't look like it does anything and I don't see this change in the original: https://github.com/apache/spark/pull/17686/files

Will look into this...

Agreed, will remove.

srowen · 2017-10-20T07:36:29Z

resource-managers/mesos/src/main/scala/org/apache/spark/deploy/mesos/ui/DriverPage.scala

@@ -0,0 +1,180 @@
+/*


Likewise this isn't part of the backport is it? https://github.com/apache/spark/pull/17686/files

I'll look into this as well...

I'm not sure what I did to make this whole file look new, but I've copied the 1.6 current and reapplied stripXSS locally. Waiting for my build to pass to commit again.

srowen · 2017-10-20T07:36:33Z

python/pyspark/mllib/classification.py

            self._weightsMatrix = None
        else:
-            self._dataWithBiasSize = self._coeff.size / (self._numClasses - 1)
+            self._dataWithBiasSize = self._coeff.size // (self._numClasses - 1)


I had to apply this to get past a python unit test failure. My assumption is that the NewSparkPullRequestBuilder is on a different version of numpy than when the Spark 1.6 branch was last built. The current python unit test failure looks like it has to do with a novel version of SciPy.

This is already fixed in the 2.0 branch, btw. Just was never applied to 1.6. [SPARK-20862]

SparkQA · 2017-10-20T09:42:11Z

Test build #3958 has finished for PR 19528 at commit cb1609b.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

felixcheung · 2017-10-21T06:58:49Z

core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala


 import scala.collection.mutable.{HashMap, ListBuffer}
 import scala.xml._
+import scala.collection.JavaConverters._


is this needed?

felixcheung · 2017-10-23T16:59:15Z

Jenkins, retest this please

SparkQA · 2017-10-23T19:00:20Z

Test build #82990 has finished for PR 19528 at commit 76ad8c5.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

ambauma · 2017-10-27T13:42:02Z

I'm unable to duplicate the PySpark failures locally. I assume I need a specific version of SciPy to duplicate the error. Is there a way I could get what versions the build server is running? Something like:
sorted(["%s==%s" % (i.key, i.version) for i in pip.get_installed_distributions()]) for python and python 3.4?

felixcheung · 2017-10-31T05:15:53Z

@shaneknapp - could you help check - what version of SciPy Jenkins is running with? thanks!

shaneknapp · 2017-12-14T22:00:05Z

python2.7: 0.17.0
python3: 0.18.1

shaneknapp · 2018-01-18T19:15:41Z

ok to test

felixcheung · 2018-01-20T07:36:33Z

Jenkins test this please

SparkQA · 2018-01-20T07:37:55Z

Test build #86410 has started for PR 19528 at commit 76ad8c5.

jiangxb1987 · 2018-04-04T13:26:44Z

retest this please

SparkQA · 2018-04-04T15:42:40Z

Test build #4149 has finished for PR 19528 at commit 76ad8c5.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

jiangxb1987 · 2018-06-12T00:16:06Z

retest this please

felixcheung · 2018-06-12T06:31:05Z

Jenkins test this please

SparkQA · 2018-06-12T06:33:19Z

Test build #91690 has started for PR 19528 at commit 76ad8c5.

HyukjinKwon · 2018-07-16T02:58:08Z

retest this please

SparkQA · 2018-07-16T03:03:33Z

Test build #93074 has started for PR 19528 at commit 76ad8c5.

HyukjinKwon · 2018-07-16T07:17:28Z

retest this please

SparkQA · 2018-07-16T07:26:27Z

Test build #93094 has finished for PR 19528 at commit 76ad8c5.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2018-07-26T00:31:38Z

Can one of the admins verify this patch?

n-marion and others added 3 commits October 9, 2017 15:28

Initial Merge of SPARK-20393 to 1.6 branch

d9a45aa

Removing what I believe is extra code never intended for the Spark 1.…

630854a

…6 branch from the merge of SPARK-20393

Adding back in DriverPage.scala

ffe3e98

dongjoon-hyun reviewed Oct 18, 2017

View reviewed changes

ambauma changed the title ~~[SPARK-20393] [Core] Existing patch applied to 1.6 branch.~~ [SPARK-20393][WEBU UI][1.6] Strengthen Spark to prevent XSS vulnerabilities Oct 18, 2017

Added missing XSS strip based on comparison to 2.0 branch.

cafe18d

srowen requested changes Oct 20, 2017

View reviewed changes

felixcheung mentioned this pull request Oct 20, 2017

[SPARK-20393][WEBU UI][BACKPORT-2.0] Strengthen Spark to prevent XSS vulnerabilities #19538

Closed

felixcheung reviewed Oct 21, 2017

View reviewed changes

Changes based on comments.

76ad8c5

srowen mentioned this pull request Aug 20, 2018

[BUILD] Close stale PRs #22159

Closed

asfgit closed this in b8788b3 Aug 21, 2018


		package org.apache.spark.ui.jobs

		import javax.servlet.http.HttpServletRequest

[SPARK-20393][WEBU UI][1.6] Strengthen Spark to prevent XSS vulnerabilities #19528

[SPARK-20393][WEBU UI][1.6] Strengthen Spark to prevent XSS vulnerabilities #19528

Uh oh!

Conversation

ambauma commented Oct 18, 2017

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

ambauma commented Oct 18, 2017

Uh oh!

srowen commented Oct 19, 2017

Uh oh!

ambauma commented Oct 19, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

srowen commented Oct 19, 2017

Uh oh!

ambauma commented Oct 19, 2017

Uh oh!

SparkQA commented Oct 19, 2017

Uh oh!

ambauma commented Oct 20, 2017

Uh oh!

ambauma commented Oct 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ambauma commented Oct 20, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 20, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

felixcheung commented Oct 23, 2017

Uh oh!

SparkQA commented Oct 23, 2017

Uh oh!

ambauma commented Oct 27, 2017

Uh oh!

felixcheung commented Oct 31, 2017

Uh oh!

shaneknapp commented Dec 14, 2017

Uh oh!

shaneknapp commented Jan 18, 2018

Uh oh!

felixcheung commented Jan 20, 2018

Uh oh!

SparkQA commented Jan 20, 2018

Uh oh!

jiangxb1987 commented Apr 4, 2018

Uh oh!

SparkQA commented Apr 4, 2018

Uh oh!

jiangxb1987 commented Jun 12, 2018

Uh oh!

felixcheung commented Jun 12, 2018

Uh oh!

SparkQA commented Jun 12, 2018

Uh oh!

HyukjinKwon commented Jul 16, 2018

ambauma commented Oct 19, 2017 •

edited

Loading

ambauma commented Oct 20, 2017 •

edited

Loading