[SPARK-40397][BUILD] Upgrade `org.scalatestplus:selenium` to 3.12.13 #37868

LuciferYang · 2022-09-13T16:07:22Z

What changes were proposed in this pull request?

The main change of this pr as follows:

Upgrade org.scalatestplus:selenium from org.scalatestplus:selenium-3-141:3.2.10.0 to org.scalatestplus:selenium-4-2:3.2.13.0 and upgrade selenium-java from 3.141.59 to 4.2.2, htmlunit-driver from 2.62.0 to 3.62.0
okio upgrade from 1.14.0 to 1.15.0 due to both selenium-java and kubernetes-client depends on okio 1.15.0 and maven's nearby choice has also changed from 1.14.0 to 1.15.0

Why are the changes needed?

Use the same version as other org.scalatestplus series dependencies, the release notes as follows:

Does this PR introduce any user-facing change?

No

How was this patch tested?

Pass GitHub Actions
Manual test:
- ChromeUISeleniumSuite

build/sbt -Dguava.version=31.1-jre -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest.default.exclude.tags="" -Phive -Phive-thriftserver "core/testOnly org.apache.spark.ui.ChromeUISeleniumSuite"

[info] ChromeUISeleniumSuite:
Starting ChromeDriver 105.0.5195.52 (412c95e518836d8a7d97250d62b29c2ae6a26a85-refs/branch-heads/5195@{#853}) on port 53917
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
[info] - SPARK-31534: text for tooltip should be escaped (4 seconds, 447 milliseconds)
[info] - SPARK-31882: Link URL for Stage DAGs should not depend on paged table. (841 milliseconds)
[info] - SPARK-31886: Color barrier execution mode RDD correctly (297 milliseconds)
[info] - Search text for paged tables should not be saved (1 second, 676 milliseconds)
[info] Run completed in 11 seconds, 819 milliseconds.
[info] Total number of tests run: 4
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 4, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 25 s, completed 2022-9-14 20:12:28

ChromeUIHistoryServerSuite

build/sbt -Dguava.version=31.1-jre -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest.default.exclude.tags="" -Phive -Phive-thriftserver "core/testOnly org.apache.spark.deploy.history.ChromeUIHistoryServerSuite"

[info] ChromeUIHistoryServerSuite:
Starting ChromeDriver 105.0.5195.52 (412c95e518836d8a7d97250d62b29c2ae6a26a85-refs/branch-heads/5195@{#853}) on port 58567
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
[info] - ajax rendered relative links are prefixed with uiRoot (spark.ui.proxyBase) (2 seconds, 416 milliseconds)
[info] Run completed in 8 seconds, 936 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 30 s, completed 2022-9-14 20:11:34

LuciferYang · 2022-09-13T16:08:17Z

dev/deps/spark-deps-hadoop-2-hive-2.3

 objenesis/3.2//objenesis-3.2.jar
 okhttp/3.12.12//okhttp-3.12.12.jar
-okio/1.14.0//okio-1.14.0.jar
+okio/1.15.0//okio-1.15.0.jar


The reason for the change is explained in #37867

When using selenium 3.1, Maven chose nearby okio 1.14.0., both selenium 4.2 and k8s-client rely on 1.15.0 now

LuciferYang · 2022-09-13T16:11:27Z

pom.xml

+      </dependency>
+      <dependency>
+        <groupId>org.seleniumhq.selenium</groupId>
+        <artifactId>selenium-remote-driver</artifactId>


Explicit management selenium-remote-driver because it requires exclude netty, guava, bytebuddy as before

Why not just only exclude netty, guava, and bytebuddy rather than selenium-remote-driver?

When using selenium 3.1, netty, guava, and bytebuddy is the direct dependency of selenium-java, so we can directly exclude them.

But after migrating to selenium 4.2 , netty, guava, and bytebuddy is directly dependency of selenium-remote-driver, not selenium-java.

Hmm, I think we can exclude indirect dependencies.
I confirmed with build/mvn -Phive -Phive-thriftserver -DskipTests dependency:tree.

jackson-module-scala also excludes an indirect dependency.

spark/pom.xml

Lines 1001 to 1011 in a2304e4

<dependency>

<groupId>com.fasterxml.jackson.module</groupId>

<artifactId>jackson-module-scala_${scala.binary.version}</artifactId>

<version>${fasterxml.jackson.version}</version>

<exclusions>

<exclusion>

<groupId>com.google.guava</groupId>

<artifactId>guava</artifactId>

</exclusion>

</exclusions>

</dependency>

Amazing!!! Let me have a try

like

<dependency> <groupId>org.seleniumhq.selenium</groupId> <artifactId>selenium-java</artifactId> <version>${selenium.version}</version> <scope>test</scope> <exclusions> <exclusion> <groupId>com.google.guava</groupId> <artifactId>guava</artifactId> </exclusion> <exclusion> <groupId>io.netty</groupId> <artifactId>*</artifactId> </exclusion> <exclusion> <groupId>net.bytebuddy</groupId> <artifactId>byte-buddy</artifactId> </exclusion> </exclusions> </dependency>

?

Yes. But as you noticed, we may need some additional exclusions.

HyukjinKwon · 2022-09-14T03:16:50Z

cc @sarutak FYI

LuciferYang · 2022-09-14T03:17:47Z

thanks @HyukjinKwon

LuciferYang · 2022-09-14T09:37:41Z

pom.xml

            <artifactId>guava</artifactId>
          </exclusion>
+          <exclusion>
+            <groupId>com.google.auto.service</groupId>


com.google.auto.service:autoservice and com.google.auto.service:auto-service-annotations need guava,try add excludes.

Otherwise, mvn will fail to compile:

[WARNING] Unexpected javac output: 错误: 服务配置文件不正确, 或构造处理程序对象javax.annotation.processing.Processor: Provider com.google.auto.service.processor.AutoServicebe instantiated: java.lang.NoClassDefFoundError: com/google/common/collect/Multimap时抛出异常错误. [WARNING] javac exited with exit code 1 [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for Spark Project Parent POM 3.4.0-SNAPSHOT: [INFO] [INFO] Spark Project Parent POM ........................... SUCCESS [ 4.620 s] [INFO] Spark Project Tags ................................. FAILURE [ 4.051 s] [INFO] Spark Project Local DB ............................. SKIPPED [INFO] Spark Project Networking ........................... SKIPPED [INFO] Spark Project Shuffle Streaming Service ............ SKIPPED [INFO] Spark Project Unsafe ............................... SKIPPED [INFO] Spark Project Launcher ............................. SKIPPED [INFO] Spark Project Core ................................. SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 9.556 s [INFO] Finished at: 2022-09-14T17:20:07+08:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.7.1:testCompile (scala-test-compile-first) on project spark-tags_2.12: Execution scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:4.7.1:testCompile failed: javac returned non-zero exit code: CompileFailed -> [Help 1]

sarutak · 2022-09-14T11:08:02Z

Before this change, ChromeUISeleniumSuite and ChromeUIHistoryServerSuite can run with:

build/sbt -Dguava.version=27.0-jre -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest.default.exclude.tags="" -Phive -Phive-thriftserver 'testOnly Chrome*Suite'

After this change, they need newer version of Guava.

build/sbt -Dguava.version=31.1-jre -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest.default.exclude.tags="" -Phive -Phive-thriftserver 'testOnly Chrome*Suite'

But I think it's OK. Spark itself still depends on Guava 14.0.1.

LuciferYang · 2022-09-14T12:17:04Z

Before this change, ChromeUISeleniumSuite and ChromeUIHistoryServerSuite can run with:
build/sbt -Dguava.version=27.0-jre -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest.default.exclude.tags="" -Phive -Phive-thriftserver 'testOnly Chrome*Suite'
After this change, they need newer version of Guava.
build/sbt -Dguava.version=31.1-jre -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest.default.exclude.tags="" -Phive -Phive-thriftserver 'testOnly Chrome*Suite'
But I think it's OK. Spark itself still depends on Guava 14.0.1.

Manual test and supplemented with pr description. Thanks ~

srowen · 2022-09-14T13:13:02Z

pom.xml

    <jpam.version>1.1</jpam.version>
-    <selenium.version>3.141.59</selenium.version>
+    <selenium.version>4.2.2</selenium.version>
+    <htmlunit-driver.version>3.62.0</htmlunit-driver.version>


Huh, so we're using two different versions of different htmlunit components? what if all of htmlunit is updated?

No, htmlunit-driver is in SeleniumHQ, the latest version is 3.64.0, htmlUnit is in HtmlUnit, and the latest version is 2.64.0

https://github.com/SeleniumHQ/htmlunit-driver/releases

https://github.com/HtmlUnit/htmlunit/releases

Got it, and can it match the selenium version then or are they released pretty differently?

This is a different project maintained by two organizations, the major versions are different

Yes, they match the version of selenium

sarutak · 2022-09-15T01:40:41Z

Merging to master.

### What changes were proposed in this pull request? The main change of this pr as follows: - Upgrade `org.scalatestplus:selenium` from `org.scalatestplus:selenium-3-141:3.2.10.0` to `org.scalatestplus:selenium-4-2:3.2.13.0` and upgrade selenium-java from `3.141.59` to `4.2.2`, `htmlunit-driver` from `2.62.0` to `3.62.0` - okio upgrade from `1.14.0` to `1.15.0` due to both selenium-java and kubernetes-client depends on okio 1.15.0 and maven's nearby choice has also changed from 1.14.0 to 1.15.0 ### Why are the changes needed? Use the same version as other `org.scalatestplus` series dependencies, the release notes as follows: - https://github.com/scalatest/scalatestplus-selenium/releases/tag/release-3.2.11.0-for-selenium-4.1 - https://github.com/scalatest/scalatestplus-selenium/releases/tag/release-3.2.12.0-for-selenium-4.1 - https://github.com/scalatest/scalatestplus-selenium/releases/tag/release-3.2.12.1-for-selenium-4.1 - https://github.com/scalatest/scalatestplus-selenium/releases/tag/release-3.2.13.0-for-selenium-4.2 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass GitHub Actions - Manual test: - ChromeUISeleniumSuite ``` build/sbt -Dguava.version=31.1-jre -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest.default.exclude.tags="" -Phive -Phive-thriftserver "core/testOnly org.apache.spark.ui.ChromeUISeleniumSuite" ``` ``` [info] ChromeUISeleniumSuite: Starting ChromeDriver 105.0.5195.52 (412c95e518836d8a7d97250d62b29c2ae6a26a85-refs/branch-heads/5195{apache#853}) on port 53917 Only local connections are allowed. Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe. ChromeDriver was started successfully. [info] - SPARK-31534: text for tooltip should be escaped (4 seconds, 447 milliseconds) [info] - SPARK-31882: Link URL for Stage DAGs should not depend on paged table. (841 milliseconds) [info] - SPARK-31886: Color barrier execution mode RDD correctly (297 milliseconds) [info] - Search text for paged tables should not be saved (1 second, 676 milliseconds) [info] Run completed in 11 seconds, 819 milliseconds. [info] Total number of tests run: 4 [info] Suites: completed 1, aborted 0 [info] Tests: succeeded 4, failed 0, canceled 0, ignored 0, pending 0 [info] All tests passed. [success] Total time: 25 s, completed 2022-9-14 20:12:28 ``` - ChromeUIHistoryServerSuite ``` build/sbt -Dguava.version=31.1-jre -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest.default.exclude.tags="" -Phive -Phive-thriftserver "core/testOnly org.apache.spark.deploy.history.ChromeUIHistoryServerSuite" ``` ``` [info] ChromeUIHistoryServerSuite: Starting ChromeDriver 105.0.5195.52 (412c95e518836d8a7d97250d62b29c2ae6a26a85-refs/branch-heads/5195{apache#853}) on port 58567 Only local connections are allowed. Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe. ChromeDriver was started successfully. [info] - ajax rendered relative links are prefixed with uiRoot (spark.ui.proxyBase) (2 seconds, 416 milliseconds) [info] Run completed in 8 seconds, 936 milliseconds. [info] Total number of tests run: 1 [info] Suites: completed 1, aborted 0 [info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0 [info] All tests passed. [success] Total time: 30 s, completed 2022-9-14 20:11:34 ``` Closes apache#37868 from LuciferYang/SPARK-40397. Authored-by: yangjie01 <[email protected]> Signed-off-by: Kousuke Saruta <[email protected]>

LuciferYang added 4 commits September 13, 2022 16:11

for test

c67a3a1

upgrade okio

e6c149d

htmlunit-driver.version

44ba081

remove todo

d42308e

LuciferYang commented Sep 13, 2022

View reviewed changes

github-actions bot added BUILD CORE DSTREAM SQL labels Sep 13, 2022

address comments

8c13e52

LuciferYang commented Sep 14, 2022

View reviewed changes

sarutak approved these changes Sep 14, 2022

View reviewed changes

srowen reviewed Sep 14, 2022

View reviewed changes

srowen approved these changes Sep 14, 2022

View reviewed changes

sarutak closed this in 40590e6 Sep 15, 2022

	<dependency>
	<groupId>com.fasterxml.jackson.module</groupId>
	<artifactId>jackson-module-scala_${scala.binary.version}</artifactId>
	<version>${fasterxml.jackson.version}</version>
	<exclusions>
	<exclusion>
	<groupId>com.google.guava</groupId>
	<artifactId>guava</artifactId>
	</exclusion>
	</exclusions>
	</dependency>

[SPARK-40397][BUILD] Upgrade org.scalatestplus:selenium to 3.12.13 #37868

[SPARK-40397][BUILD] Upgrade org.scalatestplus:selenium to 3.12.13 #37868

Uh oh!

Conversation

LuciferYang commented Sep 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LuciferYang Sep 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Sep 14, 2022

Uh oh!

LuciferYang commented Sep 14, 2022

Uh oh!

LuciferYang Sep 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sarutak commented Sep 14, 2022

Uh oh!

LuciferYang commented Sep 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LuciferYang Sep 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sarutak commented Sep 15, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[SPARK-40397][BUILD] Upgrade `org.scalatestplus:selenium` to 3.12.13 #37868

[SPARK-40397][BUILD] Upgrade `org.scalatestplus:selenium` to 3.12.13 #37868

LuciferYang commented Sep 13, 2022 •

edited

Loading

LuciferYang Sep 14, 2022 •

edited

Loading

LuciferYang Sep 14, 2022 •

edited

Loading

LuciferYang commented Sep 14, 2022 •

edited

Loading

LuciferYang Sep 14, 2022 •

edited

Loading