Skip to content

[SPARK-27544][PYTHON][TEST][BRANCH-2.4] Fix Python test script to work on Scala-2.12 build #24439

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

[SPARK-27544][PYTHON][TEST][BRANCH-2.4] Fix Python test script to work on Scala-2.12 build #24439

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions python/run-tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,7 @@ def print_red(text):
LOGGER = logging.getLogger()

# Find out where the assembly jars are located.
# Later, add back 2.12 to this list:
# for scala in ["2.11", "2.12"]:
for scala in ["2.11"]:
for scala in ["2.11", "2.12"]:
build_dir = os.path.join(SPARK_HOME, "assembly", "target", "scala-" + scala)
if os.path.isdir(build_dir):
SPARK_DIST_CLASSPATH = os.path.join(build_dir, "jars", "*")
Copy link
Member

@felixcheung felixcheung Apr 23, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't this going to add both scala jar 2.11 and 2.12 into class path at the same time?

Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Apr 23, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for review, @felixcheung . We assume that the Spark developers don't build both profiles without cleanup. This is a standard Spark way used since Spark 2.0.0.

Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Apr 23, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, as we see here, although we officially deprecated Scala-2.11 at Spark 2.4.1, it seems that we don't test Scala-2.12 actually during Spark 2.4.2 vote process. The community is still on Scala-2.11. It's time to switch branch-2.4 to Scala-2.12 if possible.

Copy link
Member

@dbtsai dbtsai Apr 23, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 on this. For next 2.4.3, we should have branch-2.4 default to Scala 2.12 as we will remove Scala 2.11 support in Spark 3.0

Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Apr 23, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @dbtsai . +1 for that

Expand Down