[DCOS-54813] Base tech update from 2.4.0 to 2.4.3

[alembiewski]

What changes were proposed in this pull request?

Spark version updated to 2.4.3

Cherry-picked commits:

• [DCOS-34549] Mesos label NPE fix ([DCOS-34549] Mesos label NPE fix #60)

• Refactoring of metrics naming to add mesos semantics and avoid clashing with existing Spark metrics (Improved naming for Mesos metrics and metric sources #58)

• [DCOS-34235] spark.mesos.executor.memoryOverhead equivalent for the Driver when running on Mesos ([DCOS-34235] spark.mesos.executor.memoryOverhead equivalent for the Driver when running on Mesos #55)

• [DCOS-51453] Added Hadoop 2.9 profile ([DCOS-51453][HADOOP] Add Hadoop 2.9 profile #53)

• [DCOS-51454] Remove irrelevant Mesos REPL test ([DCOS-51454] Remove irrelevant Mesos REPL test #54)

• [DCOS-51158] Improved Task ID assignment for Executor tasks ([DCOS-51158] Improved Task ID assignment for Executor tasks #52)

• [DCOS-40974] Mesos checkpointing support for Spark Drivers ([DCOS-40974] Mesos checkpointing support for Spark Drivers #51)

• [DCOS-49020] Specify user in CommandInfo for Spark Driver launched on Mesos ([DCOS-49020] Specify user in CommandInfo for Spark Driver launched on Mesos #49)

• Revert "[SPARK-25088][CORE][MESOS][DOCS] Update Rest Server docs & defaults."

• [DCOS-46585] Fix supervised driver retry logic for outdated tasks ([DCOS-46585] Fix supervised driver retry logic for outdated tasks #46)

• Spark Dispatcher support for launching applications in the same virtual network by default ([DCOS-46369] Spark Dispatcher should run Spark Drivers and Executors in the same virtual network if specified #45)

• CNI Support for Docker containerizer, binding to SPARK_LOCAL_IP instead of 0.0.0.0 to properly advertise executors during shuffle ([DCOS-45840] CNI support for shuffle jobs #44)

• Upgrade of Hadoop, ZooKeeper, and Jackson libraries to fix CVEs. Updates for JSON-related tests. ([DCOS-45820] Upgrade of Hadoop, ZooKeeper, and Jackson libraries to fix CVEs #43)

• [DCOS-39150][SPARK] Support unique Executor IDs in cluster managers ([DCOS-39150][SPARK] Support unique Executor IDs in cluster managers #36)

• [SPARK-23941][MESOS] Mesos task failed on specific spark app name ([SPARK-23941][MESOS] Mesos task failed on specific spark app name #33)

• [SPARK-692] Added spark.mesos.executor.gpus to specify the number of Executor CPUs

• [SPARK-723][SPARK-740] Add Metrics to Dispatcher and Driver

• File Based Secrets support

• Support for DSCOS_SERVICE_ACCOUNT_CREDENTIAL environment variable in MesosClusterScheduler

How was this patch tested?

integration tests;
unit test (spark-243-maven-test-log.log)

[samvantran]

Nice work @alembiewski! Thanks for adding the logs for unit tests and I can see all the ITs passed too. You've added the more recent commits to the master list of cherry picks to include for every bump?

@akirillov we should start that process for upstreaming as it looks like the number is starting to increase in our fork

[alembiewski]

Thanks @samvantran! Yes, the list in the PR's description includes all the recent commits.

master list of cherry picks to include for every bump

Just to check, do we have a special docs, like wiki page, that should be updated as well?

[akirillov]

Thanks, @alembiewski!

Although the fact that both unit and integration tests have passed serves as good evidence that everything works, it's really hard to provide a review for 270 commits and 400+ changed files. What I'd suggest: can you please try creating a copy of the upstream 2.4.3 release branch, name it custom-branch-2.4.3 (no x) and create a PR against it so that the PR contains only cherry-picks from Mesosphere?

This way the review is possible as we apply only Mesosphere changes on top of the upstream release (in fact this is the same case here but there's a chance we could have missed something because the actual review is so complicated). As all the merges and fixes have already taken place in the scope of this PR, this shouldn't be a huge problem.

WDYT?
cc @samvantran

[akirillov]

Thanks for updating the PR @alembiewski. Cherry-picked commits and the PR overall LGTM. I re-ran tests and CI looks good, however, unit tests fail on the same test (ran the full suite twice):

KafkaSourceStressForDontFailOnDataLossSuite:
- stress test for failOnDataLoss=false *** FAILED ***
  org.apache.spark.sql.streaming.StreamingQueryException: Query [id = 8a8114de-e47a-453e-954e-71056152e4db, runId = 94c165ac-8ba9-49ad-bb90-09d3869bcda4] terminated with exception: Timeout of 3000ms expired before the position for partition failOnDataLoss-3-0 could be determined
  at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:297)
  at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:193)
  ...
  Cause: org.apache.kafka.common.errors.TimeoutException: Timeout of 3000ms expired before the position for partition failOnDataLoss-3-0 could be determined

Please find a build log attached: spark_mvn_build.log

[alembiewski]

Hmm, I would say, that it's quite strange, I ran them today on a private AWS instance, following the instructions from our wiki, and they seem to pass. The box's description below:

Instance type: m5.xlarge
Availability zone: us-west-2a
AMI ID: ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20190722.1 (ami-06f2f779464715dc5)

Built from the proper commit:

ubuntu@ip-172-31-18-40:~/spark$ git rev-parse HEAD
b5a9366f693e8c2ef363680948c06c43cb423950

Build log: mvn.log

Based on exception type, could it be related to some network issues?

[akirillov]

After rerunning the tests in a different AWS region and instance type, everything looks good!

Build log attached: spark-build-mvn.log