[SPARK-35280][K8S] Promote KubernetesUtils to DeveloperApi #32406

dongjoon-hyun · 2021-04-30T06:16:43Z

What changes were proposed in this pull request?

Since SPARK-22757, KubernetesUtils has been used as an important utility class by all K8s modules and ExternalClusterManagers. This PR aims to promote KubernetesUtils to DeveloperApi in order to maintain it officially in a backward compatible way at Apache Spark 3.2.0.

Why are the changes needed?

Apache Spark 3.1.1 makes Kubernetes module GA and provides an extensible external cluster manager framework. To have ExternalClusterManager for K8s environment, KubernetesUtils class is crucial and needs to be stable. By promoting to a subset of K8s developer API, we can maintain these more sustainable way and give a better and stable functionality to K8s users.

In this PR, Since annotations denote the last function signature changes because these are going to become public at Apache Spark 3.2.0.

Version	Function Name
2.3.0	parsePrefixedKeyValuePairs
2.3.0	requireNandDefined
2.3.0	parsePrefixedKeyValuePairs
2.4.0	parseMasterUrl
3.0.0	requireBothOrNeitherDefined
3.0.0	requireSecondIfFirstIsDefined
3.0.0	selectSparkContainer
3.0.0	formatPairsBundle
3.0.0	formatPodState
3.0.0	containersDescription
3.0.0	containerStatusDescription
3.0.0	formatTime
3.0.0	uniqueID
3.0.0	buildResourcesQuantities
3.0.0	uploadAndTransformFileUris
3.0.0	uploadFileUri
3.0.0	requireBothOrNeitherDefined
3.0.0	buildPodWithServiceAccount
3.0.0	isLocalAndResolvable
3.1.1	renameMainAppResource
3.1.1	addOwnerReference
3.2.0	loadPodFromTemplate

Does this PR introduce any user-facing change?

Yes, but this is new API additions.

How was this patch tested?

Pass the CIs.

dongjoon-hyun · 2021-04-30T06:32:00Z

Hi, @attilapiros . Could you review this PR, please?

SparkQA · 2021-04-30T06:54:49Z

Test build #138101 has finished for PR 32406 at commit c405cab.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-30T07:40:06Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42621/

SparkQA · 2021-04-30T07:45:12Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42621/

dongjoon-hyun · 2021-04-30T15:26:22Z

Hi, @viirya . Could you review this?

attilapiros

I found these two issues.
After they are addressed LGTM.

attilapiros · 2021-04-30T16:35:41Z

...ce-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala

  }

+  @Since("3.2.0")
  def loadPodFromTemplate(


This was introduced by https://issues.apache.org/jira/browse/SPARK-24434.
Which went into Spark 3.0.0.

https://github.com/apache/spark/blob/branch-3.0/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala#L82

As I mentioned in the PR title, the signature is change recently (2fa792a), @attilapiros .
I know that because I changed it. :)

Please see the following PR description.

In this PR, Since annotations denote the last function signature changes because these are going to become public at Apache Spark 3.2.0.

attilapiros · 2021-04-30T16:48:05Z

...ce-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala

  }

+  @Since("3.0.0")
  def buildPodWithServiceAccount(serviceAccount: Option[String], pod: SparkPod): Option[Pod] = {


This was backported to 2.4.8:
https://github.com/apache/spark/blob/branch-2.4/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala#L66

It's not released yet, @attilapiros ! :)

2.4.8 RC3 will fail again. Please see this ancient bug fix.

[SPARK-35278][SQL] Invoke should find the method with correct number of parameters #32404 (comment)

viirya

Looks okay to make them as DeveloperApi.

dongjoon-hyun · 2021-04-30T18:33:55Z

Thank you, @viirya and @attilapiros .
Although some APIs are very stable since 2.3.0, this PR starts to annotate these APIs as @Unstable at Apache Spark 3.2.0. This is just a beginning to give a better dev experience in K8s area.

dongjoon-hyun · 2021-04-30T18:38:09Z

Thank you so much for your review, @viirya and @attilapiros !

dongjoon-hyun · 2021-04-30T18:38:21Z

Then, I'll merge this with 2 approvals. :)

Since SPARK-22757, `KubernetesUtils` has been used as an important utility class by all K8s modules and `ExternalClusterManager`s. This PR aims to promote `KubernetesUtils` to `DeveloperApi` in order to maintain it officially in a backward compatible way at Apache Spark 3.2.0. Apache Spark 3.1.1 makes `Kubernetes` module GA and provides an extensible external cluster manager framework. To have `ExternalClusterManager` for K8s environment, `KubernetesUtils` class is crucial and needs to be stable. By promoting to a subset of K8s developer API, we can maintain these more sustainable way and give a better and stable functionality to K8s users. In this PR, `Since` annotations denote the last function signature changes because these are going to become public at Apache Spark 3.2.0. | Version | Function Name | |-|-| | 2.3.0 | parsePrefixedKeyValuePairs | | 2.3.0 | requireNandDefined | | 2.3.0 | parsePrefixedKeyValuePairs | | 2.4.0 | parseMasterUrl | | 3.0.0 | requireBothOrNeitherDefined | | 3.0.0 | requireSecondIfFirstIsDefined | | 3.0.0 | selectSparkContainer | | 3.0.0 | formatPairsBundle | | 3.0.0 | formatPodState | | 3.0.0 | containersDescription | | 3.0.0 | containerStatusDescription | | 3.0.0 | formatTime | | 3.0.0 | uniqueID | | 3.0.0 | buildResourcesQuantities | | 3.0.0 | uploadAndTransformFileUris | | 3.0.0 | uploadFileUri | | 3.0.0 | requireBothOrNeitherDefined | | 3.0.0 | buildPodWithServiceAccount | | 3.0.0 | isLocalAndResolvable | | 3.1.1 | renameMainAppResource | | 3.1.1 | addOwnerReference | | 3.2.0 | loadPodFromTemplate | Yes, but this is new API additions. Pass the CIs. Closes apache#32406 from dongjoon-hyun/SPARK-35280. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…erApi` ### What changes were proposed in this pull request? This PR aims to promote `SparkKubernetesClientFactory` as **stable** `DeveloperApi` in order to maintain it officially in a backward compatible way at Apache Spark 4.0.0. ### Why are the changes needed? Like SPARK-35280 and SPARK-37497, `SparkKubernetesClientFactory` is also able to be used to develop new `ExternalClusterManager` for K8s environment. - #32406 - #34751 ### Does this PR introduce _any_ user-facing change? No. Previously, it was `private[spark]`. ### How was this patch tested? Manual review because this is only a visibility change. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #42709 from dongjoon-hyun/SPARK-44995. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…erApi` ### What changes were proposed in this pull request? This PR aims to promote `SparkKubernetesClientFactory` as **stable** `DeveloperApi` in order to maintain it officially in a backward compatible way at Apache Spark 4.0.0. ### Why are the changes needed? Like SPARK-35280 and SPARK-37497, `SparkKubernetesClientFactory` is also able to be used to develop new `ExternalClusterManager` for K8s environment. - apache#32406 - apache#34751 ### Does this PR introduce _any_ user-facing change? No. Previously, it was `private[spark]`. ### How was this patch tested? Manual review because this is only a visibility change. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#42709 from dongjoon-hyun/SPARK-44995. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit c596fce) Signed-off-by: Dongjoon Hyun <[email protected]>

[SPARK-35280][K8S] Promote KubernetesUtils to DeveloperApi

7e0e2ed

github-actions bot added the KUBERNETES label Apr 30, 2021

Use 3.1.1 instead of 3.1.0

c405cab

attilapiros reviewed Apr 30, 2021

View reviewed changes

viirya approved these changes Apr 30, 2021

View reviewed changes

attilapiros approved these changes Apr 30, 2021

View reviewed changes

dongjoon-hyun closed this in 4e8701a Apr 30, 2021

dongjoon-hyun deleted the SPARK-35280 branch April 30, 2021 18:39

dongjoon-hyun mentioned this pull request Aug 28, 2023

[SPARK-44995][K8S] Promote SparkKubernetesClientFactory to DeveloperApi #42709

Closed

[SPARK-35280][K8S] Promote KubernetesUtils to DeveloperApi #32406

[SPARK-35280][K8S] Promote KubernetesUtils to DeveloperApi #32406

Uh oh!

Conversation

dongjoon-hyun commented Apr 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

dongjoon-hyun commented Apr 30, 2021

Uh oh!

SparkQA commented Apr 30, 2021

Uh oh!

SparkQA commented Apr 30, 2021

Uh oh!

SparkQA commented Apr 30, 2021

Uh oh!

dongjoon-hyun commented Apr 30, 2021

Uh oh!

attilapiros left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

attilapiros Apr 30, 2021

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Apr 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Apr 30, 2021

Choose a reason for hiding this comment

Uh oh!

attilapiros Apr 30, 2021

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Apr 30, 2021

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Apr 30, 2021

Choose a reason for hiding this comment

Uh oh!

viirya left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Apr 30, 2021

Uh oh!

dongjoon-hyun commented Apr 30, 2021

Uh oh!

dongjoon-hyun commented Apr 30, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dongjoon-hyun commented Apr 30, 2021 •

edited

Loading

attilapiros left a comment •

edited

Loading

dongjoon-hyun Apr 30, 2021 •

edited

Loading