Skip to content

Commit d9e14f2

Browse files
zwangshengpan3793
zwangsheng
and
pan3793
committed
[KYUUBI #4623][K8S] KubernetesApplicationOperation uses Informer instead of list
### _Why are the changes needed?_ Close #4623 To reduce the pressure on the Api Server (which use the kubernetes client polls with label to find the `spark driver pod` when multiple Rest Application are running at the same time), use informer, the kubernetes-recommended method of maintaining the application state. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request - [x] Run CI Closes #4625 from zwangsheng/KYUUBI_4623. Closes #4623 a415bef [Cheng Pan] nit 136d0db [Cheng Pan] 171 b5d3c23 [Cheng Pan] re-generate conf bf14ad8 [Cheng Pan] nit 9ee7e04 [Cheng Pan] nit 301162e [Cheng Pan] nit 1d42692 [Cheng Pan] nit b95d7a6 [Cheng Pan] improve cc8d2c7 [zwangsheng] fix comments d017baf [zwangsheng] Set resycn 0 28f9a70 [zwangsheng] Reorder func & slow get app info 22d9c16 [zwangsheng] fix setting 8e09403 [zwangsheng] fix comments 10965d3 [zwangsheng] Rename fileter function => isSparkEnginePod b026771 [zwangsheng] rename 78c9fdb [zwangsheng] fix comments 6d31f70 [zwangsheng] Fix IT Test f43bba2 [zwangsheng] fix 17e4f55 [zwangsheng] debug be8da79 [zwangsheng] debug 0db45a5 [zwangsheng] retest a93786a [zwangsheng] Fix style 652ee83 [zwangsheng] Add Setting & Debug 4add7e4 [zwangsheng] improve 1f43412 [zwangsheng] remove unused import 35acd61 [zwangsheng] fix compile 05dfc59 [zwangsheng] [KYUUBI #4623][Improvement][K8S] Remove cached app info when out of time 4ab530e [zwangsheng] [KYUUBI #4623][Improvement][K8S] kubernetesApplicationOperation Using Informer instead of list Lead-authored-by: zwangsheng <[email protected]> Co-authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
1 parent 92f191a commit d9e14f2

File tree

5 files changed

+160
-88
lines changed

5 files changed

+160
-88
lines changed

‎docs/deployment/settings.md‎

Lines changed: 13 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -169,7 +169,7 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
169169
| kyuubi.engine.spark.python.env.archive | &lt;undefined&gt; | Portable Python env archive used for Spark engine Python language mode. | string | 1.7.0 |
170170
| kyuubi.engine.spark.python.env.archive.exec.path | bin/python | The Python exec path under the Python env archive. | string | 1.7.0 |
171171
| kyuubi.engine.spark.python.home.archive | &lt;undefined&gt; | Spark archive containing $SPARK_HOME/python directory, which is used to init session Python worker for Python language mode. | string | 1.7.0 |
172-
| kyuubi.engine.submit.timeout | PT30S | Period to tolerant Driver Pod ephemerally invisible after submitting. In some Resource Managers, e.g. K8s, the Driver Pod is not invisible immediately after `spark-submit` is returned. | duration | 1.7.1 |
172+
| kyuubi.engine.submit.timeout | PT30S | Period to tolerant Driver Pod ephemerally invisible after submitting. In some Resource Managers, e.g. K8s, the Driver Pod is not visible immediately after `spark-submit` is returned. | duration | 1.7.1 |
173173
| kyuubi.engine.trino.event.loggers | JSON | A comma-separated list of engine history loggers, where engine/session/operation etc events go.<ul> <li>JSON: the events will be written to the location of kyuubi.engine.event.json.log.path</li> <li>JDBC: to be done</li> <li>CUSTOM: to be done.</li></ul> | seq | 1.7.0 |
174174
| kyuubi.engine.trino.extra.classpath | &lt;undefined&gt; | The extra classpath for the Trino query engine, for configuring other libs which may need by the Trino engine | string | 1.6.0 |
175175
| kyuubi.engine.trino.java.options | &lt;undefined&gt; | The extra Java options for the Trino query engine | string | 1.6.0 |
@@ -292,17 +292,18 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
292292

293293
### Kubernetes
294294

295-
| Key | Default | Meaning | Type | Since |
296-
|-----------------------------------------------|-------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|-------|
297-
| kyuubi.kubernetes.authenticate.caCertFile | &lt;undefined&gt; | Path to the CA cert file for connecting to the Kubernetes API server over TLS from the kyuubi. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
298-
| kyuubi.kubernetes.authenticate.clientCertFile | &lt;undefined&gt; | Path to the client cert file for connecting to the Kubernetes API server over TLS from the kyuubi. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
299-
| kyuubi.kubernetes.authenticate.clientKeyFile | &lt;undefined&gt; | Path to the client key file for connecting to the Kubernetes API server over TLS from the kyuubi. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
300-
| kyuubi.kubernetes.authenticate.oauthToken | &lt;undefined&gt; | The OAuth token to use when authenticating against the Kubernetes API server. Note that unlike, the other authentication options, this must be the exact string value of the token to use for the authentication. | string | 1.7.0 |
301-
| kyuubi.kubernetes.authenticate.oauthTokenFile | &lt;undefined&gt; | Path to the file containing the OAuth token to use when authenticating against the Kubernetes API server. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
302-
| kyuubi.kubernetes.context | &lt;undefined&gt; | The desired context from your kubernetes config file used to configure the K8s client for interacting with the cluster. | string | 1.6.0 |
303-
| kyuubi.kubernetes.master.address | &lt;undefined&gt; | The internal Kubernetes master (API server) address to be used for kyuubi. | string | 1.7.0 |
304-
| kyuubi.kubernetes.namespace | default | The namespace that will be used for running the kyuubi pods and find engines. | string | 1.7.0 |
305-
| kyuubi.kubernetes.trust.certificates | false | If set to true then client can submit to kubernetes cluster only with token | boolean | 1.7.0 |
295+
| Key | Default | Meaning | Type | Since |
296+
|-----------------------------------------------------|-------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
297+
| kyuubi.kubernetes.authenticate.caCertFile | &lt;undefined&gt; | Path to the CA cert file for connecting to the Kubernetes API server over TLS from the kyuubi. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
298+
| kyuubi.kubernetes.authenticate.clientCertFile | &lt;undefined&gt; | Path to the client cert file for connecting to the Kubernetes API server over TLS from the kyuubi. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
299+
| kyuubi.kubernetes.authenticate.clientKeyFile | &lt;undefined&gt; | Path to the client key file for connecting to the Kubernetes API server over TLS from the kyuubi. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
300+
| kyuubi.kubernetes.authenticate.oauthToken | &lt;undefined&gt; | The OAuth token to use when authenticating against the Kubernetes API server. Note that unlike, the other authentication options, this must be the exact string value of the token to use for the authentication. | string | 1.7.0 |
301+
| kyuubi.kubernetes.authenticate.oauthTokenFile | &lt;undefined&gt; | Path to the file containing the OAuth token to use when authenticating against the Kubernetes API server. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
302+
| kyuubi.kubernetes.context | &lt;undefined&gt; | The desired context from your kubernetes config file used to configure the K8s client for interacting with the cluster. | string | 1.6.0 |
303+
| kyuubi.kubernetes.master.address | &lt;undefined&gt; | The internal Kubernetes master (API server) address to be used for kyuubi. | string | 1.7.0 |
304+
| kyuubi.kubernetes.namespace | default | The namespace that will be used for running the kyuubi pods and find engines. | string | 1.7.0 |
305+
| kyuubi.kubernetes.terminatedApplicationRetainPeriod | PT5M | The period for which the Kyuubi server retains application information after the application terminates. | duration | 1.7.1 |
306+
| kyuubi.kubernetes.trust.certificates | false | If set to true then client can submit to kubernetes cluster only with token | boolean | 1.7.0 |
306307

307308
### Metadata
308309

‎integration-tests/kyuubi-kubernetes-it/src/test/scala/org/apache/kyuubi/kubernetes/test/spark/SparkOnKubernetesTestsSuite.scala‎

Lines changed: 8 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -208,19 +208,18 @@ class KyuubiOperationKubernetesClusterClusterModeSuite
208208
batchRequest.getConf.asScala.toMap,
209209
batchRequest)
210210

211-
val session = sessionManager.getSession(sessionHandle).asInstanceOf[KyuubiBatchSessionImpl]
212-
val batchJobSubmissionOp = session.batchJobSubmissionOp
213-
214-
eventually(timeout(3.minutes), interval(50.milliseconds)) {
215-
val appInfo = batchJobSubmissionOp.getOrFetchCurrentApplicationInfo
216-
assert(appInfo.nonEmpty)
217-
assert(appInfo.exists(_.state == RUNNING))
218-
assert(appInfo.exists(_.name.startsWith(driverPodNamePrefix)))
211+
// wait for driver pod start
212+
eventually(timeout(3.minutes), interval(5.second)) {
213+
// trigger k8sOperation init here
214+
val appInfo = k8sOperation.getApplicationInfoByTag(sessionHandle.identifier.toString)
215+
assert(appInfo.state == RUNNING)
216+
assert(appInfo.name.startsWith(driverPodNamePrefix))
219217
}
220218

221219
val killResponse = k8sOperation.killApplicationByTag(sessionHandle.identifier.toString)
222220
assert(killResponse._1)
223-
assert(killResponse._2 startsWith "Operation of deleted appId:")
221+
assert(killResponse._2 endsWith "is completed")
222+
assert(killResponse._2 contains sessionHandle.identifier.toString)
224223

225224
eventually(timeout(3.minutes), interval(50.milliseconds)) {
226225
val appInfo = k8sOperation.getApplicationInfoByTag(sessionHandle.identifier.toString)

‎kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1171,6 +1171,15 @@ object KyuubiConf {
11711171
.booleanConf
11721172
.createWithDefault(false)
11731173

1174+
val KUBERNETES_TERMINATED_APPLICATION_RETAIN_PERIOD: ConfigEntry[Long] =
1175+
buildConf("kyuubi.kubernetes.terminatedApplicationRetainPeriod")
1176+
.doc("The period for which the Kyuubi server retains application information after " +
1177+
"the application terminates.")
1178+
.version("1.7.1")
1179+
.timeConf
1180+
.checkValue(_ > 0, "must be positive number")
1181+
.createWithDefault(Duration.ofMinutes(5).toMillis)
1182+
11741183
// ///////////////////////////////////////////////////////////////////////////////////////////////
11751184
// SQL Engine Configuration //
11761185
// ///////////////////////////////////////////////////////////////////////////////////////////////
@@ -2553,7 +2562,7 @@ object KyuubiConf {
25532562
val ENGINE_SUBMIT_TIMEOUT: ConfigEntry[Long] =
25542563
buildConf("kyuubi.engine.submit.timeout")
25552564
.doc("Period to tolerant Driver Pod ephemerally invisible after submitting. " +
2556-
"In some Resource Managers, e.g. K8s, the Driver Pod is not invisible immediately " +
2565+
"In some Resource Managers, e.g. K8s, the Driver Pod is not visible immediately " +
25572566
"after `spark-submit` is returned.")
25582567
.version("1.7.1")
25592568
.timeConf

‎kyuubi-server/src/main/scala/org/apache/kyuubi/engine/ApplicationOperation.scala‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,11 @@ case class ApplicationInfo(
100100
}
101101
}
102102

103+
object ApplicationInfo {
104+
val NOT_FOUND: ApplicationInfo = ApplicationInfo(null, null, ApplicationState.NOT_FOUND)
105+
val UNKNOWN: ApplicationInfo = ApplicationInfo(null, null, ApplicationState.UNKNOWN)
106+
}
107+
103108
object ApplicationOperation {
104109
val NOT_FOUND = "APPLICATION_NOT_FOUND"
105110
}

0 commit comments

Comments
 (0)