[SPARK-24761][SQL] Adding of isModifiable() to RuntimeConfig #21730
Conversation
In the PR, I propose to extend `RuntimeConfig` by new method `isModifiable()` which returns `true` if a config parameter can be modified at runtime (in notebooks). For static SQL and core parameters, the method returns `false`.
python/pyspark/sql/conf.py
Outdated
| @ignore_unicode_prefix | ||
| @since(2.4) | ||
| def isModifiable(self, key): | ||
| """Is the configuration property modifiable or not.""" |
There was a problem hiding this comment.
Why not make all doc comments across languages identical?
There was a problem hiding this comment.
From which language would you prefer the comment? or maybe the comment should contain more information? Please, let me know if something is not clear, and we should explain that in the comment.
There was a problem hiding this comment.
I am going to change to:
Indicates that the configuration property with the given key modifiable in the current session or not.
There was a problem hiding this comment.
Minor language tweak suggestion:
Indicates whether the configuration property with the given key is modifiable in the current session.
|
Test build #92721 has finished for PR 21730 at commit
|
|
Test build #92722 has finished for PR 21730 at commit
|
|
|
||
| /** | ||
| * Indicates whether the configuration property with the given key | ||
| * is modifiable in the current session. |
There was a problem hiding this comment.
nit: If a key is not a valid conf, this function will still return false. Should we specify this in the comments?
There was a problem hiding this comment.
I added @return tag with this info.
| val conf = newConf() | ||
|
|
||
| // SQL configs | ||
| assert(!conf.isModifiable("spark.sql.sources.schemaStringLengthThreshold")) |
There was a problem hiding this comment.
nit: add a case for nonexistent sql conf key?
|
Test build #92886 has finished for PR 21730 at commit
|
5 participants
What changes were proposed in this pull request?
In the PR, I propose to extend
RuntimeConfigby new methodisModifiable()which returnstrueif a config parameter can be modified at runtime (for current session state). For static SQL and core parameters, the method returnsfalse.How was this patch tested?
Added new test to
RuntimeConfigSuitefor checking Spark core and SQL parameters.