[SPARK-31498][SQL][DOCS] Dump public static sql configurations through doc generation #28274

yaooqinn · 2020-04-20T11:41:27Z

What changes were proposed in this pull request?

Currently, only the non-static public SQL configurations are dump to public doc, we'd better also add those static public ones as the command set -v

This PR force call StaticSQLConf to buildStaticConf.

Why are the changes needed?

Fix missing SQL configurations in doc

Does this PR introduce any user-facing change?

NO

How was this patch tested?

add unit test and verify locally to see if public static SQL conf is in docs/sql-config.html

…generation

yaooqinn · 2020-04-20T11:43:49Z

cc @cloud-fan @dongjoon-hyun @maropu @HyukjinKwon thanks for review.

sql/core/src/main/scala/org/apache/spark/sql/api/python/PythonSQLUtils.scala

SparkQA · 2020-04-20T17:04:44Z

Test build #121527 has finished for PR 28274 at commit 9324ba9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-04-20T18:32:54Z

Test build #121539 has finished for PR 28274 at commit 991a7ec.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-04-21T05:58:20Z

Test build #121553 has finished for PR 28274 at commit 284ab70.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2020-04-21T06:32:41Z

Test build #121559 has finished for PR 28274 at commit b61b46c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-04-21T07:05:01Z

Test build #121562 has finished for PR 28274 at commit 4fca3fd.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

yaooqinn · 2020-04-21T07:07:26Z

retest this please

docs/configuration.md

cloud-fan · 2020-04-21T08:57:56Z

docs/configuration.md

+
+#### Static SQL Configuration
+
+Static SQL configurations are cross-session, immutable Spark SQL configurations. External users can query the static sql config values via `SparkSession.conf` or via set command, e.g. `SET spark.sql.extensions;`, but cannot set/unset them.


We should also mention how to set them, e.g. by config file or by setting SparkConf that is used to create SparkSession.

cloud-fan · 2020-04-21T08:59:59Z

sql/core/src/main/scala/org/apache/spark/sql/api/python/PythonSQLUtils.scala

+  def listStaticSQLConfigs(): Array[(String, String, String, String)] = {
+    val conf = new SQLConf()
+    // Force to build static SQL configurations
+    StaticSQLConf.WAREHOUSE_PATH.key -> ()


what's special about WAREHOUSE_PATH?

Nothing special, just find something to invoke? Couldn't find another proper way to build static ones to SQLConf, any advice?

just write StaticSQLConf doesn't work?

thanks, it works. there must be some wrong with my environment when I try this way last time.

SparkQA · 2020-04-21T09:55:54Z

Test build #121572 has finished for PR 28274 at commit 4fca3fd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-04-21T11:34:31Z

Test build #121580 has finished for PR 28274 at commit 1de696e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

yaooqinn · 2020-04-21T11:36:18Z

retest this please

SparkQA · 2020-04-21T14:30:58Z

Test build #121582 has finished for PR 28274 at commit 81e043f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-04-21T15:09:46Z

Test build #121583 has finished for PR 28274 at commit 81e043f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-04-22T10:16:37Z

thanks, merging to master/3.0!

…h doc generation ### What changes were proposed in this pull request? Currently, only the non-static public SQL configurations are dump to public doc, we'd better also add those static public ones as the command `set -v` This PR force call StaticSQLConf to buildStaticConf. ### Why are the changes needed? Fix missing SQL configurations in doc ### Does this PR introduce any user-facing change? NO ### How was this patch tested? add unit test and verify locally to see if public static SQL conf is in `docs/sql-config.html` Closes #28274 from yaooqinn/SPARK-31498. Authored-by: Kent Yao <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit 2c2062e) Signed-off-by: Wenchen Fan <[email protected]>

HyukjinKwon · 2020-04-22T11:17:08Z

sql/gen-sql-config-docs.py


 if __name__ == "__main__":
+    if len(sys.argv) != 2:
+        print("Usage: ./bin/spark-submit sql/gen-sql-config-docs.py <static|runtime>")


You could just call the function twice instead of adding an argument considering that SQL configuration gen is pretty cheap. Let's fix it next time when we happen to touch here.

HyukjinKwon · 2020-04-22T11:20:22Z

docs/configuration.md

+
+{% for static_file in site.static_files %}
+    {% if static_file.name == 'generated-runtime-sql-config-table.html' %}
+        {% include_relative generated-runtime-sql-config-table.html %}


I intentionally removed the leading spaces here because they are considered actual white spaces. Newer liquid syntax supports to ignore these white spaces but I didn't use in case old Jykill is used.

Seems like it could make the html format malformed in some cases given my rough testing. Let's remove these leading white spaces next time.

Might the problem be that ### Spark SQL content was inside the for-loop before?

No, we already use the markdown and Liquid syntax together at https://github.com/apache/spark/blob/master/docs/sql-ref-functions-builtin.md

HyukjinKwon · 2020-04-22T11:21:28Z

docs/configuration.md

+#### Static SQL Configuration
+
+Static SQL configurations are cross-session, immutable Spark SQL configurations. They can be set with final values by the config file
+and command-line options with `--conf/-c` prefixed, or by setting `SparkConf` that are used to create `SparkSession`.


You could mention spark-default.conf too, but it's okay.

by the config file might have the same meaning but it could be better if we use spark-default.conf

…eral meanings in sql configuration doc ### What changes were proposed in this pull request? ```scala spark.sql.session.timeZone spark.sql.warehouse.dir ``` these 2 configs are nondeterministic and vary with environments Besides, reflect code in `gen-sql-config-docs.py` via #28274 (comment) and `configuration.md` via #28274 (comment) ### Why are the changes needed? doc fix ### Does this PR introduce any user-facing change? no ### How was this patch tested? verify locally ![image](https://user-images.githubusercontent.com/8326978/80179099-5e7da200-8632-11ea-803f-d47a93151869.png) Closes #28322 from yaooqinn/SPARK-31550. Authored-by: Kent Yao <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>

…eral meanings in sql configuration doc ### What changes were proposed in this pull request? ```scala spark.sql.session.timeZone spark.sql.warehouse.dir ``` these 2 configs are nondeterministic and vary with environments Besides, reflect code in `gen-sql-config-docs.py` via #28274 (comment) and `configuration.md` via #28274 (comment) ### Why are the changes needed? doc fix ### Does this PR introduce any user-facing change? no ### How was this patch tested? verify locally ![image](https://user-images.githubusercontent.com/8326978/80179099-5e7da200-8632-11ea-803f-d47a93151869.png) Closes #28322 from yaooqinn/SPARK-31550. Authored-by: Kent Yao <[email protected]> Signed-off-by: HyukjinKwon <[email protected]> (cherry picked from commit 5ba467c) Signed-off-by: HyukjinKwon <[email protected]>

[SPARK-31498][SQL] Dump public static sql configurations through doc …

9324ba9

…generation

probot-autolabeler bot added PYTHON SQL labels Apr 20, 2020

yaooqinn changed the title ~~[SPARK-31498][SQL] Dump public static sql configurations through doc generation~~ [SPARK-31498][SQL][DOCS] Dump public static sql configurations through doc generation Apr 20, 2020

HyukjinKwon reviewed Apr 20, 2020

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/api/python/PythonSQLUtils.scala Outdated Show resolved Hide resolved

separate sql confs

991a7ec

probot-autolabeler bot added DOCS INFRA labels Apr 20, 2020

yaooqinn added 4 commits April 21, 2020 09:53

fix test

284ab70

Merge branch 'master' into SPARK-31498

9621155

nit

b61b46c

fix

4fca3fd

cloud-fan reviewed Apr 21, 2020

View reviewed changes

docs/configuration.md Outdated Show resolved Hide resolved

cloud-fan reviewed Apr 21, 2020

View reviewed changes

yaooqinn added 2 commits April 21, 2020 17:22

doc

5a42b0d

doc

1de696e

simplify

81e043f

yaooqinn requested review from HyukjinKwon, cloud-fan and maropu April 22, 2020 01:47

cloud-fan closed this in 2c2062e Apr 22, 2020

HyukjinKwon reviewed Apr 22, 2020

View reviewed changes

yaooqinn deleted the SPARK-31498 branch April 22, 2020 11:18

HyukjinKwon reviewed Apr 22, 2020

View reviewed changes

maropu mentioned this pull request Apr 25, 2020

[SPARK-31550][SQL][DOCS] Set nondeterministic configurations with general meanings in sql configuration doc #28322

Closed

This was referenced Dec 11, 2023

[SPARK-46395][DOCS] Assign Spark configs to groups for use in documentation #44300

Closed

[SPARK-46357] Replace incorrect documentation use of setConf with conf.set #44290

Closed


		#### Static SQL Configuration

		Static SQL configurations are cross-session, immutable Spark SQL configurations. External users can query the static sql config values via `SparkSession.conf` or via set command, e.g. `SET spark.sql.extensions;`, but cannot set/unset them.

[SPARK-31498][SQL][DOCS] Dump public static sql configurations through doc generation #28274

[SPARK-31498][SQL][DOCS] Dump public static sql configurations through doc generation #28274

Uh oh!

Conversation

yaooqinn commented Apr 20, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

yaooqinn commented Apr 20, 2020

Uh oh!

Uh oh!

SparkQA commented Apr 20, 2020

Uh oh!

SparkQA commented Apr 20, 2020

Uh oh!

SparkQA commented Apr 21, 2020

Uh oh!

SparkQA commented Apr 21, 2020

Uh oh!

SparkQA commented Apr 21, 2020

Uh oh!

yaooqinn commented Apr 21, 2020

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 21, 2020

Uh oh!

SparkQA commented Apr 21, 2020

Uh oh!

yaooqinn commented Apr 21, 2020

Uh oh!

SparkQA commented Apr 21, 2020

Uh oh!

SparkQA commented Apr 21, 2020

Uh oh!

cloud-fan commented Apr 22, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yaooqinn Apr 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

yaooqinn Apr 22, 2020 •

edited

Loading