[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options #27459

nchammas · 2020-02-04T21:40:01Z

What changes were proposed in this pull request?

This PR adds a doc builder for Spark SQL's configuration options.

Here's what the new Spark SQL config docs look like (configuration.html.zip):

Compare this to the current docs:

Why are the changes needed?

There is no visibility into the various Spark SQL configs on the config docs page.

Does this PR introduce any user-facing change?

No, apart from new documentation.

How was this patch tested?

I tested this manually by building the docs and reviewing them in my browser.

SparkQA · 2020-02-04T21:49:27Z

Test build #117859 has finished for PR 27459 at commit 7742fc1.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

nchammas

Thanks for the reference to #18702, @HyukjinKwon. It was very helpful in getting me started.

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

sql/core/src/main/scala/org/apache/spark/sql/api/python/PythonSQLUtils.scala

sql/gen-sql-markdown.py

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

SparkQA · 2020-02-04T21:57:49Z

Test build #117860 has finished for PR 27459 at commit a3b6d19.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-02-04T22:05:39Z

Test build #117862 has finished for PR 27459 at commit b80e6f2.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-02-05T00:24:44Z

Test build #117864 has finished for PR 27459 at commit 914630f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

sql/gen-sql-markdown.py

HyukjinKwon · 2020-02-05T00:40:53Z

retest this please

gatorsmile

Could we have a test to ensure the internal SQLConf will not be added to the generated doc?

gatorsmile · 2020-02-05T01:05:55Z

@nchammas Good work! To make it easier to review, could you attach the generated doc in the PR description?

sql/gen-sql-markdown.py

nchammas · 2020-02-05T02:49:55Z

@gatorsmile

To make it easier to review, could you attach the generated doc in the PR description?

Done.

nchammas · 2020-02-05T03:11:59Z

Could we have a test to ensure the internal SQLConf will not be added to the generated doc?

Perhaps we should just add a test that confirms that one or two specific internal configs are not in the output of SQLConf.getAllDefinedConfs().

Or maybe, if the concern is strictly about the docs, the test should be against the newly added _list_sql_configs()?

SparkQA · 2020-02-05T03:22:25Z

Test build #117868 has finished for PR 27459 at commit 914630f.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon

Looks good otherwise.

sql/create-docs.sh

nchammas · 2020-02-07T17:14:48Z

I've updated the screenshot and attached HTML in the PR description to match the latest output.

SparkQA · 2020-02-07T19:28:39Z

Test build #118042 has finished for PR 27459 at commit 4c32cf2.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

nchammas · 2020-02-07T20:13:36Z

Retest this please.

SparkQA · 2020-02-07T23:15:32Z

Test build #118044 has finished for PR 27459 at commit 4c32cf2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-02-08T01:41:24Z

Test build #118048 has finished for PR 27459 at commit b08bac4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-02-09T10:20:32Z

Merged to master and branch-3.0.

Thanks for working on this, @nchammas

HyukjinKwon · 2020-02-09T10:21:21Z

cc @rxin, @cloud-fan, @gatorsmile, @dongjoon-hyun, @srowen (who I remember I talked about this). Now all external SQL configurations are documented automatically.

…tions ### What changes were proposed in this pull request? This PR adds a doc builder for Spark SQL's configuration options. Here's what the new Spark SQL config docs look like ([configuration.html.zip](https://github.com/apache/spark/files/4172109/configuration.html.zip)): ![Screen Shot 2020-02-07 at 12 13 23 PM](https://user-images.githubusercontent.com/1039369/74050007-425b5480-49a3-11ea-818c-42700c54d1fb.png) Compare this to the [current docs](http://spark.apache.org/docs/3.0.0-preview2/configuration.html#spark-sql): ![Screen Shot 2020-02-04 at 4 55 10 PM](https://user-images.githubusercontent.com/1039369/73790828-24a5a980-476f-11ea-998c-12cd613883e8.png) ### Why are the changes needed? There is no visibility into the various Spark SQL configs on [the config docs page](http://spark.apache.org/docs/3.0.0-preview2/configuration.html#spark-sql). ### Does this PR introduce any user-facing change? No, apart from new documentation. ### How was this patch tested? I tested this manually by building the docs and reviewing them in my browser. Closes #27459 from nchammas/SPARK-30510-spark-sql-options. Authored-by: Nicholas Chammas <[email protected]> Signed-off-by: HyukjinKwon <[email protected]> (cherry picked from commit 339c0f9) Signed-off-by: HyukjinKwon <[email protected]>

gatorsmile · 2020-02-14T18:31:47Z

If possible, we can add the version info in each SQLConf and add it into this doc too?

cc @beliefer Are you willing to do this?

beliefer · 2020-02-15T03:02:59Z

@gatorsmile Thanks for your call. I will take a look.

### What changes were proposed in this pull request? This PR makes the following refinements to the workflow for building docs: * Install Python and Ruby consistently using pyenv and rbenv across both the docs README and the release Dockerfile. * Pin the Python and Ruby versions we use. * Pin all direct Python and Ruby dependency versions. * Eliminate any use of `sudo pip`, which the Python community discourages, or `sudo gem`. ### Why are the changes needed? This PR should increase the consistency and reproducibility of the doc-building process by managing Python and Ruby in a more consistent way, and by eliminating unused or outdated code. Here's a possible example of an issue building the docs that would be addressed by the changes in this PR: #27459 (comment) ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Manual tests: * I was able to build the Docker image successfully, minus the final part about `RUN useradd`. * I am unable to run `do-release-docker.sh` because I am not a committer and don't have the required GPG key. * I built the docs locally and viewed them in the browser. I think I need a committer to more fully test out these changes. Closes #27534 from nchammas/SPARK-30731-building-docs. Authored-by: Nicholas Chammas <[email protected]> Signed-off-by: Sean Owen <[email protected]>

…tions ### What changes were proposed in this pull request? This PR adds a doc builder for Spark SQL's configuration options. Here's what the new Spark SQL config docs look like ([configuration.html.zip](https://github.com/apache/spark/files/4172109/configuration.html.zip)): ![Screen Shot 2020-02-07 at 12 13 23 PM](https://user-images.githubusercontent.com/1039369/74050007-425b5480-49a3-11ea-818c-42700c54d1fb.png) Compare this to the [current docs](http://spark.apache.org/docs/3.0.0-preview2/configuration.html#spark-sql): ![Screen Shot 2020-02-04 at 4 55 10 PM](https://user-images.githubusercontent.com/1039369/73790828-24a5a980-476f-11ea-998c-12cd613883e8.png) ### Why are the changes needed? There is no visibility into the various Spark SQL configs on [the config docs page](http://spark.apache.org/docs/3.0.0-preview2/configuration.html#spark-sql). ### Does this PR introduce any user-facing change? No, apart from new documentation. ### How was this patch tested? I tested this manually by building the docs and reviewing them in my browser. Closes apache#27459 from nchammas/SPARK-30510-spark-sql-options. Authored-by: Nicholas Chammas <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>

### What changes were proposed in this pull request? This PR makes the following refinements to the workflow for building docs: * Install Python and Ruby consistently using pyenv and rbenv across both the docs README and the release Dockerfile. * Pin the Python and Ruby versions we use. * Pin all direct Python and Ruby dependency versions. * Eliminate any use of `sudo pip`, which the Python community discourages, or `sudo gem`. ### Why are the changes needed? This PR should increase the consistency and reproducibility of the doc-building process by managing Python and Ruby in a more consistent way, and by eliminating unused or outdated code. Here's a possible example of an issue building the docs that would be addressed by the changes in this PR: apache#27459 (comment) ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Manual tests: * I was able to build the Docker image successfully, minus the final part about `RUN useradd`. * I am unable to run `do-release-docker.sh` because I am not a committer and don't have the required GPG key. * I built the docs locally and viewed them in the browser. I think I need a committer to more fully test out these changes. Closes apache#27534 from nchammas/SPARK-30731-building-docs. Authored-by: Nicholas Chammas <[email protected]> Signed-off-by: Sean Owen <[email protected]>

nchammas added 4 commits February 4, 2020 16:28

include sql configs from html file

f2bb001

generate html table of sql configs

f0137f6

example of follow-up that is needed to clean docstrings

a6045f7

back to a plain list

7742fc1

nchammas commented Feb 4, 2020

View reviewed changes

style

a3b6d19

nchammas commented Feb 4, 2020

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala Show resolved Hide resolved

nchammas requested a review from HyukjinKwon February 4, 2020 21:52

nchammas added 2 commits February 4, 2020 16:58

python style

886d01e

catch missed special default values

b80e6f2

fix reference

914630f

HyukjinKwon reviewed Feb 5, 2020

View reviewed changes

sql/gen-sql-markdown.py Outdated Show resolved Hide resolved

HyukjinKwon changed the title ~~[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options~~ [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options Feb 5, 2020

gatorsmile reviewed Feb 5, 2020

View reviewed changes

nchammas commented Feb 5, 2020

View reviewed changes

sql/gen-sql-markdown.py Outdated Show resolved Hide resolved

HyukjinKwon reviewed Feb 5, 2020

View reviewed changes

sql/gen-sql-markdown.py Outdated Show resolved Hide resolved

nchammas added 4 commits February 4, 2020 21:17

tweak create-docs info message

a8ad179

use markdown via mkdocs

e09f890

explain why .toArray

851d557

use markdown from within mkdocs

36fd916

fix config reference + undo ConfigEntry removal

6ca51cd

HyukjinKwon approved these changes Feb 7, 2020

View reviewed changes

sql/create-docs.sh Outdated Show resolved Hide resolved

nchammas added 5 commits February 7, 2020 10:47

tweak default formatting

e48eb34

remove check for private configs

3292108

tweak info message from create-docs.sh

617a69d

remove dup default

452bf98

stringify certain defaults so they display right

4c32cf2

300 -> 5 * 60

b08bac4

maropu mentioned this pull request Feb 9, 2020

[SPARK-30703][SQL][DOCS] Add a document for the ANSI mode #27489

Closed

HyukjinKwon closed this in 339c0f9 Feb 9, 2020

nchammas deleted the SPARK-30510-spark-sql-options branch February 10, 2020 01:27

nchammas mentioned this pull request Feb 10, 2020

[SPARK-30879][DOCS] Refine workflow for building docs #27534

Closed

maropu mentioned this pull request Feb 15, 2020

[SPARK-30703][SQL][DOCS][FollowUp] Declare the ANSI SQL compliance options as experimental #27590

Closed

This was referenced Apr 10, 2020

[SPARK-31369][SQL][DOCS] Documentation for JSON Functions #28170

Closed

[SPARK-31349][SQL][DOCS] Document built-in aggregate functions in SQL Reference #28120

Closed

nchammas mentioned this pull request Dec 11, 2023

[SPARK-46395][DOCS] Assign Spark configs to groups for use in documentation #44300

Closed

[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options #27459

[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options #27459

Uh oh!

Conversation

nchammas commented Feb 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Feb 4, 2020

Uh oh!

nchammas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Feb 4, 2020

Uh oh!

SparkQA commented Feb 4, 2020

Uh oh!

SparkQA commented Feb 5, 2020

Uh oh!

Uh oh!

HyukjinKwon commented Feb 5, 2020

Uh oh!

gatorsmile left a comment

Choose a reason for hiding this comment

Uh oh!

gatorsmile commented Feb 5, 2020

Uh oh!

Uh oh!

Uh oh!

nchammas commented Feb 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nchammas commented Feb 5, 2020

Uh oh!

SparkQA commented Feb 5, 2020

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nchammas commented Feb 7, 2020

Uh oh!

SparkQA commented Feb 7, 2020

Uh oh!

nchammas commented Feb 7, 2020

Uh oh!

SparkQA commented Feb 7, 2020

Uh oh!

SparkQA commented Feb 8, 2020

Uh oh!

HyukjinKwon commented Feb 9, 2020

Uh oh!

HyukjinKwon commented Feb 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gatorsmile commented Feb 14, 2020

Uh oh!

beliefer commented Feb 15, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

nchammas commented Feb 4, 2020 •

edited

Loading

nchammas commented Feb 5, 2020 •

edited

Loading

HyukjinKwon commented Feb 9, 2020 •

edited

Loading