[SPARK-40142][PYTHON][SQL][FOLLOW-UP] Make pyspark.sql.functions examples self-contained (part 2, 32 functions) #37592

khalidmammadov · 2022-08-20T21:21:32Z

What changes were proposed in this pull request?

Docstring improvements

Why are the changes needed?

To help users to understand pyspark API

Does this PR introduce any user-facing change?

Yes, documentation

How was this patch tested?

bundle exec jekyll serve --host 0.0.0.0

python/pyspark/sql/functions.py

HyukjinKwon · 2022-08-21T00:52:17Z

Thanks for doing this, @khalidmammadov

AmplabJenkins · 2022-08-21T04:00:34Z

Can one of the admins verify this patch?

khalidmammadov · 2022-08-21T21:39:02Z

Thanks for doing this, @khalidmammadov

my pleasure

HyukjinKwon · 2022-08-22T02:17:27Z

python/pyspark/sql/functions.py

+    --------
+    >>> df = spark.range(1)
+    >>> df.select(expm1(lit(1))).first()  # doctest: +ELLIPSIS
+    Row(EXPM1(1)=1.71828...)


This is a nice trick. In this way, the test won't be flaky even when the precision changes. If you use show(), it might break because of dashes (e.g., if 1.321 become 1.32). BTW you can remove # doctest: +ELLIPSIS since that's defined in the bottom of this file.

HyukjinKwon · 2022-08-22T02:18:47Z

python/pyspark/sql/functions.py

+    +------------------+
+    |       var_pop(id)|
+    +------------------+
+    |2.9166666666666665|


I think we should probably do ELLIPSIS (with the trick you did w/ Row(...)) since float representation is flaky in Python (and JDK 11/17 IIRC)

HyukjinKwon · 2022-08-22T02:18:52Z

python/pyspark/sql/functions.py

+    +------------------+
+    |       skewness(c)|
+    +------------------+
+    |0.7071067811865475|


HyukjinKwon · 2022-08-22T02:19:12Z

python/pyspark/sql/functions.py

+    +-----------------+
+    |   stddev_pop(id)|
+    +-----------------+
+    |1.707825127659933|


HyukjinKwon · 2022-08-22T02:19:18Z

python/pyspark/sql/functions.py

+    +------------------+
+    |   stddev_samp(id)|
+    +------------------+
+    |1.8708286933869707|


HyukjinKwon · 2022-08-22T02:19:22Z

python/pyspark/sql/functions.py

+    +------------------+
+    |   stddev_samp(id)|
+    +------------------+
+    |1.8708286933869707|


HyukjinKwon

LGTM otherwise

python/pyspark/sql/functions.py

HyukjinKwon · 2022-08-23T00:37:22Z

Merged to master.

HyukjinKwon · 2022-08-23T00:37:32Z

Thanks for working on this @khalidmammadov

khalidmammadov · 2022-08-23T06:09:33Z

Thanks for working on this @khalidmammadov

No worries, I will pick up next batch then from functions

…ples self-contained (part 5, ~28 functions) ### What changes were proposed in this pull request? It's part of the Pyspark docstrings improvement series (#37592, #37662, #37686) In this PR I mainly covered missing parts in the docstrings adding some more examples where it needed. ### Why are the changes needed? To improve PySpark documentation ### Does this PR introduce _any_ user-facing change? Yes, documentation ### How was this patch tested? PYTHON_EXECUTABLE=python3.9 ./dev/lint-python ./python/run-tests --testnames pyspark.sql.functions Closes #37786 from khalidmammadov/docstrings_funcs_part_5. Authored-by: Khalid Mammadov <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

…ples self-contained (part 6, ~50 functions) ### What changes were proposed in this pull request? It's part of the Pyspark docstrings improvement series (#37592, #37662, #37686, #37786) In this PR I mainly covered missing parts in the docstrings adding some more examples where it needed. ### Why are the changes needed? To improve PySpark documentation ### Does this PR introduce _any_ user-facing change? Yes, documentation ### How was this patch tested? ``` PYTHON_EXECUTABLE=python3.9 ./dev/lint-python ./python/run-tests --testnames pyspark.sql.functions bundle exec jekyll build ``` Closes #37797 from khalidmammadov/docstrings_funcs_part_6. Authored-by: Khalid Mammadov <[email protected]> Signed-off-by: Sean Owen <[email protected]>

…ples self-contained (part 7, ~30 functions) ### What changes were proposed in this pull request? It's part of the Pyspark docstrings improvement series (#37592, #37662, #37686, #37786, #37797) In this PR I mainly covered missing parts in the docstrings adding some more examples where it needed. ### Why are the changes needed? To improve PySpark documentation ### Does this PR introduce _any_ user-facing change? Yes, documentation ### How was this patch tested? ``` PYTHON_EXECUTABLE=python3.9 ./dev/lint-python ./python/run-tests --testnames pyspark.sql.functions bundle exec jekyll build ``` Closes #37850 from khalidmammadov/docstrings_funcs_part_7. Lead-authored-by: Khalid Mammadov <[email protected]> Co-authored-by: khalidmammadov <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

…ples self-contained (part 7, ~30 functions) ### What changes were proposed in this pull request? It's part of the Pyspark docstrings improvement series (apache#37592, apache#37662, apache#37686, apache#37786, apache#37797) In this PR I mainly covered missing parts in the docstrings adding some more examples where it needed. ### Why are the changes needed? To improve PySpark documentation ### Does this PR introduce _any_ user-facing change? Yes, documentation ### How was this patch tested? ``` PYTHON_EXECUTABLE=python3.9 ./dev/lint-python ./python/run-tests --testnames pyspark.sql.functions bundle exec jekyll build ``` Closes apache#37850 from khalidmammadov/docstrings_funcs_part_7. Lead-authored-by: Khalid Mammadov <[email protected]> Co-authored-by: khalidmammadov <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

…ples self-contained (FINAL) ### What changes were proposed in this pull request? It's part of the Pyspark docstrings improvement series (#37592, #37662, #37686, #37786, #37797, #37850) In this PR I mainly covered missing parts in the docstrings adding some more examples where it needed. I have also made all examples self explanatory by providing DataFrame creation command where it was missing for clarity to a user. This should complete "my take" on `functions.py` docstrings & example improvements. ### Why are the changes needed? To improve PySpark documentation ### Does this PR introduce _any_ user-facing change? Yes, documentation ### How was this patch tested? ``` PYTHON_EXECUTABLE=python3.9 ./dev/lint-python ./python/run-tests --testnames pyspark.sql.functions bundle exec jekyll build ``` Closes #37988 from khalidmammadov/docstrings_funcs_part_8. Authored-by: Khalid Mammadov <[email protected]> Signed-off-by: Sean Owen <[email protected]>

…ples self-contained (FINAL) ### What changes were proposed in this pull request? It's part of the Pyspark docstrings improvement series (apache/spark#37592, apache/spark#37662, apache/spark#37686, apache/spark#37786, apache/spark#37797, apache/spark#37850) In this PR I mainly covered missing parts in the docstrings adding some more examples where it needed. I have also made all examples self explanatory by providing DataFrame creation command where it was missing for clarity to a user. This should complete "my take" on `functions.py` docstrings & example improvements. ### Why are the changes needed? To improve PySpark documentation ### Does this PR introduce _any_ user-facing change? Yes, documentation ### How was this patch tested? ``` PYTHON_EXECUTABLE=python3.9 ./dev/lint-python ./python/run-tests --testnames pyspark.sql.functions bundle exec jekyll build ``` Closes #37988 from khalidmammadov/docstrings_funcs_part_8. Authored-by: Khalid Mammadov <[email protected]> Signed-off-by: Sean Owen <[email protected]>

Improve docstrings with examples

dc80a50

github-actions bot added CORE PYTHON SQL labels Aug 20, 2022

srowen requested changes Aug 20, 2022

View reviewed changes

python/pyspark/sql/functions.py Show resolved Hide resolved

python/pyspark/sql/functions.py Outdated Show resolved Hide resolved

HyukjinKwon reviewed Aug 21, 2022

View reviewed changes

python/pyspark/sql/functions.py Outdated Show resolved Hide resolved

khalidmammadov added 2 commits August 21, 2022 15:17

Fixes

f8c2bcf

Fix floating precision values for consistency

329093d

HyukjinKwon reviewed Aug 22, 2022

View reviewed changes

python/pyspark/sql/functions.py Outdated

+------------------+

| skewness(c)|

+------------------+

|0.7071067811865475|

Copy link

Member

HyukjinKwon Aug 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

HyukjinKwon reviewed Aug 22, 2022

View reviewed changes

python/pyspark/sql/functions.py Outdated

+-----------------+

| stddev_pop(id)|

+-----------------+

|1.707825127659933|

Copy link

Member

HyukjinKwon Aug 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

HyukjinKwon reviewed Aug 22, 2022

View reviewed changes

HyukjinKwon approved these changes Aug 22, 2022

View reviewed changes

One liner for floating results

3bc962d

HyukjinKwon reviewed Aug 23, 2022

View reviewed changes

Apply suggestions from code review

0edee8b

HyukjinKwon closed this in d2f012e Aug 23, 2022

This was referenced Sep 3, 2022

[SPARK-40142][PYTHON][SQL][FOLLOW-UP] Make pyspark.sql.functions examples self-contained (part 5, ~28 functions) #37786

Closed

[SPARK-40142][PYTHON][SQL][FOLLOW-UP] Make pyspark.sql.functions examples self-contained (part 6, ~50 functions) #37797

Closed

khalidmammadov mentioned this pull request Sep 10, 2022

[SPARK-40142][PYTHON][SQL][FOLLOW-UP] Make pyspark.sql.functions examples self-contained (part 7, ~30 functions) #37850

Closed

khalidmammadov mentioned this pull request Sep 25, 2022

[SPARK-40142][PYTHON][SQL][FOLLOW-UP] Make pyspark.sql.functions examples self-contained (FINAL) #37988

Closed

[SPARK-40142][PYTHON][SQL][FOLLOW-UP] Make pyspark.sql.functions examples self-contained (part 2, 32 functions) #37592

[SPARK-40142][PYTHON][SQL][FOLLOW-UP] Make pyspark.sql.functions examples self-contained (part 2, 32 functions) #37592

Uh oh!

Conversation

khalidmammadov commented Aug 20, 2022

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

HyukjinKwon commented Aug 21, 2022

Uh oh!

AmplabJenkins commented Aug 21, 2022

Uh oh!

khalidmammadov commented Aug 21, 2022

Uh oh!

HyukjinKwon Aug 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon Aug 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon Aug 22, 2022

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon Aug 22, 2022

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon Aug 22, 2022

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon Aug 22, 2022

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

HyukjinKwon commented Aug 23, 2022

Uh oh!

HyukjinKwon commented Aug 23, 2022

Uh oh!

khalidmammadov commented Aug 23, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HyukjinKwon Aug 22, 2022 •

edited

Loading

HyukjinKwon Aug 22, 2022 •

edited

Loading