Apache Spark 2.4.0 Release: Announcements & Release Notes #158

cloud-fan · 2018-11-05T05:22:47Z

This patch adds the release notes and relevant announcements for Spark 2.4.0.

The first commit just adds the docs of the latest RC, please review the second commit.

Release Notes: https://github.com/cloud-fan/spark-website/blob/62847a7d4b106ad6ac1aee2016e226b56c7e9d1a/releases/_posts/2018-11-05-spark-release-2-4-0.md

cloud-fan · 2018-11-05T05:23:31Z

cc @rxin @vanzin @srowen @holdenk @felixcheung @gatorsmile

felixcheung

it's too many to list or review on github - I reviewed those not under site/docs, LGTM

kiszk · 2018-11-05T07:32:56Z

releases/_posts/2018-11-05-spark-release-2-4-0.md

+ - **Major features**
+   - **Barrier Execution Mode**: [[SPARK-24374](https://issues.apache.org/jira/browse/SPARK-24374)] Support Barrier Execution Mode in the scheduler, to better integrate with deep learning frameworks.
+   - **Scala 2.12 Support**: [[SPARK-14220](https://issues.apache.org/jira/browse/SPARK-14220)] Add experimental Scala 2.12 support. Now you can build Spark with Scala 2.12 and write Spark applications in Scala 2.12.
+   - **Higher-order functions**: [[SPARK-23899](https://issues.apache.org/jira/browse/SPARK-23899)] Add a lof of new built-in functions, including high-order functions, to deal with complex data types easier.


lof -> lot

also higher-order instead of high-order?

kiszk · 2018-11-05T07:40:30Z

releases/_posts/2018-11-05-spark-release-2-4-0.md

+
+ - **PySpark**
+   - [[SPARK-24215](https://issues.apache.org/jira/browse/SPARK-24215)] Implement eager evaluation for DataFrame APIs
+   - [[SPARK-22274](https://issues.apache.org/jira/browse/SPARK-22274)] User-defined aggregation functions with pandas UDF


nit: pandas -> Pandas?

kiszk · 2018-11-05T07:41:05Z

releases/_posts/2018-11-05-spark-release-2-4-0.md

+ - **PySpark**
+   - [[SPARK-24215](https://issues.apache.org/jira/browse/SPARK-24215)] Implement eager evaluation for DataFrame APIs
+   - [[SPARK-22274](https://issues.apache.org/jira/browse/SPARK-22274)] User-defined aggregation functions with pandas UDF
+   - [[SPARK-22239](https://issues.apache.org/jira/browse/SPARK-22239)] User-defined window functions with pandas UDF


gatorsmile · 2018-11-05T07:42:37Z

releases/_posts/2018-11-05-spark-release-2-4-0.md

+
+### Credits
+Last but not least, this release would not have been possible without the following contributors:
+Achuth17, Adam Bradbury, Adamyuanyuan, Adelbert Chang, Ala Luszczak, Aleksandr Koriagin, Alessandro Bellina, Alessandro Solimando, Andrew Korzhuev, Anton Okolnychyi, Antonio Murgia, Arseniy Tashoyan, Artem Rudoy, Arun Mahadevan, Asher Saban, Bago Amirbekian, Benjamin Peterson, Bo Meng, Bogdan Raducanu, Bounkong Khamphousone, Brandon Krieger, Brian Lindblom, Bruce Robbins, Bryan Cutler, Cheng Lian, Chongguang LIU, Chris Horn, Chris Martin, Cody Koeninger, DB Tsai, Daniel Sakuma, Daniel Van Der Ende, Darcy Shen, David Vogelbacher, Devaraj K, Dhruve Ashar, Dilip Biswal, Dongjoon Hyun, DylanGuedes, Efim Poberezkin, Eric Chang, Eric Liang, Erik Erlandson, Eyal Farago, Fangshi Li, Felix Cheung, Feng Liu, Fernando Pereira, Florent Pepin, Fokko Driesprong, Gabor Somogyi, Gengliang Wang, Ger Van Rossum, Gera Shegalov, Goun Na, Hao Ren, Henry Robinson, Herman Van Hovell, Hieu Huynh, Holden Karau, Huang Tengfei, Huaxin Gao, Hyukjin Kwon, Ilan Filonenko, Imran Rashid, Jacek Laskowski, Jake Charland, James Thompson, James Yu, Jaroslav Chladek, Jeff Zhang, JiahuiJiang, Jim Kleckner, Joey Krabacher, John Zhuge, Jongyoul Lee, Jooseong Kim, Jose Torres, Joseph Bradley, Joseph K. Bradley, Josh Rosen, Julien Cuquemelle, Juliusz Sompolski, Jungtaek Lim, KaiXinXIaoLei, Kallman, Steven, Karthik Palaniappan, Kaya Kupferschmidt, Kazuaki Ishizaki, Kelley Robinson, Kent Yao, Kevin Yu, KevinZwx, Koert Kuipers, Kousuke Saruta, Kris Mok, LantaoJin, Lee Dongjin, Lemonjing, Li Jin, Liang-Chi Hsieh, Lu WANG, LucaCanali, Marcelo Vanzin, Marco Gaido, Marek Novotny, Mario Molina, Mark Petruska, Maryann Xue, Mathieu St-Louis, Matthew Cheah, Matthew Tovbin, Mauro Palsgraaf, Maxim Gekk, Michael (Stu) Stewart, Michael Allman, Michael Chirico, Michael Mior, Michal Switakowski, Mihaly Toth, Miklos C, Miles Yucht, Misha Dmitriev, Mukul Murthy, Mykhailo Shtelma, Neal Song, Ngone51, Nihar Sheth, Nolan Emirot, Norman Maurer, Onur Satici, Patrick McGloin, Patrick Pisciuneri, Paul Mackles, Peter Toth, Prashant Sharma, Rao Fu, Ray Burgemeestre, Rekha Joshi, Reynold Xin, Reza Safi, Ricardo Martinelli De Oliveira, Rob Vesse, Robert Kruszewski, Rong Tang, Ryan Blue, Sahil Takiar, Saisai Shao, Sandeep Singh, Sandor Murakozi, Sanket Chintapalli, Santiago Saavedra, Sean Owen, Sean Suchter, Sebastian Arzt, Shane Knapp, Shixiong Zhu, Soham Aurangabadkar, Stacy Kerkela, Stan Zhai, Stavros Kontopoulos, Steve Loughran, Sunitha Kambhampati, Takeshi Yamamuro, Takuya UESHIN, Tathagata Das, Ted Yu, Teng Peng, Thiruvasakan Paramasivan, Thomas Graves, Tom Saleeba, Vayda, Oleksandr: IT (PRG), Vinod KC, Vladimir Kuriatkov, Wang Gengliang, Weichen Xu, Wenbo Zhao, Wenchen Fan, William Sheu, XD-DENG, Xiangrui Meng, Xianjin YE, Xianyang Liu, Xiao Li, Xiaogang Tu, Xiayun Sun, Xingbo Jiang, Yacine Mazari, Yash Sharma, Ye Zhou, Yinan Li, Yogesh Garg, Yuanbo Liu, Yuanjian Li, Yuchen Huo, Yuexin Zhang, Yuming Wang, Yuval Itzchakov, Zhan Zhang, Zhang Le, Zheng RuiFeng, Zoltan C. Toth


We need to update https://github.com/apache/spark/blob/master/dev/create-release/known_translations for the github nicknames.

apache/spark#22949

kiszk · 2018-11-05T08:24:55Z

Here is a set of output by misspell. I think that Euclidian is intentionally used while I am not familiar with ML.

$ spark-website > ../../bin/misspell site/docs/2.4.0/
site/docs/2.4.0/api/java/org/apache/spark/ExecutorPlugin.html:110:27: "compatability" is a misspelling of "compatibility"
site/docs/2.4.0/api/java/org/apache/spark/ml/stat/Summarizer.html:312:16: "Euclidian" is a misspelling of "Euclidean"
site/docs/2.4.0/api/java/org/apache/spark/ml/stat/Summarizer.html:345:16: "Euclidian" is a misspelling of "Euclidean"
site/docs/2.4.0/api/java/org/apache/spark/mllib/clustering/EMLDAOptimizer.html:130:34: "Teh" is a misspelling of "The"
site/docs/2.4.0/api/java/org/apache/spark/mllib/stat/MultivariateOnlineSummarizer.html:220:23: "Euclidian" is a misspelling of "Euclidean"
site/docs/2.4.0/api/java/org/apache/spark/mllib/stat/MultivariateOnlineSummarizer.html:401:23: "Euclidian" is a misspelling of "Euclidean"
site/docs/2.4.0/api/java/org/apache/spark/storage/ReadableChannelFileRegion.html:190:36: "transfered" is a misspelling of "transferred"
site/docs/2.4.0/api/python/_modules/pyspark/ml/stat.html:375:40: "Euclidian" is a misspelling of "Euclidean"
site/docs/2.4.0/api/scala/lib/scheduler.js:62:41: "existant" is a misspelling of "existent"
site/docs/2.4.0/api/scala/lib/jquery.layout.js:2312:4: "dependancy" is a misspelling of "dependency"
site/docs/2.4.0/api/scala/lib/jquery.layout.js:2629:46: "Destory" is a misspelling of "Destroy"
site/docs/2.4.0/api/scala/lib/jquery.layout.js:4767:4: "dependancies" is a misspelling of "dependencies"
site/docs/2.4.0/api/scala/lib/jquery.layout.js:4768:4: "dependancies" is a misspelling of "dependencies"
site/docs/2.4.0/api/scala/lib/jquery.layout.js:5123:4: "dependancies" is a misspelling of "dependencies"
site/docs/2.4.0/api/scala/lib/jquery.layout.js:5404:4: "dependancies" is a misspelling of "dependencies"
site/docs/2.4.0/api/scala/lib/jquery.layout.js:5434:58: "occassionally" is a misspelling of "occasionally"
site/docs/2.4.0/api/python/searchindex.js:1:138287: "composit" is a misspelling of "compost"
site/docs/2.4.0/api/python/searchindex.js:1:138596: "concret" is a misspelling of "concert"
site/docs/2.4.0/api/python/searchindex.js:1:139122: "convers" is a misspelling of "converts"
site/docs/2.4.0/api/python/searchindex.js:1:141263: "decomposit" is a misspelling of "decompose"
site/docs/2.4.0/api/python/searchindex.js:1:144690: "euclidian" is a misspelling of "euclidean"
site/docs/2.4.0/api/python/searchindex.js:1:152810: "includ" is a misspelling of "include"
site/docs/2.4.0/api/python/searchindex.js:1:163447: "occurr" is a misspelling of "occur"
site/docs/2.4.0/api/python/searchindex.js:1:166110: "portugues" is a misspelling of "portuguese"
site/docs/2.4.0/api/scala/org/apache/spark/ml/stat/Summarizer$.html:722:839: "Euclidian" is a misspelling of "Euclidean"
site/docs/2.4.0/api/scala/org/apache/spark/mllib/clustering/EMLDAOptimizer.html:55:33: "Teh" is a misspelling of "The"
site/docs/2.4.0/api/scala/org/apache/spark/mllib/clustering/package.html:173:33: "Teh" is a misspelling of "The"
site/docs/2.4.0/api/scala/org/apache/spark/mllib/stat/MultivariateOnlineSummarizer.html:533:38: "Euclidian" is a misspelling of "Euclidean"
site/docs/2.4.0/api/scala/org/apache/spark/mllib/stat/MultivariateOnlineSummarizer.html:533:133: "Euclidian" is a misspelling of "Euclidean"
site/docs/2.4.0/api/sql/css/theme.css:12:16937: "adn" is a misspelling of "and"
site/docs/2.4.0/api/sql/js/highlight.pack.js:1:11462: "lenght" is a misspelling of "length"
site/docs/2.4.0/api/sql/js/highlight.pack.js:1:92171: "seperate" is a misspelling of "separate"
site/docs/2.4.0/api/sql/js/highlight.pack.js:2:13040: "recieved" is a misspelling of "received"
site/docs/2.4.0/api/python/pyspark.ml.html:32824:16: "Euclidian" is a misspelling of "Euclidean"
site/docs/2.4.0/api/java/index-all.html:29454:23: "Euclidian" is a misspelling of "Euclidean"
$ spark-website > ../../bin/misspell release-process.md 
release-process.md:198:30: "maintance" is a misspelling of "maintenance"

Otherwise, we could address them if possible (not related to 2.4 release). I am neutral on them.

$ spark-website > ../../bin/misspell README.md _* [a-q]* releases screencasts security.md sitemap.xml sql streaming [t-z]*
news/_posts/2014-09-11-spark-1-1-0-released.md:16:451: "libary" is a misspelling of "library"
news/_posts/2016-03-09-spark-1-6-1-released.md:14:232: "signficant" is a misspelling of "significant"
releases/_posts/2014-08-05-spark-release-1-0-2.md:38:20: "consistant" is a misspelling of "consistent"
releases/_posts/2015-06-11-spark-release-1-4-0.md:28:82: "compatiblity" is a misspelling of "compatibility"
releases/_posts/2013-09-25-spark-release-0-8-0.md:16:159: "pacakge" is a misspelling of "package"
releases/_posts/2015-03-13-spark-release-1-3-0.md:19:603: "Finaly" is a misspelling of "Finally"
releases/_posts/2017-07-11-spark-release-2-2-0.md:68:13: "Availablity" is a misspelling of "Availability"

cloud-fan · 2018-11-05T08:33:07Z

@kiszk thanks for checking them! Can you open a JIRA ticket to fix the typos in the doc? These typos need to be fixed in the spark repo, and they can be available in 2.4.1

For other typos that are not related to 2.4.0, feel free to open a PR to fix them. But I'm not sure if we can still update previous posts.

kiszk · 2018-11-05T08:33:22Z

Since the above problem exists before 2.4, it is ok to fix now or to fix in another PR.

kiszk · 2018-11-05T08:38:59Z

Got it. Since most of file are generated from Spark repo or derived from other libs, it would be good to address in other PRs. We would appreciate it if this PR could address the following long-existing problem.

$ spark-website > ../../bin/misspell release-process.md 
release-process.md:198:30: "maintance" is a misspelling of "maintenance"

cloud-fan · 2018-11-05T09:01:45Z

@kiszk fixed at #157

srowen · 2018-11-05T14:48:07Z

@kiszk yeah this should be a pull request against apache/spark to fix typos. That's great! I didn't know about misspell. EDIT: I just opened a PR for the rest at apache/spark#22950

## What changes were proposed in this pull request? Fix typos and misspellings, per apache/spark-website#158 (comment) ## How was this patch tested? Existing tests. Closes #22950 from srowen/Typos. Authored-by: Sean Owen <[email protected]> Signed-off-by: Sean Owen <[email protected]> (cherry picked from commit c0d1bf0) Signed-off-by: Sean Owen <[email protected]>

releases/_posts/2018-11-05-spark-release-2-4-0.md

dongjoon-hyun · 2018-11-08T06:06:59Z

js/downloads.js

+// 2.4.0+
+var packagesV9 = [hadoop2p7, hadoop2p6, hadoopFree, scala2p12_hadoopFree, sources];

+addRelease("2.4.0", new Date("11/05/2018"), packagesV9, true, true);


Is it better to change this since the announcement is delayed? Also, file name also the old date.

news/_posts/2018-11-05-spark-2-4-0-released.md

I checked 2.3.0, the date is when the vote passed. Let me update.

gatorsmile · 2018-11-08T14:41:21Z

LGTM

Thanks! Merged to master.

## What changes were proposed in this pull request? Fix typos and misspellings, per apache/spark-website#158 (comment) ## How was this patch tested? Existing tests. Closes apache#22950 from srowen/Typos. Authored-by: Sean Owen <[email protected]> Signed-off-by: Sean Owen <[email protected]>

## What changes were proposed in this pull request? Fix typos and misspellings, per apache/spark-website#158 (comment) ## How was this patch tested? Existing tests. Closes apache#22950 from srowen/Typos. Authored-by: Sean Owen <[email protected]> Signed-off-by: Sean Owen <[email protected]> (cherry picked from commit c0d1bf0) Signed-off-by: Sean Owen <[email protected]>

## What changes were proposed in this pull request? Fix typos and misspellings, per apache/spark-website#158 (comment) ## How was this patch tested? Existing tests. Closes #22950 from srowen/Typos. Authored-by: Sean Owen <[email protected]> Signed-off-by: Sean Owen <[email protected]> (cherry picked from commit c0d1bf0) Signed-off-by: Sean Owen <[email protected]> (cherry picked from commit 8526f2e)

cloud-fan added 2 commits November 2, 2018 16:01

Add docs for Spark 2.4.0 and update the latest link

52917ac

update website

62847a7

felixcheung approved these changes Nov 5, 2018

View reviewed changes

kiszk reviewed Nov 5, 2018

View reviewed changes

gatorsmile reviewed Nov 5, 2018

View reviewed changes

address comments

02a3641

srowen approved these changes Nov 5, 2018

View reviewed changes

srowen mentioned this pull request Nov 5, 2018

[MINOR] Fix typos and misspellings apache/spark#22950

Closed

HyukjinKwon approved these changes Nov 6, 2018

View reviewed changes

gatorsmile reviewed Nov 6, 2018

View reviewed changes

releases/_posts/2018-11-05-spark-release-2-4-0.md Show resolved Hide resolved

update behavior changes

09c864d

dongjoon-hyun reviewed Nov 8, 2018

View reviewed changes

update date

3726c32

asfgit merged commit 3726c32 into apache:asf-site Nov 8, 2018

Apache Spark 2.4.0 Release: Announcements & Release Notes #158

Apache Spark 2.4.0 Release: Announcements & Release Notes #158

Uh oh!

Conversation

cloud-fan commented Nov 5, 2018

Uh oh!

cloud-fan commented Nov 5, 2018

Uh oh!

felixcheung left a comment

Choose a reason for hiding this comment

Uh oh!

kiszk Nov 5, 2018

Choose a reason for hiding this comment

Uh oh!

ueshin Nov 5, 2018

Choose a reason for hiding this comment

Uh oh!

kiszk Nov 5, 2018

Choose a reason for hiding this comment

Uh oh!

kiszk Nov 5, 2018

Choose a reason for hiding this comment

Uh oh!

gatorsmile Nov 5, 2018

Choose a reason for hiding this comment

Uh oh!

cloud-fan Nov 5, 2018

Choose a reason for hiding this comment

Uh oh!

kiszk commented Nov 5, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cloud-fan commented Nov 5, 2018

Uh oh!

kiszk commented Nov 5, 2018

Uh oh!

kiszk commented Nov 5, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cloud-fan commented Nov 5, 2018

Uh oh!

srowen commented Nov 5, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

dongjoon-hyun Nov 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fan Nov 8, 2018

Choose a reason for hiding this comment

Uh oh!

gatorsmile commented Nov 8, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

kiszk commented Nov 5, 2018 •

edited

Loading

kiszk commented Nov 5, 2018 •

edited

Loading

srowen commented Nov 5, 2018 •

edited

Loading

dongjoon-hyun Nov 8, 2018 •

edited

Loading