[SPARK-27271][SQL] Migrate Text to File Data Source V2 #24207

gengliangwang · 2019-03-25T17:56:29Z

What changes were proposed in this pull request?

Migrate Text source to File Data Source V2

How was this patch tested?

Unit test

SparkQA · 2019-03-25T19:58:19Z

Test build #103922 has finished for PR 24207 at commit 96664cc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class TextWriteBuilder(

SparkQA · 2019-03-26T07:05:02Z

Test build #103943 has finished for PR 24207 at commit 520cb67.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-03-26T07:36:59Z

retest this please

SparkQA · 2019-03-26T11:08:08Z

Test build #103962 has finished for PR 24207 at commit 520cb67.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-03-27T12:18:23Z

Test build #104004 has finished for PR 24207 at commit 4343ab2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk

LGTM

MaxGekk · 2019-03-27T15:40:59Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileScan.scala

    val maxSplitBytes = FilePartition.maxSplitBytes(sparkSession, selectedPartitions)
    val splitFiles = selectedPartitions.flatMap { partition =>
-      partition.files.flatMap { file =>
+      partition.files.filter(_.getLen > 0).flatMap { file =>


Don't forget to remove this.

@MaxGekk Thanks. This is code change from #24227 and I will remove it once #24227 is merged.

gengliangwang · 2019-03-27T17:29:38Z

@cloud-fan @HyukjinKwon @dongjoon-hyun This is ready for review.

SparkQA · 2019-03-27T22:09:06Z

Test build #104025 has finished for PR 24207 at commit c74dddd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-03-27T23:48:47Z

Looks good too. Let me take a closer look if that's not merged soon before getting it in.

gengliangwang · 2019-03-31T15:45:37Z

@cloud-fan @HyukjinKwon @dongjoon-hyun This is ready for review again.

dongjoon-hyun · 2019-03-31T16:42:06Z

Thank you for updating, @gengliangwang . I'll take a look today.

SparkQA · 2019-03-31T19:29:21Z

Test build #104141 has finished for PR 24207 at commit 1ac450c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/text/TextOutputWriter.scala

...ore/src/main/scala/org/apache/spark/sql/execution/datasources/v2/text/TextDataSourceV2.scala

...in/scala/org/apache/spark/sql/execution/datasources/v2/text/TextPartitionReaderFactory.scala

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/text/TextTable.scala

dongjoon-hyun · 2019-04-01T22:06:27Z

@gengliangwang . The PR looks correct to me. I left only a few minor comments.

HyukjinKwon

LGTM too

gengliangwang · 2019-04-02T15:53:43Z

I am working on another PR about file source V2 framework. I will revisit this one later.

SparkQA · 2019-04-08T14:19:17Z

Test build #104381 has finished for PR 24207 at commit 66b893d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gengliangwang · 2019-04-08T15:14:10Z

@dongjoon-hyun @HyukjinKwon @cloud-fan This PR is ready for review again.
Previously, this PR is was blocked by #24284 and #24296
Thanks in advance!

dongjoon-hyun

+1, LGTM. Merged to master.

Thank you, @gengliangwang , @HyukjinKwon , @MaxGekk .

…FileSuite ### What changes were proposed in this pull request? Add V1/V2 tests for TextSuite and WholeTextFileSuite ### Why are the changes needed? This is missing part since #24207. We should have these tests for test coverage. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Unit tests. Closes #28335 from gengliangwang/testV2Suite. Authored-by: Gengliang Wang <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…FileSuite ### What changes were proposed in this pull request? Add V1/V2 tests for TextSuite and WholeTextFileSuite ### Why are the changes needed? This is missing part since #24207. We should have these tests for test coverage. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Unit tests. Closes #28335 from gengliangwang/testV2Suite. Authored-by: Gengliang Wang <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 16b9615) Signed-off-by: Dongjoon Hyun <[email protected]>

gengliangwang force-pushed the textV2 branch from 96664cc to 520cb67 Compare March 26, 2019 03:33

gengliangwang force-pushed the textV2 branch from 520cb67 to 4343ab2 Compare March 27, 2019 07:37

MaxGekk approved these changes Mar 27, 2019

View reviewed changes

gengliangwang force-pushed the textV2 branch from 4343ab2 to c74dddd Compare March 27, 2019 17:28

gengliangwang changed the title ~~[WIP][SPARK-27271][SQL] Migrate Text to File Data Source V2~~ [SPARK-27271][SQL] Migrate Text to File Data Source V2 Mar 27, 2019

gengliangwang changed the title ~~[SPARK-27271][SQL] Migrate Text to File Data Source V2~~ [WIP][SPARK-27271][SQL] Migrate Text to File Data Source V2 Mar 30, 2019

gengliangwang force-pushed the textV2 branch from c74dddd to 1ac450c Compare March 31, 2019 14:52

gengliangwang changed the title ~~[WIP][SPARK-27271][SQL] Migrate Text to File Data Source V2~~ [SPARK-27271][SQL] Migrate Text to File Data Source V2 Mar 31, 2019

dongjoon-hyun reviewed Apr 1, 2019

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/text/TextOutputWriter.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Apr 1, 2019

View reviewed changes

...ore/src/main/scala/org/apache/spark/sql/execution/datasources/v2/text/TextDataSourceV2.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Apr 1, 2019

View reviewed changes

...in/scala/org/apache/spark/sql/execution/datasources/v2/text/TextPartitionReaderFactory.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Apr 1, 2019

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/text/TextTable.scala Outdated Show resolved Hide resolved

HyukjinKwon approved these changes Apr 1, 2019

View reviewed changes

HyukjinKwon changed the title ~~[SPARK-27271][SQL] Migrate Text to File Data Source V2~~ [WIP][SPARK-27271][SQL] Migrate Text to File Data Source V2 Apr 2, 2019

gengliangwang added 2 commits April 8, 2019 17:38

text v2

2e7c0a9

address comments

66b893d

gengliangwang force-pushed the textV2 branch from 1ac450c to 66b893d Compare April 8, 2019 09:52

gengliangwang changed the title ~~[WIP][SPARK-27271][SQL] Migrate Text to File Data Source V2~~ [SPARK-27271][SQL] Migrate Text to File Data Source V2 Apr 8, 2019

dongjoon-hyun approved these changes Apr 8, 2019

View reviewed changes

dongjoon-hyun closed this in d50603a Apr 8, 2019

gengliangwang mentioned this pull request Apr 25, 2020

[SPARK-31560][SQL][Tests] Add V1/V2 tests for TextSuite and WholeTextFileSuite #28335

Closed

[SPARK-27271][SQL] Migrate Text to File Data Source V2 #24207

[SPARK-27271][SQL] Migrate Text to File Data Source V2 #24207

Uh oh!

Conversation

gengliangwang commented Mar 25, 2019 • edited by dongjoon-hyun Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Mar 25, 2019

Uh oh!

SparkQA commented Mar 26, 2019

Uh oh!

HyukjinKwon commented Mar 26, 2019

Uh oh!

SparkQA commented Mar 26, 2019

Uh oh!

SparkQA commented Mar 27, 2019

Uh oh!

MaxGekk left a comment

Choose a reason for hiding this comment

Uh oh!

MaxGekk Mar 27, 2019

Choose a reason for hiding this comment

Uh oh!

gengliangwang Mar 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gengliangwang commented Mar 27, 2019

Uh oh!

SparkQA commented Mar 27, 2019

Uh oh!

HyukjinKwon commented Mar 27, 2019

Uh oh!

gengliangwang commented Mar 31, 2019

Uh oh!

dongjoon-hyun commented Mar 31, 2019

Uh oh!

SparkQA commented Mar 31, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dongjoon-hyun commented Apr 1, 2019

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

gengliangwang commented Apr 2, 2019

Uh oh!

SparkQA commented Apr 8, 2019

Uh oh!

gengliangwang commented Apr 8, 2019

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gengliangwang commented Mar 25, 2019 •

edited by dongjoon-hyun

Loading

gengliangwang Mar 27, 2019 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading