Skip to content

Conversation

@tianyi
Copy link
Contributor

@tianyi tianyi commented Feb 13, 2015

This PR fix the issue SPARK-3365.
The reason is Spark generated wrong schema for the type List in ScalaReflection.scala
for example:

the generated schema for type Seq[String] is:

{"name":"x","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}}`

the generated schema for type List[String] is:

{"name":"x","type":{"type":"struct","fields":[]},"nullable":true,"metadata":{}}`

@AmplabJenkins
Copy link

Can one of the admins verify this patch?

@liancheng
Copy link
Contributor

ok to test

@liancheng
Copy link
Contributor

This LGTM, let's wait for Jenkins. Thanks for working on this!

@liancheng
Copy link
Contributor

add to whitelist

@SparkQA
Copy link

SparkQA commented Feb 13, 2015

Test build #27411 has started for PR 4581 at commit a8bd6fe.

  • This patch merges cleanly.

@liancheng
Copy link
Contributor

@tianyi After rethinking about this, now I tend to the solution you proposed originally on JIRA, namely move the Product clause below Seq. Usually types like List[T] which extend Seq[T] with Product should be viewed as a Seq at the first place.

@SparkQA
Copy link

SparkQA commented Feb 13, 2015

Test build #27412 has started for PR 4581 at commit a097e86.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Feb 13, 2015

Test build #27411 has finished for PR 4581 at commit a8bd6fe.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27411/
Test PASSed.

@SparkQA
Copy link

SparkQA commented Feb 13, 2015

Test build #27412 has finished for PR 4581 at commit a097e86.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27412/
Test PASSed.

@liancheng
Copy link
Contributor

Thanks! Merging to master and branch-1.3.

asfgit pushed a commit that referenced this pull request Feb 13, 2015
This PR fix the issue SPARK-3365.
The reason is Spark generated wrong schema for the type `List` in `ScalaReflection.scala`
for example:

the generated schema for type `Seq[String]` is:
```
{"name":"x","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}}`
```

the generated schema for type `List[String]` is:
```
{"name":"x","type":{"type":"struct","fields":[]},"nullable":true,"metadata":{}}`
```

Author: tianyi <[email protected]>

Closes #4581 from tianyi/SPARK-3365 and squashes the following commits:

a097e86 [tianyi] change the order of resolution in ScalaReflection.scala

(cherry picked from commit 1c8633f)
Signed-off-by: Cheng Lian <[email protected]>
@asfgit asfgit closed this in 1c8633f Feb 13, 2015
@tianyi tianyi deleted the SPARK-3365 branch February 14, 2015 01:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants