[SPARK-20431][SS][FOLLOWUP] Specify a schema by using a DDL-formatted string in DataStreamReader

[HyukjinKwon]

What changes were proposed in this pull request?

This pr supported a DDL-formatted string in DataStreamReader.schema.
This fix could make users easily define a schema without importing the type classes.

For example,

scala> spark.readStream.schema("col0 INT, col1 DOUBLE").load("/tmp/abc").printSchema()
root
 |-- col0: integer (nullable = true)
 |-- col1: double (nullable = true)

How was this patch tested?

Added tests in DataStreamReaderWriterSuite.

[SparkQA]

Test build #78352 has started for PR 18373 at commit 8df0cd2.

[HyukjinKwon]

cc @gatorsmile, @zsxwing and @maropu.

[maropu]

Nit: Capitalize integer? (I remember @gatorsmile suggested before)

[maropu]

Thanks for follow-up! IMHO this syntax helps much, so we could support this in other cases for defining schemas (e.g., spark.createDataFrame)? (I think it's impossible to support in all the cases though, is it possible to do in frequently-used APIs...?)

[shaneknapp]

test this please

[HyukjinKwon]

I skimmed SparkSession, Dataset and functions.scala. I think I can add it ,in SparkSession , to

def createDataFrame(rowRDD: RDD[Row], schema: StructType)
def createDataFrame(rowRDD: JavaRDD[Row], schema: StructType)
def createDataFrame(rows: java.util.List[Row], schema: StructType)

Please let me know. If we are not sure for now, I guess it might be okay as is as a proper follow-up.

[HyukjinKwon]

Thanks for triggering tests and fixing Jenkins @shaneknapp.

[SparkQA]

Test build #78354 has finished for PR 18373 at commit 8df0cd2.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #78367 has finished for PR 18373 at commit 55f1d07.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

@amoussoubaruch, sounds unrelated with this PR. Probably, question should go to mailing list.

[HyukjinKwon]

cc @cloud-fan, would you have some time to look into this maybe?

[cloud-fan]

thanks, merging to master!

@amoussoubaruch please post your question to dev list, instead of randomly picking a PR...