[SPARK-13390][SQL]Fix the issue due to Iterator.map().toSeq is not Serializable

[zsxwing]

What changes were proposed in this pull request?

scala.collection.Iterator's methods (e.g., map, filter) will return an AbstractIterator which is not Serializable. E.g.,

scala> val iter = Array(1, 2, 3).iterator.map(_ + 1)
iter: Iterator[Int] = non-empty iterator

scala> println(iter.isInstanceOf[Serializable])
false

If we call something like Iterator.map(...).toSeq, it will create a Stream that contains a non-serializable AbstractIterator field and make the Stream be non-serializable.

This PR uses toArray instead of toSeq to fix such issue in def createDataFrame(data: java.util.List[_], beanClass: Class[_]): DataFrame.

How was the this patch tested?

Jenkins tests.

[SparkQA]

Test build #51694 has finished for PR 11313 at commit 07a88b5.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zsxwing]

cc @holdenk @srowen

[holdenk]

Maybe good to add a test which would have failed before this change?

[holdenk]

toArray could be kind of expensive for large local iterators - maybe we should only do this if necessary?

[JoshRosen]

I think you have to materialize it anyways in order to support multiple scans.

[holdenk]

yah I know we need to materialize anyways (toSeq forces that too on Iterators) - but I meant more the cost of copying all the objects to an array (see https://github.com/scala/scala/blob/v2.11.1/src/library/scala/collection/TraversableOnce.scala#L269 ) so maybe only calling toArray if necessary would be worth while (but perhaps not).

[srowen]

I think it was the same before -- it was materialized into a Seq. I think Josh is suggesting this is inherently necessary before making a LocalRelation so there's no meaningful opportunity for lazy eval (?) If so then this seems fine.

[zsxwing]

Just found this happened to be fixed in #10511. rows will be materialized in LocalTableScan.

I'm going to resubmit this patch against branch-1.6 since this is not necessary for master.

[zsxwing]

See #11334 instead.