[SPARK-2135][SQL] Use planner for in-memory scans

[marmbrus]

No description provided.

[marmbrus]

@liancheng, please take a look.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15740/

[liancheng]

We don't invalidate cached table content here. Maybe we should add a link to the corresponding JIRA ticket to remind that.

[liancheng]

LGTM :)

[tgravescs]

@marmbrus @liancheng I think this broke branch-1.0 from building:

Error I am seeing:
[INFO] Compiling 5 Scala sources to/sql/core/target/scala-2.10/classes...
[ERROR] /sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala:173: too many arguments for method pruneFilterProject: (projectList: Seq[org.apache.spark.sql.catalyst.expressions.NamedExpression], filterPredicates: Seq[org.apache.spark.sql.catalyst.expressions.Expression], scanBuilder: Seq[org.apache.spark.sql.catalyst.expressions.Attribute] => org.apache.spark.sql.execution.SparkPlan)org.apache.spark.sql.execution.SparkPlan
[ERROR] pruneFilterProject(

[liancheng]

@tgravescs @marmbrus I believe the reason of the compilation failure is that PR #863 was not merged into branch-1.0. That PR fixes a Parquet related bug and should be merged into branch-1.0. I'll try locally to see whether we can get it merged now, or I'll try to rebase that PR branch.

[marmbrus]

Thanks for pointing this out @tgravescs.

@liancheng We should probably just fully update branch-1.0 with master as far as Spark SQL goes. We avoiding merging a few patches right near the 1.0 release, but there is no reason to keep them out of 1.0.1.

[liancheng]

@marmbrus Just realized that the Parquet filter predicate pushdown optimization was also not merged. So I'd suggest we either merge both #511 and #863, or submit another PR for branch-1.0 to remove the prunePushDownFilters parameter and related logic and make branch-10 compilable.

[marmbrus]

Here's a PR to bring 1.0 back in line with master: #1078