[SPARK-18091] [SQL] Deep if expressions cause Generated SpecificUnsafeProjection code to exceed JVM code size limit #15620

kapilsingh5050 · 2016-10-25T09:42:04Z

What changes were proposed in this pull request?

Fix for SPARK-18091 which is a bug related to large if expressions causing generated SpecificUnsafeProjection code to exceed JVM code size limit.

This PR changes if expression's code generation to place its predicate, true value and false value expressions' generated code in separate methods in context so as to never generate too long combined code.

How was this patch tested?

Added a unit test and also tested manually with the application (having transformations similar to the unit test) which caused the issue to be identified in the first place.

kapilsingh5050 · 2016-10-25T09:47:37Z

@rxin @liancheng @sameeragarwal Please review

cloud-fan · 2016-11-10T07:24:58Z

ok to test.

SparkQA · 2016-11-10T07:29:07Z

Test build #68451 has finished for PR 15620 at commit 62f7b4e.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-11-10T11:40:55Z

Test build #68464 has finished for PR 15620 at commit 6059572.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

kapilsingh5050 · 2016-11-18T06:37:18Z

I ran the tests on my machine and they passed. @cloud-fan Can you please trigger a re-run of test build?

cloud-fan · 2016-11-18T07:36:31Z

retest this please

SparkQA · 2016-11-18T07:37:34Z

Test build #68832 has started for PR 15620 at commit 6059572.

SparkQA · 2016-11-18T09:24:19Z

Test build #68838 has finished for PR 15620 at commit 19622e2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

kapilsingh5050 · 2016-11-18T09:38:42Z

@cloud-fan the same tests which failed earlier are failing again. Could you issue retest request one more time?

cloud-fan · 2016-11-18T11:28:12Z

retest this please

SparkQA · 2016-11-18T12:19:18Z

Test build #68851 has finished for PR 15620 at commit 19622e2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2016-11-19T03:16:19Z

Looks like it is not flaky test. The test failure is due to compilation error of janino: A local variable can not be accessed in the generated evaluation function.

kapilsingh5050 · 2016-11-22T17:58:47Z

I did not want to run tests using ./dev/run-tests on my machine because it'll take hours. I ran the mllib module tests using "mvn test" which worked fine on my machine. What could be the difference between such a run on my machine and the Jenkins Test Build? Please guide me on further steps.

cloud-fan · 2016-11-24T02:16:58Z

can you try sbt? The jenkins run tests using sbt instead of maven

viirya · 2016-11-24T13:11:42Z

I've tested one of the failed tests locally with sbt. It failed.

kapilsingh5050 · 2016-11-25T08:46:49Z

Let me try running the unit tests with sbt

kapilsingh5050 · 2016-11-27T14:03:45Z

I have been able to reproduce the failures using sbt. Investigating the cause.

rxin · 2016-11-29T05:56:30Z

@kapilsingh5050 any update?

kapilsingh5050 · 2016-11-29T06:24:40Z

I'm not sure whether I understand the problem completely. It seems WholeStage CodeGen wants the complete stage code in a single function. The fix I made violates this because it creates separate methods for codegen of If expression's condition, if block and then block (of course, when the total code is beyond a threshold).

Now, if my assumption about WholeStage CodeGen is correct then how does the split expressions logic work? Or maybe the problem is specific to initialisation of input attributes of a stage.

kapilsingh5050 · 2016-11-29T11:24:27Z

The unit tests passed after incorporating related changes from SPARK-15327 and SPARK-17115

SparkQA · 2016-11-29T14:03:25Z

Test build #69324 has finished for PR 15620 at commit 6a7e9ac.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

kapilsingh5050 · 2016-11-29T14:07:35Z

@viirya @cloud-fan @rxin Please review and merge

cloud-fan · 2016-11-30T08:16:09Z

...talyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala

The limitation is 64k, we don't need to be so conservative

I used the same limit as in following change:
https://github.com/apache/spark/pull/14692/files#diff-8bcc5aea39c73d4bf38aef6f6951d42cL595

which is based on some benchmarks. In addition, for JIT to do its optimisations, I think the limit is 8k.

cloud-fan · 2016-11-30T08:18:02Z

...talyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala

please follow the existing code style in Spark, i.e.

def xxx( param1: xxx, param2: xxx): xxx

Will update

cloud-fan · 2016-11-30T08:18:33Z

...talyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala

looks like we don't need to pass in the expr, but only its data type

Yep, I'll fix it

viirya · 2016-11-30T08:20:07Z

...talyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala

So if condition and true/false expressions are bound to currentVars, we still exceed JVM code size limit?

I'm curious about how other similar patches handle whole stage codegen, any ideas?

@viirya I'm not sure but looking at the conversation on following change:
#13235
it looks like that

We will pass needed local variables to the separated functions.

I am not sure if we want to do it here. It will add a bit complexity. If for simplicity, I think we can not to support it now.

I think the conversation on the change I mentioned also went like there will be some refactoring required in whole stage code generation to support that

cloud-fan · 2016-12-02T13:58:05Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CodeGenerationSuite.scala

+    }
+
+    val expressions = Seq(strExpr)
+    val plan = GenerateUnsafeProjection.generate(expressions, true)


nit: other similar tests use GenerateMutableProjection to test the codegen, any specific reason we have to use GenerateUnsafeProjection here?

The issue I had encountered involved GenerateUnsafeProjection and I was trying to reproduce the same in the unit test so I went with it. I can't think of any other reason. Any specific reason you guys use GenerateMutableProjection?

no specific reason too.

viirya · 2016-12-02T15:10:40Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CodeGenerationSuite.scala

    assert(actual(0) == cases)
  }

+  test("SPARK-18091: split large if expressions into blocks due to JVM code size limit") {


This test looks better now.

viirya · 2016-12-02T15:14:05Z

This looks good now.

SparkQA · 2016-12-02T15:32:31Z

Test build #69569 has finished for PR 15620 at commit 260c8e8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

kapilsingh5050 · 2016-12-02T16:50:27Z

@viirya @cloud-fan Are we good to merge? Please let me know if there is any other action on my end.

viirya · 2016-12-03T04:02:28Z

LGTM and cc @cloud-fan for another check.

…Projection code to exceed JVM code size limit ## What changes were proposed in this pull request? Fix for SPARK-18091 which is a bug related to large if expressions causing generated SpecificUnsafeProjection code to exceed JVM code size limit. This PR changes if expression's code generation to place its predicate, true value and false value expressions' generated code in separate methods in context so as to never generate too long combined code. ## How was this patch tested? Added a unit test and also tested manually with the application (having transformations similar to the unit test) which caused the issue to be identified in the first place. Author: Kapil Singh <[email protected]> Closes #15620 from kapilsingh5050/SPARK-18091-IfCodegenFix. (cherry picked from commit e463678) Signed-off-by: Wenchen Fan <[email protected]>

cloud-fan · 2016-12-04T09:17:24Z

thanks, merging to master/2.1!

kapilsingh5050 · 2016-12-04T11:12:51Z

@cloud-fan Can we merge this change to 1.6 and 2.0 too?

…Projection code to exceed JVM code size limit ## What changes were proposed in this pull request? Fix for SPARK-18091 which is a bug related to large if expressions causing generated SpecificUnsafeProjection code to exceed JVM code size limit. This PR changes if expression's code generation to place its predicate, true value and false value expressions' generated code in separate methods in context so as to never generate too long combined code. ## How was this patch tested? Added a unit test and also tested manually with the application (having transformations similar to the unit test) which caused the issue to be identified in the first place. Author: Kapil Singh <[email protected]> Closes #15620 from kapilsingh5050/SPARK-18091-IfCodegenFix. (cherry picked from commit e463678) Signed-off-by: Wenchen Fan <[email protected]>

cloud-fan · 2016-12-04T12:39:13Z

merged to 2.0. Can you send a new PR to backport this to 1.6? There are a lot of code changes beween 1.6 and 2.1, it's safer to open a PR and run all tests.

kapilsingh5050 · 2016-12-05T13:41:29Z

@cloud-fan created #16146 for backporting to 1.6

srowen · 2016-12-09T21:34:58Z

@cloud-fan @kapilsingh5050 it's weird, but it looks like after this change, all of the Maven-based 2.0 Jenkins jobs now time out. It looks consistent: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/

They all end like this:

CodeGenerationSuite:
- multithreaded eval
- metrics are recorded on compile
- SPARK-8443: split wide projections into blocks due to JVM code size limit
- SPARK-13242: case-when expression with large number of branches (or cases)
Exception in thread "ScalaTest-dispatcher" java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:3332)
	at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421)
	at java.lang.StringBuilder.append(StringBuilder.java:136)
	at scala.collection.mutable.StringBuilder.append(StringBuilder.scala:200)
	at scala.collection.TraversableOnce$$anonfun$addString$1.apply(TraversableOnce.scala:364)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
	at scala.collection.TraversableOnce$class.addString(TraversableOnce.scala:357)
	at scala.collection.mutable.ArrayOps$ofRef.addString(ArrayOps.scala:186)
	at scala.collection.TraversableOnce$class.mkString(TraversableOnce.scala:323)
	at scala.collection.mutable.ArrayOps$ofRef.mkString(ArrayOps.scala:186)
	at scala.collection.TraversableOnce$class.mkString(TraversableOnce.scala:325)
	at scala.collection.mutable.ArrayOps$ofRef.mkString(ArrayOps.scala:186)
	at org.scalatest.tools.Fragment.toPossiblyColoredText(Fragment.scala:28)
	at org.scalatest.tools.PrintReporter.printPossiblyInColor(PrintReporter.scala:136)
	at org.scalatest.tools.StringReporter$$anonfun$apply$1.apply(StringReporter.scala:180)
	at org.scalatest.tools.StringReporter$$anonfun$apply$1.apply(StringReporter.scala:180)
	at scala.collection.Iterator$class.foreach(Iterator.scala:893)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
	at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
	at org.scalatest.tools.StringReporter.apply(StringReporter.scala:180)
	at org.scalatest.tools.PrintReporter.apply(PrintReporter.scala:141)
	at org.scalatest.DispatchReporter$Propagator$$anonfun$run$1.apply(DispatchReporter.scala:240)
	at org.scalatest.DispatchReporter$Propagator$$anonfun$run$1.apply(DispatchReporter.scala:239)
	at scala.collection.immutable.List.foreach(List.scala:381)
	at org.scalatest.DispatchReporter$Propagator.run(DispatchReporter.scala:239)
	at java.lang.Thread.run(Thread.java:745)
Build timed out (after 210 minutes). Marking the build as aborted.
Build was aborted
Archiving artifacts
Recording test results
Finished: ABORTED

cloud-fan · 2016-12-10T04:59:54Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CodeGenerationSuite.scala

+
+    var strExpr: Expression = inputStrAttr
+    for (_ <- 1 to 13) {
+      strExpr = If(EqualTo(Decode(Encode(strExpr, "utf-8"), "utf-8"), inputStrAttr),


cc @srowen I think this is the root cause. This test is an overkill, although this PR fixed the code size limitation problem, this test may still hit constants pool size limitation, which is a known limiation and has not been fixed yet. It seems that maven and sbt have different JVM settings when run test, so the problem only exists at maven side.

I'm going to submit a PR to simplify this test a bit.

Very interested in this, we know with 5 iterations instead of a 13 we don't get the problem (which makes sense as we'd only be generating so much code, not hitting the 64k constant pool size limit). I'm testing with IBM's SDK for Java so curious if it manifests itself in a different way for us or if we have a problem to fix on our end.

I have log files exceeding 2 GB from the test printing the generated code on failure.

If we add prints for the strExpr we see something like

The problem is

CodeGenerationSuite: - multithreaded eval - metrics are recorded on compile - SPARK-8443: split wide projections into blocks due to JVM code size limit - SPARK-13242: case-when expression with large number of branches (or cases) - SPARK-18091: split large if expressions into blocks due to JVM code size limit *** FAILED *** java.util.concurrent.ExecutionException: java.lang.Exception: failed to compile: org.codehaus.janino.JaninoRuntimeException: Constant pool for class org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection has grown past JVM limit of 0xFFFF /* 001 */ public java.lang.Object generate(Object[] references) { /* 002 */ return new SpecificUnsafeProjection(references); /* 003 */ } /* 004 */ /* 005 */ class SpecificUnsafeProjection extends org.apache.spark.sql.catalyst.expressions.UnsafeProjection { /* 006 */ /* 007 */ private Object[] references; /* 008 */ private boolean isNull42; /* 009 */ private boolean value42; /* 010 */ private boolean isNull43; /* 011 */ private UTF8String value43; /* 012 */ private boolean isNull44; /* 013 */ private UTF8String value44; /* 014 */ private boolean isNull58;

With my prints:

debug, row: [StringForTesting] input string attr: input[0, string, true] in the loop strExpr is: if ((decode(encode(input[0, string, true], utf-8), utf-8) = input[0, string, true])) input[0, string, true] else input[0, string, true] in the loop strExpr is: if ((decode(encode(if ((decode(encode(input[0, string, true], utf-8), utf-8) = input[0, string, true])) input[0, string, true] else input[0, string, true], utf-8), utf-8) = input[0, string, true])) if ((decode(encode(input[0, string, true], utf-8), utf-8) = input[0, string, true])) input[0, string, true] else input[0, string, true] else if ((decode(encode(input[0, string, true], utf-8), utf-8) = input[0, string, true])) input[0, string, true] else input[0, string, true] in the loop strExpr is: if ((decode(encode(if ((decode(encode(if ((decode(encode(input[0, string, true], utf-8), utf-8) = input[0, string, true])) input[0, string, true] else input[0, string, true], utf-8), utf-8) = input[0, string, true])) if ((decode(encode(input[0, string, true], utf-8), utf-8) = input[0, string, true])) input[0, string, true] else input[0, string, true] else if ((decode(encode(input[0, string, true], utf-8), utf-8) = input[0, string, true])) input[0, string, true] else input[0, string, true], utf-8), utf-8) = input[0, string, true])) if ((decode(encode(if ((decode(encode(input[0, string, true], utf-8), utf-8) = input[0, string, true])) input[0, string, true] else input[0, string, true], utf-8), utf-8) = input[0, string, true])) if ((decode(encode(input[0, string, true], utf-8), utf-8) = input[0, string, true])) input[0, string, true] else input[0, string, true] else if ((decode(encode(input[0, string, true], utf-8), utf-8) = input[0, string, true])) input[0, string, true] else input[0, string, true] else if ((decode(encode(if ((decode(encode(input[0, string, true], utf-8), utf-8) = input[0, string, true])) input[0, string, true] else input[0, string, true], utf-8), utf-8) = input[0, string, true])) if ((decode(encode(input[0, string, true], utf-8), utf-8) = input[0, string, true])) input[0, string, true] else input[0, string, true] else if ((decode(encode(input[0, string, true], utf-8), utf-8) = input[0, string, true])) input[0, string, true] else input[0, string, true] etc

Is this the same issue we're referring to? I've also seen timeouts and our Jenkins farm has 5h limits, I see the problem against branch-2.0, branch-2.1, master, but didn't see it for Spark 2.1.0 RC1.

@a-roberts I think this is merged into branch-2.1 after RC1 coming out.

## What changes were proposed in this pull request? After #15620 , all of the Maven-based 2.0 Jenkins jobs time out consistently. As I pointed out in #15620 (comment) , it seems that the regression test is an overkill and may hit constants pool size limitation, which is a known issue and hasn't been fixed yet. Since #15620 only fix the code size limitation problem, we can simplify the test to avoid hitting constants pool size limitation. ## How was this patch tested? test only change Author: Wenchen Fan <[email protected]> Closes #16244 from cloud-fan/minor. (cherry picked from commit 9abd05b) Signed-off-by: Sean Owen <[email protected]>

## What changes were proposed in this pull request? After #15620 , all of the Maven-based 2.0 Jenkins jobs time out consistently. As I pointed out in #15620 (comment) , it seems that the regression test is an overkill and may hit constants pool size limitation, which is a known issue and hasn't been fixed yet. Since #15620 only fix the code size limitation problem, we can simplify the test to avoid hitting constants pool size limitation. ## How was this patch tested? test only change Author: Wenchen Fan <[email protected]> Closes #16244 from cloud-fan/minor.

## What changes were proposed in this pull request? After #15620 , all of the Maven-based 2.0 Jenkins jobs time out consistently. As I pointed out in #15620 (comment) , it seems that the regression test is an overkill and may hit constants pool size limitation, which is a known issue and hasn't been fixed yet. Since #15620 only fix the code size limitation problem, we can simplify the test to avoid hitting constants pool size limitation. ## How was this patch tested? test only change Author: Wenchen Fan <[email protected]> Closes #16244 from cloud-fan/minor. (cherry picked from commit 9abd05b) Signed-off-by: Sean Owen <[email protected]>

cloud-fan · 2016-12-12T02:43:19Z

the test is good now https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.0-test-maven-hadoop-2.2/

## What changes were proposed in this pull request? After apache#15620 , all of the Maven-based 2.0 Jenkins jobs time out consistently. As I pointed out in apache#15620 (comment) , it seems that the regression test is an overkill and may hit constants pool size limitation, which is a known issue and hasn't been fixed yet. Since apache#15620 only fix the code size limitation problem, we can simplify the test to avoid hitting constants pool size limitation. ## How was this patch tested? test only change Author: Wenchen Fan <[email protected]> Closes apache#16244 from cloud-fan/minor.

…Projection code to exceed JVM code size limit ## What changes were proposed in this pull request? Fix for SPARK-18091 which is a bug related to large if expressions causing generated SpecificUnsafeProjection code to exceed JVM code size limit. This PR changes if expression's code generation to place its predicate, true value and false value expressions' generated code in separate methods in context so as to never generate too long combined code. ## How was this patch tested? Added a unit test and also tested manually with the application (having transformations similar to the unit test) which caused the issue to be identified in the first place. Author: Kapil Singh <[email protected]> Closes apache#15620 from kapilsingh5050/SPARK-18091-IfCodegenFix.

## What changes were proposed in this pull request? After apache#15620 , all of the Maven-based 2.0 Jenkins jobs time out consistently. As I pointed out in apache#15620 (comment) , it seems that the regression test is an overkill and may hit constants pool size limitation, which is a known issue and hasn't been fixed yet. Since apache#15620 only fix the code size limitation problem, we can simplify the test to avoid hitting constants pool size limitation. ## How was this patch tested? test only change Author: Wenchen Fan <[email protected]> Closes apache#16244 from cloud-fan/minor.

…Projection code to exceed JVM code size limit ## What changes were proposed in this pull request? Fix for SPARK-18091 which is a bug related to large if expressions causing generated SpecificUnsafeProjection code to exceed JVM code size limit. This PR changes if expression's code generation to place its predicate, true value and false value expressions' generated code in separate methods in context so as to never generate too long combined code. ## How was this patch tested? Added a unit test and also tested manually with the application (having transformations similar to the unit test) which caused the issue to be identified in the first place. Author: Kapil Singh <[email protected]> Closes apache#15620 from kapilsingh5050/SPARK-18091-IfCodegenFix.

## What changes were proposed in this pull request? After apache#15620 , all of the Maven-based 2.0 Jenkins jobs time out consistently. As I pointed out in apache#15620 (comment) , it seems that the regression test is an overkill and may hit constants pool size limitation, which is a known issue and hasn't been fixed yet. Since apache#15620 only fix the code size limitation problem, we can simplify the test to avoid hitting constants pool size limitation. ## How was this patch tested? test only change Author: Wenchen Fan <[email protected]> Closes apache#16244 from cloud-fan/minor.

…essions ## What changes were proposed in this pull request? A frequently reported issue of Spark is the Java 64kb compile error. This is because Spark generates a very big method and it's usually caused by 3 reasons: 1. a deep expression tree, e.g. a very complex filter condition 2. many individual expressions, e.g. expressions can have many children, operators can have many expressions. 3. a deep query plan tree (with whole stage codegen) This PR focuses on 1. There are already several patches(apache#15620 apache#18972 apache#18641) trying to fix this issue and some of them are already merged. However this is an endless job as every non-leaf expression has this issue. This PR proposes to fix this issue in `Expression.genCode`, to make sure the code for a single expression won't grow too big. According to maropu 's benchmark, no regression is found with TPCDS (thanks maropu !): https://docs.google.com/spreadsheets/d/1K3_7lX05-ZgxDXi9X_GleNnDjcnJIfoSlSCDZcL4gdg/edit?usp=sharing ## How was this patch tested? existing test Author: Wenchen Fan <[email protected]> Author: Wenchen Fan <[email protected]> Closes apache#19767 from cloud-fan/codegen.

kapilsingh5050 force-pushed the SPARK-18091-IfCodegenFix branch from 62f7b4e to 6059572 Compare November 10, 2016 10:43

kapilsingh5050 force-pushed the SPARK-18091-IfCodegenFix branch 2 times, most recently from 3213f2e to 19622e2 Compare November 18, 2016 08:35

kapilsingh5050 force-pushed the SPARK-18091-IfCodegenFix branch from 19622e2 to 6a7e9ac Compare November 29, 2016 11:20

cloud-fan reviewed Nov 30, 2016

View reviewed changes

viirya reviewed Nov 30, 2016

View reviewed changes

cloud-fan reviewed Dec 2, 2016

View reviewed changes

viirya reviewed Dec 2, 2016

View reviewed changes

asfgit closed this in e463678 Dec 4, 2016

cloud-fan reviewed Dec 10, 2016

View reviewed changes

cloud-fan mentioned this pull request Dec 10, 2016

[SQL][minor] simplify a test to fix the maven tests #16244

Closed

cloud-fan mentioned this pull request Nov 16, 2017

[SPARK-22543][SQL] fix java 64kb compile error for deeply nested expressions #19767

Closed

[SPARK-18091] [SQL] Deep if expressions cause Generated SpecificUnsafeProjection code to exceed JVM code size limit #15620

[SPARK-18091] [SQL] Deep if expressions cause Generated SpecificUnsafeProjection code to exceed JVM code size limit #15620

Uh oh!

Conversation

kapilsingh5050 commented Oct 25, 2016

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

kapilsingh5050 commented Oct 25, 2016

Uh oh!

cloud-fan commented Nov 10, 2016

Uh oh!

SparkQA commented Nov 10, 2016

Uh oh!

SparkQA commented Nov 10, 2016

Uh oh!

kapilsingh5050 commented Nov 18, 2016

Uh oh!

cloud-fan commented Nov 18, 2016

Uh oh!

SparkQA commented Nov 18, 2016

Uh oh!

SparkQA commented Nov 18, 2016

Uh oh!

kapilsingh5050 commented Nov 18, 2016

Uh oh!

cloud-fan commented Nov 18, 2016

Uh oh!

SparkQA commented Nov 18, 2016

Uh oh!

viirya commented Nov 19, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kapilsingh5050 commented Nov 22, 2016

Uh oh!

cloud-fan commented Nov 24, 2016

Uh oh!

viirya commented Nov 24, 2016

Uh oh!

kapilsingh5050 commented Nov 25, 2016

Uh oh!

kapilsingh5050 commented Nov 27, 2016

Uh oh!

rxin commented Nov 29, 2016

Uh oh!

kapilsingh5050 commented Nov 29, 2016

Uh oh!

kapilsingh5050 commented Nov 29, 2016

Uh oh!

SparkQA commented Nov 29, 2016

Uh oh!

kapilsingh5050 commented Nov 29, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

viirya commented Nov 19, 2016 •

edited

Loading