Skip to content

Conversation

@ksakellis
Copy link

HiveFromSpark read the kv1.txt file from SPARK_HOME/examples/src/main/resources/kv1.txt which assumed
you had a source tree checked out. Now we copy the kv1.txt file to a temporary file and delete it when
the jvm shuts down. This allows us to run this example outside of a spark source tree.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this works for non-empty directories. You'd need to recursively delete it. Instead of creating a temp directory, I'd just use File.createTempFile(), unless the name of the file reeeeeally needs to be kv1.txt.

Also, the paranoid in me would close all open streams in a finally block, but not super important in this code.

@vanzin
Copy link
Contributor

vanzin commented Dec 6, 2014

Looks ok aside from the cleanup issue.

@SparkQA
Copy link

SparkQA commented Dec 6, 2014

Test build #24201 has started for PR 3628 at commit fca97ee.

  • This patch merges cleanly.

@marmbrus
Copy link
Contributor

marmbrus commented Dec 6, 2014

Thanks for doing this! Mind adding [SQL] to the PR title?

@ksakellis ksakellis changed the title [SPARK-4774] Makes HiveFromSpark more portable [SPARK-4774] [SQL] Makes HiveFromSpark more portable Dec 6, 2014
HiveFromSpark read the kv1.txt file from
SPARK_HOME/examples/src/main/resources/kv1.txt which assumed
you had a source tree checked out. Now we copy the
kv1.txt file to a temporary file and delete it when
the jvm shutsdown. This allows us to run this example
outside of a spark source tree.
@SparkQA
Copy link

SparkQA commented Dec 6, 2014

Test build #24202 has started for PR 3628 at commit 6770f83.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Dec 6, 2014

Test build #24201 has finished for PR 3628 at commit fca97ee.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24201/
Test PASSed.

@SparkQA
Copy link

SparkQA commented Dec 6, 2014

Test build #24202 has finished for PR 3628 at commit 6770f83.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24202/
Test PASSed.

@marmbrus
Copy link
Contributor

marmbrus commented Dec 8, 2014

LGTM

asfgit pushed a commit that referenced this pull request Dec 8, 2014
HiveFromSpark read the kv1.txt file from SPARK_HOME/examples/src/main/resources/kv1.txt which assumed
you had a source tree checked out. Now we copy the kv1.txt file to a temporary file and delete it when
the jvm shuts down. This allows us to run this example outside of a spark source tree.

Author: Kostas Sakellis <[email protected]>

Closes #3628 from ksakellis/kostas-spark-4774 and squashes the following commits:

6770f83 [Kostas Sakellis] [SPARK-4774] [SQL] Makes HiveFromSpark more portable

(cherry picked from commit d6a972b)
Signed-off-by: Michael Armbrust <[email protected]>
@asfgit asfgit closed this in d6a972b Dec 8, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants