Skip to content

Conversation

@pwendell
Copy link
Contributor

By default Snappy uses java.io.tempdir for copying the Snappy native library.
If two users run Spark jobs on the same machine it can cause an exception when
the second user tries to access or overwrite the snappy file created by the
first user. This will fail Spark jobs out-of-the-box if they are run on a
machine shared by different users.

Snappy does expose a mechanism to customize the temp directory via a system
property. This system property is read in a static block inside of Snappy code.

I've added a "best effort" fix for this where we try to set the system
property in a static block before Snappy reads it. I've tested it and it does
work, but it relies on static initialization order which is brittle. I.e.
if user code accesses Snappy libraries first this could not-work.

An alternative work-around for users is to explicitly set

org.xerial.snappy.tempdir

themselves through Spark's java options. I also filed a bug upstream with
Snappy-java to ask them for better behavior here:

xerial/snappy-java#84

I think this is worth merging because in many cases it will fix the issue and
at worst it's a no-op.

By default Snappy uses java.io.tempdir for copying the Snappy native library.
If two users run Spark jobs on the same machine it can cause an exception when
the second user tries to access or overwrite the snappy file created by the
first user. This will fail Spark jobs out-of-the-box if they are run on a
machine shared by different users.

Snappy does expose a mechanism to customize the temp directory via a system
property. This system property is read in a static block inside of Snappy code.

I've added a "best effort" fix for this where we try to set the system
property in a static block before Snappy reads it. I've tested it and it does
work, but it relies on static initialization order which is brittle. I.e.
if user code accesses Snappy libraries first this could not-work.

An alternative work-around for users is to explicitly set

    org.xerial.snappy.tempdir

themselves through Spark's java options. I also filed a bug upstream with
Snappy-java to ask them for better behavior here:

xerial/snappy-java#84

I think this is worth merging because in many cases it will fix the issue and
at worst it's a no-op.
@pwendell
Copy link
Contributor Author

@JoshRosen maybe you could take a glance at this. It's one of the few remaining blockers for 1.1 in core.

@SparkQA
Copy link

SparkQA commented Aug 17, 2014

QA tests have started for PR 1991 at commit 96e5e6a.

  • This patch merges cleanly.

@pwendell pwendell closed this Aug 17, 2014
@pwendell
Copy link
Contributor Author

Replaced by a 1.1-specific patch in #1994

@SparkQA
Copy link

SparkQA commented Aug 17, 2014

QA tests have finished for PR 1991 at commit 96e5e6a.

  • This patch passes unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants