[SPARK-6943][WIP][Alternative] Show RDD DAG visualization on stage UI #5728
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 14 commits
April 16, 2015 17:33
This commit provides a mechanism to set and unset the call scope around each RDD operation defined in RDD.scala. This is useful for tagging an RDD with the scope in which it is created. This will be extended to similar methods in SparkContext.scala and other relevant files in a future commit.
This includes the scope field that we added in previous commits, and the parent IDs for tracking the lineage through the listener API.
It turns out that the previous scope information is insufficient for producing a valid dot file. In particular, the scope hierarchy was missing, but crucial to differentiate between a parent RDD being in the same encompassing scope and it being in a completely distinct scope. Also, unique scope identifiers are needed to simplify the code significantly. This commit further adds the translation logic in a UI listener that converts RDDInfos to dot files.
The previous "working" implementation frequently ran into NotSerializableExceptions. Why? ClosureCleaner doesn't like closures being wrapped in other closures, and these closures are simply not cleaned (details are intentionally omitted here). This commit reimplements scoping through annotations. All methods that should be scoped are now annotated with @RDDScope. Then, on creation, each RDD derives its scope from the stack trace, similar to how it derives its call site. This is the cleanest approach that bypasses NotSerializableExceptions with least significant limitations.
Just a small code re-organization.
Before this commit, this patch relies on a JavaScript version of GraphViz that was compiled from C. Even the minified version of this resource was ~2.5M. The main motivation for switching away from this library, however, is that this is a complete black box of which we have absolutely no control. It is not at all extensible, and if something breaks we will have a hard time understanding why. The new library, dagre-d3, is not perfect either. It does not officially support clustering of nodes; for certain large graphs, the clusters will have a lot of unnecessary whitespace. A few in the dagre-d3 community are looking into a solution, but until then we will have to live with this (minor) inconvenience.
For instance, this adds ability to throw away old stage graphs.
andrewor14
changed the title
|
Test build #31077 has finished for PR 5728 at commit
|
|
Test build #726 has finished for PR 5728 at commit
|
|
Test build #31082 has finished for PR 5728 at commit
|
|
retest this please |
|
Test build #31102 has finished for PR 5728 at commit
|
|
Closing this in favor of #5729. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closing this because the necessary closure cleaner fixes that #5729 blocks on are now merged.