[SPARK-10382] Make example code in user guide testable

[yinxusen]

A POC code for making example code in user guide testable.

@mengxr We still need to talk about the labels in code.

[SparkQA]

Test build #43703 has finished for PR 9109 at commit b0c1568.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

Do we still need this gem if we have include_example.rb?

[yinxusen]

@mengxr I add a POC example of ml#Pipeline. But I am not sure that I select right files. But it demos what we need.

[yinxusen]

It requires Pygments. So install Pygments before compile it. Actually it is the same with what defined in _config.yml.

[SparkQA]

Test build #43932 has finished for PR 9109 at commit 0dbcfe9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #43931 has finished for PR 9109 at commit 830d1b9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

@yinxusen It is possible that we only replace the markdown text and let highlight handle the syntax highlighting? Another minor comment is removing the // $example on$ and // $example off$ in the output file. Could you also include the imports in the example code? Basically, having two example blocks from the source file. Just want to make sure it works.

Ignore my first comment. Jekyll uses pygments too for syntax highlighting. So we are not introducing new dependencies.

[mengxr]

@yinxusen Could you update the PR to support multiple example code blocks in one file? It is quite comment to have imports + example code - boilerplate code in the user guide. If you show that it works, we can remove the example and then merge this PR.

[yinxusen]

@mengxr Yes, I'll do it ASAP.

[yinxusen]

@mengxr You can check the POC now. If it is OK, I will remove the example. Next time we can enjoy easy code sample insertion in docs.

[SparkQA]

Test build #44036 has finished for PR 9109 at commit 78c07ae.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

Could you add Apache header? See copy_api_dirs.rb.

[mengxr]

Yes, it looks good. Please remove the example, add Apache header, then remove [POC] from the PR title. It should be good to go.

[mengxr]

nit: if we still want to follow Spark code style, it should be select { instead of select{

[yinxusen]

Sure, will fix it.

[SparkQA]

Test build #44212 has finished for PR 9109 at commit 8557977.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

LGTM. Merged into master. Thanks! This makes the example code much easier to check. Could you make one JIRA and submit a PR to replace some example code using this? Then we can create more JIRAs, and ask community to help.

[yinxusen]

@mengxr Sure I can do that.