Skip to content

[SPARK-12429][Streaming][Doc]Add Accumulator and Broadcast example for Streaming #10385

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 4 commits into from
Closed

[SPARK-12429][Streaming][Doc]Add Accumulator and Broadcast example for Streaming #10385

wants to merge 4 commits into from

Conversation

@zsxwing
Copy link
Member

@zsxwing zsxwing commented Dec 18, 2015

This PR adds Scala, Java and Python examples to show how to use Accumulator and Broadcast in Spark Streaming to support checkpointing.

@zsxwing
Copy link
Member Author

zsxwing commented Dec 18, 2015

@tdas could you take a look before I start to add Java and Python examples?

@SparkQA
Copy link

SparkQA commented Dec 18, 2015

Test build #48018 has finished for PR 10385 at commit 9928ca5.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * public class JavaTwitterHashTagJoinSentiments\n * case class UnresolvedAlias(child: Expression, aliasName: Option[String] = None)\n

BenFradet
BenFradet reviewed Dec 18, 2015
View reviewed changes
docs/streaming-programming-guide.md Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd say: "in Spark Streaming. If you enable checkpointing and use an Accumulator or Broadcast as well, you**'ll** have to create ..."

@zsxwing zsxwing changed the title [SPARK-12429][Streaming][Doc]Add Accumulator and Broadcast Scala example for Streaming [SPARK-12429][Streaming][Doc][WIP]Add Accumulator and Broadcast example for Streaming Dec 18, 2015
@SparkQA
Copy link

SparkQA commented Dec 19, 2015

Test build #48038 has finished for PR 10385 at commit 78d15bd.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@zsxwing zsxwing changed the title [SPARK-12429][Streaming][Doc][WIP]Add Accumulator and Broadcast example for Streaming [SPARK-12429][Streaming][Doc]Add Accumulator and Broadcast example for Streaming Dec 21, 2015
@SparkQA
Copy link

SparkQA commented Dec 21, 2015

Test build #48120 has finished for PR 10385 at commit 9e241e7.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * class JavaWordBlacklist\n * class JavaDroppedWordsCounter\n

@zsxwing
Copy link
Member Author

zsxwing commented Dec 21, 2015

Added Java and Python examples.

@BenFradet
Copy link
Contributor

lgtm

tdas
tdas reviewed Dec 22, 2015
View reviewed changes
docs/streaming-programming-guide.md Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Accumulators and Broadcast variables

@tdas
Copy link
Contributor

tdas commented Dec 22, 2015

Small comments, otherwise LGTM.

zsxwing
zsxwing reviewed Dec 23, 2015
View reviewed changes
docs/programming-guide.md
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also fixed the broken Accumulator link in programming-guide.md

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

@SparkQA
Copy link

SparkQA commented Dec 23, 2015

Test build #48222 has finished for PR 10385 at commit 455968a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * class JavaWordBlacklist\n * class JavaDroppedWordsCounter\n

asfgit pushed a commit that referenced this pull request Dec 23, 2015
@zsxwing @tdas
[SPARK-12429][STREAMING][DOC] Add Accumulator and Broadcast example f…
942c057 
…or Streaming

This PR adds Scala, Java and Python examples to show how to use Accumulator and Broadcast in Spark Streaming to support checkpointing.

Author: Shixiong Zhu <[email protected]>

Closes #10385 from zsxwing/accumulator-broadcast-example.

(cherry picked from commit 20591af)
Signed-off-by: Tathagata Das <[email protected]>
@asfgit asfgit closed this in 20591af Dec 23, 2015
@zsxwing zsxwing deleted the accumulator-broadcast-example branch December 23, 2015 05:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@zsxwing @SparkQA @BenFradet @tdas