[SPARK-7396][Streaming][Example] Update KafkaWordCountProducer to use new Producer API

[jerryshao]

Otherwise it will throw exception:

Exception in thread "main" kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.
    at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
    at kafka.producer.Producer.send(Producer.scala:77)
    at org.apache.spark.examples.streaming.KafkaWordCountProducer$.main(KafkaWordCount.scala:96)
    at org.apache.spark.examples.streaming.KafkaWordCountProducer.main(KafkaWordCount.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:623)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[SparkQA]

Test build #31965 has started for PR 5936 at commit 270bbe2.

[SparkQA]

Test build #31965 timed out for PR 5936 at commit 270bbe2 after a configured wait of 150m.

[AmplabJenkins]

Merged build finished. Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31965/
Test FAILed.

[srowen]

CC @koeninger

[srowen]

Jenkins, retest this please

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[SparkQA]

Test build #31972 has started for PR 5936 at commit 270bbe2.

[SparkQA]

Test build #31972 has finished for PR 5936 at commit 270bbe2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Merged build finished. Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31972/
Test PASSed.

[koeninger]

Not that there's necessarily anything wrong with updating the producer api being used...
.. but I don't see how that exception has anything to do with the version of the producer api, or any changes to spark's kafka version.

KafkaWordCountProducer uses no spark code. It only uses kafka.producer._ (which still exists in the 0.8.2.1 artifact), and java.util.properties.

I just ran the old version of KafkaWordCountProducer against a local kafka install with no problems.

@jerryshao how were you getting that exception?

[koeninger]

FWIW I ran the new version, and it works as well, I'm just concerned about how that exception was caused.

[jerryshao]

Hi @koeninger , I just run this example with no additional configuration for the first time, I met such exception when the example is started. But again I run it, there's no such exception, instead I met some of warning logs like:

WARN common.AppInfo$: Can't read Kafka version from MANIFEST.MF. Possible cause: java.lang.NullPointerException

I guess this was introduced by the upgrading of Kafka version, since I didn't meet this before. So I rewrite this example with new Producer API, and this warning log is gone.

[jerryshao]

Anyway I think it's meaningful to update this example accordingly.

[tdas]

LGTM. Merging this. Thanks!