org.apache.spark.examples.streaming

Class KinesisWordCountASL

Object

org.apache.spark.examples.streaming.KinesisWordCountASL

All Implemented Interfaces:

Logging

public class KinesisWordCountASL
extends Object
implements Logging

Kinesis Spark Streaming WordCount example.

See http://spark.apache.org/docs/latest/streaming-kinesis.html for more details on the Kinesis Spark Streaming integration.

This example spins up 1 Kinesis Worker (Spark Streaming Receiver) per shard for the given stream. It then starts pulling from the last checkpointed sequence number of the given and .

Valid endpoint urls: http://docs.aws.amazon.com/general/latest/gr/rande.html#ak_region

This code uses the DefaultAWSCredentialsProviderChain and searches for credentials in the following order of precedence: Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY Java System Properties - aws.accessKeyId and aws.secretKey Credential profiles file - default location (~/.aws/credentials) shared by all AWS SDKs Instance profile credentials - delivered through the Amazon EC2 metadata service

Usage: KinesisWordCountASL is the name of the Kinesis stream (ie. mySparkStream) is the endpoint of the Kinesis service (ie. https://kinesis.us-east-1.amazonaws.com)

Example: $ export AWS_ACCESS_KEY_ID= $ export AWS_SECRET_KEY= $ $SPARK_HOME/bin/run-example \ org.apache.spark.examples.streaming.KinesisWordCountASL mySparkStream \ https://kinesis.us-east-1.amazonaws.com

Note that number of workers/threads should be 1 more than the number of receivers. This leaves one thread available for actually processing the data.

There is a companion helper class below called KinesisWordCountProducerASL which puts dummy data onto the Kinesis stream. Usage instructions for KinesisWordCountProducerASL are provided in that class definition.

Constructor Summary

Constructors

Constructor and Description
KinesisWordCountASL()

Method Summary

Methods

Modifier and Type	Method and Description
static void	main(String[] args)

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

KinesisWordCountASL

public KinesisWordCountASL()

Method Detail

main

public static void main(String[] args)