Clever/kinesis-to-firehose

Name: kinesis-to-firehose

Owner: Clever

Description: Consumes records from Kinesis and writes to Firehose

Created: 2017-03-17 21:35:48.0

Updated: 2018-05-11 23:36:44.0

Pushed: 2018-05-11 23:36:45.0

Homepage: null

Size: 152

Language: Go

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

kinesis-to-firehose

Consumes records from Kinesis and writes to Firehose.

Running the Consumer

To build and run the consumer, set required env vars. Then run, make run.

Env vars include:

• Kinesis input stream
• Firehose output stream
• logfile
• rate limit (allows throttling the # of records-per-second to read from each Kinesis shard)

SIS_AWS_REGION=us-west-1 \
SIS_STREAM_NAME=kinesis-test \
HOSE_AWS_REGION=us-west-2 \
HOSE_STREAM_NAME=firehose-test \
FILE=/tmp/kcl_stderr \
_LIMIT=100 \
OY_ENV="local" \
 run

This will download the jar files necessary to run the KCL, and then launch the KCL communicating with the consumer binary.

Running at Clever

You can also use ark to run locally, via ark start --local.

Note: In addition to output from the process, you may want to run tail -f /tmp/kcl_stderr to view more logs written to file. These logs aren't written to stdout/stderr, since KCL uses those for communication.