Name: lambda-refarch-streamprocessing
Owner: Amazon Web Services - Labs
Owner: AWS Samples
Description: Serverless Reference Architecture for Real-time Stream Processing
Created: 2015-09-30 23:56:06.0
Updated: 2018-01-14 14:25:59.0
Pushed: 2017-05-17 14:53:49.0
Size: 253
Language: JavaScript
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
README Languages: DE | ES | FR | IT | JP | KR | PT | RU | CN | TW
You can use AWS Lambda and Amazon Kinesis to process real-time streaming data for application activity tracking, transaction order processing, click stream analysis, data cleansing, metrics generation, log filtering, indexing, social media analysis, and IoT device data telemetry and metering. The architecture described in this diagram can be created with an AWS CloudFormation template.
The template does the following:
Creates a Kinesis Stream
Creates a DynamoDB table named <stackname>-EventData
Creates Lambda Function 1 (<stackname>-DDBEventProcessor) which receives records from Kinesis and writes records to the DynamoDB table
Creates an IAM Role and Policy to allow the event processing Lambda function read from the Kinesis Stream and write to the DynamoDB table
Creates an IAM user with permission to put events in the Kinesis stream together with credentials for the user to use in an API client
Step 1 - Create an AWS CloudFormation stack with the template. The AWS CloudFormation template completely automates the building, deployment, and configuration of all the components of the application.
Step 2 - Once the AWS CloudFormation stack has successfully been created you can do select the Outputs tab and see the AWS parameters needed in the demo Twitter client in the steps below.
Step 3 - To run the example application you need to update the code with AWS and Twitter information. Open producer/twitter2kinesis.py in a text editor.
Step 4 - To access the Twitter API you need to get access tokens. Make sure you have these available and enter the information in the following parameters:
The Twitter API parameters
umer_key = ""
umer_secret = ""
ss_token_key = ""
ss_token_secret = ""
Step 5 - Enter the values for the AWS credentials and Amazon Kinesis stream name. This is the information from the Outputs tab of the CloudFormation template you got in step 2:
AWS parameters - from the Outputs tab of the CloudFormation template
ss_key = ""
et_access_key = ""
on = ""
am_name = ""
Step 6 - Finally, before running the example code, you need Python installed together with the Python modules boto3 and TwitterAPI. If you don't have the modules already, install them using pip:
install -r requirements.txt
Step 1 - Run the producer/twitter2kinesis.py Python application from the command line to start sending tweets into the Kinesis stream.
on twitter2kinesis.py
Step 2 - In the Amazon DynamoDB management console, select the table named <stackname>-EventData and explore the records.
To remove all created resources, delete the AWS CloudFormation stack.