Name: ibm-cloud-functions-data-processing-message-hub
Owner: International Business Machines
Description: Create a serverless, event-driven application with Apache OpenWhisk on IBM Cloud Functions that executes code in response to messages or to handle streams of data records from Apache Kafka or IBM Message Hub.
Created: 2017-02-10 17:09:03.0
Updated: 2018-05-15 05:03:53.0
Pushed: 2018-03-11 14:39:28.0
Homepage: https://developer.ibm.com/code/patterns/respond-messages-handle-streams
Size: 1199
Language: Shell
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
Read this in other languages: ???.
This project shows how serverless, event-driven architectures execute code in response to messages or to handle streams of data records.
The application demonstrates two IBM Cloud Functions (based on Apache OpenWhisk) that read and write messages with IBM Message Hub (based on Apache Kafka). The use case demonstrates how actions work with data services and execute logic in response to message events.
One function, or action, is triggered by message streams of one or more data records. These records are piped to another action in a sequence (a way to link actions declaratively in a chain). The second action aggregates the message and posts a transformed summary message to another topic.
You should have a basic understanding of the Cloud Functions/OpenWhisk programming model. If not, try the action, trigger, and rule demo first.
Also, you'll need an IBM Cloud account and the latest OpenWhisk command line tool (wsk) installed and on your PATH.
As an alternative to this end-to-end example, you might also consider the more basic “building block” version of this sample.
Log into the IBM Cloud, provision a Message Hub instance, and name it kafka-broker. On the “Manage” tab of your Message Hub console create two topics: in-topic and out-topic. On the “Service credentials” tab make sure to add a new credential named Credentials-1.
Copy template.local.env to a new file named local.env and update the KAFKA_INSTANCE, SRC_TOPIC, and DEST_TOPIC values for your instance if they differ.
deploy.sh is a convenience script reads the environment variables from local.env and creates the OpenWhisk actions, triggers, and rules on your behalf. Later you will run the commands in the file directly.
ploy.sh --install
Note: If you see any error messages, refer to the Troubleshooting section below. You can also explore Alternative deployment methods.
Open one terminal window to poll the logs:
activation poll
Send a message with a set of events to process.
oduce a message, will trigger the sequence of actions
=$( base64 events.json | tr -d '\n' | tr -d '\r' )
action invoke Bluemix_${KAFKA_INSTANCE}_Credentials-1/messageHubProduce \
param topic $SRC_TOPIC \
param value "$DATA" \
param base64DecodeValue true
Use deploy.sh again to tear down the OpenWhisk actions, triggers, and rules. You will recreate them step-by-step in the next section.
ploy.sh --uninstall
This section provides a deeper look into what the deploy.sh script executes so that you understand how to work with OpenWhisk triggers, actions, rules, and packages in more detail.
Create the message-trigger trigger using the Message Hub packaged feed that listens for new messages. The package refresh will make the Message Hub service credentials and connection information available to OpenWhisk.
package refresh
trigger create message-trigger \
feed Bluemix_${KAFKA_INSTANCE}_Credentials-1/messageHubFeed \
param isJSONData true \
param topic ${SRC_TOPIC}
Upload the receive-consume action as a JavaScript action. This downloads messages when they arrive via the trigger.
action create receive-consume actions/receive-consume.js
Upload the transform-produce action. This aggregates information from the action above, and sends a summary JSON string back to another Message Hub topic.
action create transform-produce actions/transform-produce.js \
param topic ${DEST_TOPIC} \
param kafka ${KAFKA_INSTANCE}
Declare a linkage between the receive-consume and transform-produce in a sequence named message-processing-sequence.
action create message-processing-sequence --sequence receive-consume,transform-produce
Declare a rule named message-rule that links the trigger message-trigger to the sequence named message-processing-sequence.
rule create message-rule message-trigger message-processing-sequence
oduce a message, will trigger the sequence
=$( base64 events.json | tr -d '\n' | tr -d '\r' )
action invoke Bluemix_${KAFKA_INSTANCE}_Credentials-1/messageHubProduce \
param topic $SRC_TOPIC \
param value "$DATA" \
param base64DecodeValue true
Check for errors first in the OpenWhisk activation log. Tail the log on the command line with wsk activation poll or drill into details visually with the monitoring console on the IBM Cloud.
If the error is not immediately obvious, make sure you have the latest version of the wsk CLI installed. If it's older than a few weeks, download an update.
property get --cliversion
Download latest wskdeploy from the release page of openwhisk-wskdeploy project.
clone https://github.com/IBM/ibm-cloud-functions-data-processing-message-hub.git
bm-cloud-functions-data-processing-message-hub
package refresh
eploy
eploy undeploy
You can also use the following button to clone a copy of this repository and deploy to the IBM Cloud as part of a DevOps toolchain. Supply your OpenWhisk and Message Hub credentials under the Delivery Pipeline icon, click Create, then run the Deploy stage for the Delivery Pipeline.
This project was inspired by and reuses significant amount of code from this article.