Name: iot-device-trend-analysis
Owner: International Business Machines
Description: Setup and create a web application to visualize IoT device data
Created: 2018-04-03 16:44:58.0
Updated: 2018-05-02 20:00:17.0
Pushed: 2018-05-02 20:00:16.0
Homepage: https://developer.ibm.com/code/patterns/create-a-web-app-to-analyze-and-visualize-iot-device-data/
Size: 7035
Language: Python
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
In this code pattern, we will setup and create a web application to visualize IoT device data and view trends and stats of device fields across days. The IoT industries are looking for ways to analyze the use of IoT devices, and would like better understand the usage of devices. This code pattern will demonstrate using IBM solutions to read and store IoT device data, and then build an application on top of it. The code pattern uses services offered on IBM Cloud such as IBM Watson IoT Platform and Cloudant DB, in addition to deploying the application to the IBM Cloud. Once setup, the application displays the time-series field data as plots, showing device data trends and statistical analysis.
First, we will create an IBM Watson IoT Platform service which provides a platform to manage IoT devices and the data being sent across those devices. This code pattern provides directions on creating dummy IoT devices in the Watson IoT Platform, and then simulating data for those devices using a simulation feature in the IoT platform. We are interested in certain fields as part of payload for the devices to come through, as the application is designed to read those fields.
Next, we will store the devices' fields data from the Watson IoT platform into Cloudant DB. The Cloudant DB is a NoSQL JSON document store that is optimized for handling heavy workloads of read and write in the cloud. In this code pattern, we will walk through the process of creating a Cloudant DB service on the IBM Cloud, and retrieve credentials to access it through applications. The Cloudant DB interface can be launched through the service, which allows to directly manage and view our data. Once our Cloudant DB is created, we will go through steps on configuring Watson IoT Platform to store the IoT device data into it.
Once we have completed the setup, with our data coming through the Watson IoT Platform and into the Cloudant DB, we are ready to run the application to view the visualizations of data. The application first requests the user to create a dataset based on the data now being stored in the Cloudant DB. This can be viewed as our first layer of filtering assuming the IoT data scientist/analyst would like to focus on particular devices and dates. The device data could get large so it can be divided into different datasets for more focused analysis.
After creating a dataset, the user can use this application to create different visualizations for the data. This includes viewing the raw data for a device's data value across time, and then creating hourly trends plot that capture the behavior of the data per hour. The hourly trends looking at the max, min and moving average of the data which allows to capture the general trend of the field across time and capture any anomalies. Other analytical plots include comparing device's field stats (min, median, max) across days and viewing correlation between fields for a device across days. The Plotly.js library is used to create these visualizations of the data based on user inputs. The Plotly.js provides a great way to display data visually through numerous plot types and ability to manage and enhance on plots.
This web application is built on Python Flask framework, with Javascript(JS) and HTML frontend. The Python backend allows to create libraries (plotdata.py and dataset.py), which parse and analyze the JSON data from the Cloudant database. The frontend application provides a separate page for each plot type, and an associated JS script for each page in the scripts folder under static (i.e deviceCorrelationAnalysis.js). The user inputs are sent through an Ajax call to the Python backend (run.py) which retrieves the respective data and returns it. This includes the data to be plotted, which is captured by the JS script for the page and shows how to plot using the Plotly.js library. These plots are then displayed through the HTML page for the plot.
This code pattern can be useful to developer's looking to enhance IoT analysis skill, for IoT data scientists/analysts looking to create their own customized application, and anyone with interest in creating a visual analytical application.
When the reader has completed this code pattern, they will understand how to:
Use the Deploy to IBM Cloud button OR create the services and deploy the application locally.
You can deploy the application and create services directly on IBM Cloud using the 'Deploy to IBM Cloud' button. This is alternative to creating services individually and then deploying the application using cloud foundry.
Region, Organization and Space where you would like to deploy the application. Then click Deploy.
Next, you can view your toolchain. We can see the progress of deployment by clicking Delivery Pipeline.
The delivery pipeline will show the status of each stage. You can view logs and history if any issues. Once all stages pass, then your application has been successfully deployed.
Connections on the left menu.
Once your application and services are created in IBM Cloud, you will need to do the following to setup your IoT data in the Cloudant DB:
Once you have completed these steps, you should be able to create dataset in your application to start viewing trends and analysis of the IoT device data.
NOTE: These steps are only needed when running locally instead of using the
Deploy to IBM Cloudbutton.
The IBM Watson IoT Platform service provides a dashboard to manage and configure devices, read data transmitted by the devices and numerous features to managing IoT devices. First, we would like to create the IBM Watson IoT Platform service in IBM Cloud.
catalog and choosing Internet of things. Here we would like to pick the Internet of Things Platform
region, organization and space.
Lite plan for this code pattern to get started with Internet of Things Platform.
Launch button. Additionally, you can access documentation regarding the IoT Platform by going to Docs button.
Once you have created your IBM Watson IoT Platform service, you are ready to setup devices to transmit data. In this section, we will create a dummy device type, then create devices using the device type and then simulate data for those devices using the simulator feature in the IBM Watson IoT Platform.
The application in the code pattern is designed to handle payload as the following json:
eviceId": "d6a82126d",
imestamp": "2018-04-04T11:36:31.046Z",
ata": {
"connections": 58,
"deviceCount": 78,
"activeClients": 68
The deviceId and timestamp are submitted automatically by the IoT Platform, however we would like to configure the devices to send data for connections, deviceCount and activeClients. The application will plot and analyze these fields over time per device.
Before creating devices, we would need to create a device type. Device types are intended to be groups of devices which share common characteristics.
device type, first go to the IBM Watson IoT Platform options on the left side of your dashboard and choose Devices.
Device Types on your screen, and click the Add Device Type button on the top right.
Identity tab, choose Device for type, as this code pattern is focued on devices. Provide a Name and an optional Description, and then click Next.
On the Device Information, you can add attributes to the device type. These attributes are optional. Then click Next and Done. The device type should be registered and you should see a similar screen as below.
Now, we will add device to the device type we created.
Register Devices under your device type. Or go to Devices in the menu, and choose Add Device in the top right corner.
Identity tab, choose the device type and provide a Device ID. Then click Next.
Device Information and Add to Groups. You can go through the steps by clicking Next. Under the Security tab, provide an Authentication Token. Click Next and then Done.
Now we would like to setup our devices to transmit data. Ideally you would to like read in real data or can have scripts to simulate the data (see Additional Resources at the bottom for more information). For this code pattern, we will go through steps to simulate data sent through by the devices, using simulator feature part of the Watson IoT Platform dashboard.
The IoT Platform provides a simulator feature to simulate the payload transmitted by the device. In this step, we will turn on the simulator and simulate numbers for the fields to be transmitted by the device.
Settings option:
Experimental Features section. And toggle enable the Active Device Simulator.
Simulations running tab at the bottom of screen in your IoT Dashboard. Bring that up to create the simulation, by selecting Add First Simulation.
First choose a Device Type. Next, we will define the Payload to include the fields for this code pattern: connections, deviceCount and activeClients. The simulated data can look similar to below where it is generating random numbers for the fields, transmitting data every minute. You can provide an Event type name. Once done, click Save.
dropdown arrow next to your device type, and choose Use Registered Device. Here you will see your devices and can choose the device you would like to simulate.
Devices, and then go to the Recent Events tab. Here you can view the live data being transmitted for the device.
Now we are ready to store the IoT device data into a database. The IBM Cloud provide several database options including relational and non-relational options. For our data with the json payload, we would like to choose a NoSQL database such as Cloudant DB. The Cloudant DB provides for heavy read/write for an application, easy retrieval of data through API calls and a great interface to manage data directly. In this section, we will create a Cloudant DB on the IBM Cloud.
catalog and under Data & Analytics, you can find Cloudant NoSQL DB service.
Service name, and then choose the region, organization and space
Lite plan to get started. Click Create at the bottom left to create the service in your IBM Cloud.
Service credentials option on the left. Here you can generate credentials for the service by clicking on New credentials. Go ahead and create credentials for the service as we will need those for our application.
Now, we will configure our IBM Watson IoT Platform to setup Cloudant DB as daily store for our device data. The IBM Watson IoT Platform provides a straight forward way to create daily buckets for our device data. Once we are configured, we will have a database for each day in our Cloudant DB with our device data.
Extensions.
Historical Data Storage, choose Setup. Here you will see all your databases in your IBM Cloud. Find the Cloudant DB you created to store the device data.
Bucket Interval for Day and then click Done
Once the setup is complete with data coming into our Cloudant DB, we can run the application to start viewing the device data values and look for trends. To run the application, we will clone the repo, add the Cloudant credentials and then run the application through terminal.
In a directory of your choice, clone the repo:
clone https://github.com/IBM/iot-device-trend-analysis
Create a .env file in the root directory of your clone of the project repository by copying the sample .env.example file using the following command:
.env.example .env
You will need to update the .env file with credentials for the Cloudant DB which you created.
The .env file will look something like the following:
loudant NoSQL database
OUDANT_USERNAME=
OUDANT_PASSWORD=
OUDANT_URL=
Now you are ready to run your application. Go into this project's root directory
pip install -r requirements.txt to install the app's dependenciespython run.py
The application is designed to create dataset from our Cloudant DB which we would like to plot and analyze, and then analyze through plots based on user inputs. The application provides to view the raw data and view analysis on the data including trends and statistical plots. In this section, we will look at creating dataset, analyzing the data through plots and look at the code.
Before analyzing the data, you will need to define a dataset. This includes dates and device Ids from your Cloudant storage. You can do this through the app by going to Create Dataset link on the main page. This pulls all the dates and deviceIds directly from the Cloudant database. You can make selection for these fields and give a name to your dataset.
Or you can manually edit the datasets.json to fill in your database info with dates, deviceIDs, IoT database initial and name for the dataset. The file should look like below:
urrentDataset": "New1",
atasets": [
{
"databaseName": "iotp_nmghmm_default_",
"dataset": "New1",
"dates": [
"2018-01-28",
"2018-01-29",
"2018-01-30"
],
"deviceIds": [
"830ab1575",
"bc1a6275a",
"1d0c388d0",
"51bdc89e4"
]
}
Once you have defined your dataset, you are ready to analyze your data through the different options present on the homepage. Each analysis will ask for device(s) and date(s) to generate the plot. Once your plot is generated, you can explore different Plotly interactive options on the top right which can allow to download the plot, change the plot type and other actions. The Hourly Stats and Trends options will allow to view how the hourly max, min, and average are behaving across time, with trend analysis showing change in the values per hour.
This web application is built using Python Flask framework. The repo consists of Python functions to retrieve the JSON data from Cloudant DB, and Javascript frontend to use the Plotly.js library. Here we'll give a summary of the code files in the repo.
run.py: This routes the pages on the web application and manages all the GET and POST commands made using Ajax on the Javascript scripts. These calls will then call the respective functions in plotdata or dataset library to return the requested data to the web frontend.
plotdata.py: This library contains functions that pulls data from Cloudant DB through API call and parses the data according to the function and input. These functions will return a list of JSON objects, where each object will have timeStamp field and data fields that we are interested in plotting. This library includes functions such as Device_data_across_days which pull raw data per device id for a start and end date, and Hourly_stats_trends which will create a JSON to plot the hourly stats and trends
For example the JSON object for Device_data_across_days will be as follow with the raw data:
"deviceCount": 85.0,
"timeStamp": "2018-01-16T10:35:41.635Z",
"connections": 43.0,
"deviceID": "19ca0a0b6",
"activeClients": 65.0
The Hourly_stats_trends function will return JSON object with the max, min, average, and change in these value for every hour (i.e slope). In addition it provides a time stamp for that hour right at 30 mins as the middle point for that hour.
"sumActiveClients": 253.0,
"plotTimeStamp": "2018-01-16T01:30:00.000Z",
"maxActiveClients": 84.0,
"date": "2018-01-16",
"avgSlopeLastHour": -22.4,
"minActiveClients": 26.0,
"maxSlopeLastHour": -18.0,
"hour": "01",
"minSlopeLastHour": -9.0,
"countEntries": 5,
"avgActiveClients": 50.6,
"deviceID": "19ca0a0b6"
dataset.py: This library provides functions to manage datasets.json, which includes pulling dataset information, adding dataset and setting the active dataset. The functions updates and retrieve the datasets.json file accordingly.
JS scripts (i.e static/scripts/deviceDataPerDay.js): There is a JS script for each page of this application. The javascript makes GET calls to update the page with dropdown options, ensures valid user input for the page, and makes the Ajax call to get the JSON data to plot. Then it uses Plotly.js library to create a plot for that page.
To create Plotly.js plots, we will first create traces for the plot, defining our x and y axis, and type and name for the trace. The type determines what type of plot we would like i.e scatter, bar, boxplot. Next, we define data for the plots, as an array of these traces, and layout with the title for plot.
fine traces
activeClientsTrace = {
timeStampArray,
activeClientsArray,
pe: "scatter",
me: "activeClients"
deviceCountTrace = {
timeStampArray,
deviceCountArray,
pe: "scatter",
me: "deviceCount",
sible: "legendonly"
connectionsTrace = {
timeStampArray,
connectionsArray,
pe: "scatter",
me: "connections",
sible: "legendonly"
data = [activeClientsTrace, deviceCountTrace, connectionsTrace];
layout = {
tle: "Device " + id + " from " + startDate + " to " + endDate
With your data and layout defined, you are ready to call the Plotly library to create the plot. The plotly_div1 would be an id in your html page.
eate plot
ly.newPlot('plotly_div1', data, layout)
hen(
function(gd) {
Plotly.toImage(gd, {
height: 500,
width: 500
})
});
html (i.e /templates/devicePerDay.html): The html code is provided for each page, which is primarily to retrieve user input and display plots. It provides the css and JS with the ids and class for enhancing display and capturing user inputs.To deploy the application to IBM Cloud, you will need Cloud Foundary CLI installed, so you can use command-line to deploy the application. The configuration for the deployment will be in manifest.yml file, which we'll update first.
Here we will update the manifest.yml file in the folder and replace with a unique name for your application. The name that you specify determines the application's URL, such as your-application-name.mybluemix.net. Additionally - update the service names so they match what you have in IBM Cloud. The relevant portion of the manifest.yml file looks like the following:
ications:
th: .
mory: 256M
stances: 1
main: mybluemix.net
me: Iot-Analytics
st: iot-device-analytics
sk_quota: 1024M
ildpack: python_buildpack
rvices:
cloudant
The memory of 256M will allow you to run the application with the Lite plan. However as data gets larger with the retrieval, you may have to increase memory to 512M or 1024M.
To deploy the application to IBM Cloud, we will use Cloud Foundry CLI. First login to your cloud foundry account
ogin
Next, in the command line use cloud foundry command to push the application to IBM Cloud:
ush
This will take a few minutes and then provide the status of deployment. You can then go to the url to view the application.
This code pattern can be extended in several ways:
To troubleshoot your IBM Cloud application, use the logs. To see the logs, run:
ogs <application-name> --recent