Name: RADAR-Backend
Owner: RADAR-CNS
Owner: RADAR-CNS
Description: Kafka backend for processing device data
Created: 2016-09-21 10:20:59.0
Updated: 2017-12-19 07:10:58.0
Pushed: 2018-02-01 16:46:43.0
Homepage: null
Size: 4664
Language: Java
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
RADAR-Backend is a Java application based on Confluent Platform to standardize, analyze and persist data collected by RADAR-CNS data sources. It supports the backend requirements of RADAR-CNS project. The data is produced and consumed in Apache Avro format using the schema stored inside the RADAR-CNS schema repository.
RADAR-Backend provides an abstract layer to monitor and analyze streams of wearable data and write data to Hot or Cold storage. The Application Programming Interfaces (APIs) of RADAR-Backend makes the process of to integrating additional topics, wearable devices easier. It currently provides MongoDB as the Hot-storage and HDFS data store as the Cold-storage. They can be easily tuned using property files. The stream-monitors monitor topics and notify users (e.g. via emails) under given circumstances.
The following are the prerequisites to run RADAR-Backend on your machine:
Install the dependencies mentioned above.
Clone RADAR-Backend repository.
clone https://github.com/RADAR-CNS/RADAR-Backend.git
Build the project from project directory
vigate to project directory
ADAR-Backend/
ean
adlew clean
ild
adlew build
The build process creates separate jar files for each component. Built jars are located under build/libs folder.
The RADAR command-line has three subcommands: stream, monitor and mock. The stream command will start all streams, the monitor command will start all monitors, and the mock command will send mock data to the backend. Before any of these commands are issued, start the Confluent platform with the zookeeper, kafka, schema-registry and rest-proxy components. Put the build/libs/radarbackend-1.0.jar and radar.yml in the same folder, and then modify radar.yml:
In radar.yml, Specify in which mode you want to run the application. There are two alternatives: standalone and high_performance. The standalone starts one thread for each streams without checking the priority, whereas the high_performance starts as many thread as the related priority value
If auto.create.topics.enable is false in your Kafka server.properties, before starting you must create the topics manually. Create the following topics for Empatica E4 Streams
android_empatica_e4_acceleration
android_empatica_e4_acceleration_output
android_empatica_e4_battery_level
android_empatica_e4_battery_level_output
android_empatica_e4_blood_volume_pulse
android_empatica_e4_blood_volume_pulse_output
android_empatica_e4_electrodermal_activity
android_empatica_e4_electrodermal_activity_output
android_empatica_e4_heartrate_output
android_empatica_e4_inter_beat_interval
android_empatica_e4_inter_beat_interval_output
android_empatica_e4_sensor_status
android_empatica_e4_sensor_status_output
android_empatica_e4_temperature
android_empatica_e4_temperature_output
android_phone_usage_event
android_phone_usage_event_output
Run radarbackend.jar with configured radar.yml and stream argument
-jar radarbackend-1.0.jar -c path/to/radar.yml stream
The phone usage event stream uses an internal cache of 1 million elements, which may take about 50 MB of memory. Adjust org.radarcns.stream.phone.PhoneUsageStream.MAX_CACHE_SIZE to change it.
To get email notifications for Empatica E4 battery status, an email server without a password set up, for example on localhost.
For battery status monitor, configure the following
ery_monitor:
level of battery you want to monitor
vel: CRITICAL
list of email addresses to be notified
ail_address:
- notify-me@example.com
host name of your email server
ail_host: localhost
port of email server
ail_port: 25
notifying email account
ail_user: noreply@example.com
list of topics to be monitored ( related to monitor behavior)
pics:
- android_empatica_e4_battery_level
For device connection monitor, configure the following
onnect_monitor:
timeout in milliseconds -> 5 minutes
meout: 300000
ail_address:
- notify-me@example.com
ail_host: localhost
ail_port: 25
ail_user: noreply@example.com
temperature readings are sent very regularly, but
not too often.
pics:
- android_empatica_e4_temperature
Run radarbackend.jar with configured radar.yml and monitor argument
-jar radarbackend-1.0.jar -c path/to/radar.yml monitor
Configure the REST proxy setting in radar.yml:
-proxy:
st: radar-test.thehyve.net
rt: 8082
otocol: http
To send pre-made data, create a mock_data.yml YAML file with the following contents:
:
topic: topic1
file: topic1.csv
key_schema: org.radarcns.kafka.ObservationKey
value_schema: org.radarcns.passive.empatica.EmpaticaE4Acceleration
Each value has a topic to send the data to, a file containing the data, a schema class for the key and a schema class for the value. Also create a CSV file for each of these entries:
Id,sourceId,time,timeReceived,acceleration
14191933191.223,14191933193.223,[0.001;0.3222;0.6342]
14191933194.223,14191933195.223,[0.13131;0.6241;0.2423]
Note that for array entries, use brackets ([ and ]) to enclose the values and use ; as a delimiter.
To generate data on some backend_mock_empatica_e4_<> topic with a number of devices, run (substitute <num-devices> with the needed number of devices):
-jar radarbackend-1.0.jar -c path/to/radar.yml mock --devices <num-devices>
Press Ctrl-C to stop.
To generate the file data configured in point 2, run
-jar radarbackend-1.0.jar -c path/to/radar.yml mock --file mock_data.yml
The data sending will automatically be stopped.
Code should be formatted using the Google Java Code Style Guide. If you want to contribute a feature or fix browse our issues, and please make a pull request.
There are currently two APIs in RADAR-Backend: one for streaming data (RADAR-Stream) and one for monitoring topics (RADAR-Monitor). To contribute to those APIs, please mind the following.
RADAR-Stream is a layer on top of Kafka streams. Topics are processed by streams in two phases. First, a group of sensor streams aggregates data of sensors into predefined time windows (e.g., 10 seconds). Next, internal topics aggregate and transforms data that has already been processed by an earlier stream.
KafkaStreams currently communicates using master-slave model. The MasterAggregator defines the stream-master, while AggregatorWorker represents the stream-slave. The master-stream creates, starts and stops a list of stream-slaves registered with the corresponding master. While the classical Kafka Consumer requires two implementations to support standalone and group executions, the AggregatorWorker provides both behaviors with one implementation.
To extend the RADAR-Stream API, follow these steps (see the org.radarcns.passive.empatica package as an example):
createSensorStream and createStream methods to create the stream definitions.Currently, RADAR-Backend provides implementation to stream, monitor, store Empatica E4 topics data produced by RADAR-AndroidApplication. E4Worker is the MasterAggregator. The stream group E4Streams defines the following sensor topics:
And one internal topic:
DeviceTimestampExtractor implements a TimestampExtractor such that: given in input a generic Apache Avro object, it extracts a field named timeReceived. DeviceTimestampExtractor works with the entire set of sensor schemas currently available.
For the Android Phone, there is a stream to get an app category from the Google Play Store categories for app usage events.
Monitors can be used to evaluate the status of a single stream, for example whether each device is still online, has acceptable values and is transmitting at an acceptable rate. To create a new monitor, extend AbstractKafkaMonitor. To use the monitor from the command-line, modify KafkaMonitorFactory. See DisconnectMonitor for an example.
Another path to the YAML configuration file can be given with the -c flag:
stom
-jar radarbackend-1.0.jar -c path/to/radar.yml
the default log path is the jar folder