thehyve/RADAR-Backend

Name: RADAR-Backend

Owner: The Hyve

Description: Kafka backend for processing device data

Forked from: RADAR-CNS/RADAR-Backend

Created: 2016-10-11 09:05:31.0

Updated: 2016-10-11 09:05:33.0

Pushed: 2017-09-26 12:17:07.0

Homepage: null

Size: 3907

Language: Java

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

RADAR-CNS back-end

A Java application based on Confluent Platform to standardise, analyse and persistent data collected by RADAR-CNS data sources. Data is consumed and produced in Apache Avro format using the schemas stored inside the RADAR-CNS schema repository.

Currently only the Empatica E4 is supported.

Generalisation

The Kafka Streams concept has been generalised. MasterAggregator defines the stream master while AggregatorWorker represents the stream slave. The master stream creates, starts and stops a list of stream slave. Two types of slave stream have been defined:

• SensorAggregator consumes and aggregates topics without performing any transformations
• InternalAggregator modifies incoming data producing a new information While the classical Kafka Consumer requires two implementations to support standalone and group executions, the stream slave definition provides both behaviours with one implementation.

While the former consumes SensorTopic, the latter reads InternalTopic. These topics are a specialisation of AvroTopic.

• SensorTopic defines a set of topic used to consume and aggregate data as is
• InternalTopic delineates a set of topic used to consume and aggregate transformed data SensorTopics and InternalTopics drive the developer to the implementation of sets of topics containing only one topic type. To improve the flexibility, these sets are then unified by DeviceTopics. All these set should be defined following the Factory Method Pattern. It should be used for each element that has to be unique within the application (e.g. MasterAggregator)

DeviceTimestampExtractor implements a TimestampExtractor such that: given in input a generic APACHE Avro object, it extracts a field named timeReceived. DeviceTimestampExtractor works with the entire set of sensor schemas currently available.

Empatica E4

E4Worker is the MasterAggregator. E4SensorTopics and E4InternalTopics are respectively SensorTopics and InternalTopics. E4Topics (i.e. DeviceTopics) are consumed by:

• E4Acceleration: it aggregates data coming from accelerometer
• E4BatteryLevel: it aggregates battery level information
• E4BloodVolumePulse: it aggregates blood volume pulse data
• E4ElectroDermalActivity: it aggregates electrodermal activity informations
• E4HeartRate: starting from the inter beat interval, this aggregator computes the heart rate value (i.e. InternalAggregator)
• E4InterBeatInterval: it aggregates inter beat interval data
• E4Temperature: it aggregates data coming form temperature sensor

Contributing

To add additional devices, make the following steps (see the org.radarcns.empaticaE4 package as an example):

• create SensorTopics and if needed InternalTopics then unify them in DeviceTopics
• for each topic create either SensorAggregator or InternalAggregator
• define your MasterAggregator

Code should be formatted using the Google Java Code Style Guide.

If you want to contribute a feature or fix, please make a pull request

Quickstart for Empatica

Confluent Platform

art Zookeeper
n/zookeeper-server-start config/zookeeper.properties

art Kafka
n/kafka-server-start config/server.properties

art Schema Registry
n/schema-registry-start config/schema-registry.properties

art Schema Registry
n/schema-registry-start config/schema-registry.properties

art Rest Proxy
bin/kafka-rest-start ./etc/kafka-rest/kafka-rest.properties

eate all required topics
n/kafka-topics.sh --create --zookeeper <ZOOKEEPER-HOST>:<ZOOKEEPER-PORT> --replication-factor 1 --partitions 1 --topic <TOPIC-NAME>

Radar application

This is a Gradle project, to execute the following step

1. open your shell inside the project folder

   ean
   adle clean
   

Build

$ gradle jar

opy your jar in your server
odify `radar.yml` file
 specify in which mode you want to run the application. The two alternatives are `standalone` and `high_performance`. The former starts one thread for each streams without checking the priority, the latter starts as many thread as the related priority value
 insert Zookeeper server information
 insert Broker server information
 insert Schema-Registry location
 insert `log_path`, it must be a folder [OPTIONAL]
opy `radar.yml` in the same folder where you have copied the jar
un your jar

$ java -jar radarbackend-1.0.jar

NOTE
 you have installed the Confluent Platform via `apt`, the base path will be `/usr/bin/` instead of `bin/`
 you have installed the Confluent Platform via `sudo` all commands have to be run by a sudoers user 
mmands to start Zookeeper, Kafka, Schema Registry and Rest Proxy have to be run one per shell
 you have not modified the out-of-the-box configuration files of Confluent `<ZOOKEEPER-HOST>:<ZOOKEEPER-PORT>` values `localhost:2181`
e `radarbackend-<VERSION>.jar` file is located in `/project-root/build/lib`
CONFIG-FILE>` is optional, if not provided your config file has to be located in the same folder of your jar

Default

$ java -jar radarbackend-1.0.jar

Custom

$ java -jar radarbackend-1.0.jar /Users/francesco/Desktop/radar-test/config/radar.yml

e default log path is the jar folder
reads priorities value must be bigger than 0
 `AUTO.CREATE.TOPICS.ENABLE` is `FALSE` in your server.properties, before starting you must create the following topics manually: 
android_empatica_e4_acceleration
android_empatica_e4_acceleration_output
android_empatica_e4_battery_level
android_empatica_e4_battery_level_output
android_empatica_e4_blood_volume_pulse
android_empatica_e4_blood_volume_pulse_output
android_empatica_e4_electrodermal_activity
android_empatica_e4_electrodermal_activity_output
android_empatica_e4_heartrate_output
android_empatica_e4_inter_beat_interval
android_empatica_e4_inter_beat_interval_output
android_empatica_e4_sensor_status
android_empatica_e4_sensor_status_output
android_empatica_e4_temperature
android_empatica_e4_temperature_output