RADAR-base/RADAR-Docker

Name: RADAR-Docker

Owner: RADAR-CNS

Owner: RADAR-CNS

Description: Integrated Docker Stack for the RADAR mHealth Streaming Platform Components

Created: 2016-11-08 14:10:16.0

Updated: 2018-01-26 07:52:05.0

Pushed: 2018-02-02 10:45:02.0

Homepage: https://hub.docker.com/u/radarcns/dashboard/

Size: 58164

Language: Shell

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

RADAR-Docker

The dockerized RADAR stack for deploying the RADAR-CNS platform. Component repositories can be found at RADAR-CNS DockerHub org

Installation instructions

To install RADAR-CNS stack, do the following:

1. Install Docker Engine

2. Install docker-compose using the installation guide or by following our wiki.

3. Verify the Docker installation by running on the command-line:

   er --version
   er-compose --version
   

   This should show Docker version 1.12 or later and docker-compose version 1.9.0 or later.

4. Install git for your platform.

   1. For Ubuntu

       apt-get install git
      

5. Clone RADAR-Docker repository from GitHub.

   clone https://github.com/RADAR-CNS/RADAR-Docker.git
   

6. Install required component stack following the instructions below.

Usage

RADAR-Docker currently offers two component stacks to run.

1. A Docker-compose for components from Confluent Kafka Platform community
2. A Docker-compose for components from RADAR-CNS platform.

Note: on macOS, remove sudo from all docker and docker-compose commands in the usage instructions below.

Confluent Kafka platform

Confluent Kafka platform offers integration of the basic components for streaming such as Zookeeper, Kafka brokers, Schema registry and REST-Proxy.

Run this stack in a single-node setup on the command-line:

ADAR-Docker/dcompose-stack/radar-cp-stack/
 docker-compose up -d

To stop this stack, run:

 docker-compose down

RADAR-CNS platform

In addition to Confluent Kafka platform components, RADAR-CNS platform offers

• RADAR-HDFS-Connector - Cold storage of selected streams in Hadoop data storage,
• RADAR-MongoDB-Connector - Hot storage of selected streams in MongoDB,
• RADAR-Dashboard,
• RADAR-Streams - real-time aggregated streams,
• RADAR-Monitor - Status monitors,
• RADAR-HotStorage via MongoDB,
• RADAR-REST API,
• a Hadoop cluster, and
• an email server.

To run RADAR-CNS stack in a single node setup:

1. Navigate to radar-hadoop-cp-stack:

   ADAR-Docker/dcompose-stack/radar-hadoop-cp-stack/
   

2. Configure monitor settings in radar.yml:

   ery_monitor:
   vel: CRITICAL
   ail_address:
    notify-1@example.com
    notify-2@example.com
   ail_host: smtp
   ail_port: 25
   ail_user: user@example.com
   pics:
   - android_empatica_e4_battery_level
   
   onnect_monitor:
   timeout in milliseconds -> 5 minutes
   meout: 300000
   ail_address:
    notify-1@example.com
    notify-2@example.com      
   ail_host: smtp
   ail_port: 25
   ail_user: user@example.com
   temperature readings are sent very regularly, but
   not too often.
   pics:
   - android_empatica_e4_temperature
   

3. Create smtp.env and configure your email settings following smtp.env.template. Configure alternative mail providers like Amazon SES or Gmail by using the parameters of the namshi/smtp Docker image.

4. (Optional) Modify flush.size and HDFS direcotory for Cold storage in sink-hdfs.properties

   h.size=
   cs.dir=/path/to/data
   

   Note: To have different flush.size for different topics, you can create multipe property configurations for a single connector. To do that,

   4.1 Create multipe property files that have different flush.size for given topics. Examples sink-hdfs-high.properties , sink-hdfs-low.properties

   4.2 Add CONNECTOR_PROPERTY_FILE_PREFIX: <prefix-value> enviornment variable to radar-hdfs-connector service in docker-compose file.
   

   4.3 Add created property files to the radar-hdfs-connector service in docker-compose with name abides to prefix-value mentioned in CONNECTOR_PROPERTY_FILE_PREFIX

   radar-hdfs-connector:
     image: radarcns/radar-hdfs-connector-auto:0.2
     restart: on-failure
     volumes:
   - ./sink-hdfs-high.properties:/etc/kafka-connect/sink-hdfs-high.properties
   - ./sink-hdfs-low.properties:/etc/kafka-connect/sink-hdfs-low.properties
     environment:
   CONNECT_BOOTSTRAP_SERVERS: PLAINTEXT://kafka-1:9092,PLAINTEXT://kafka-2:9092,PLAINTEXT://kafka-3:9092         
   CONNECTOR_PROPERTY_FILE_PREFIX: "sink-hdfs"
   

5. Configure Hot Storage settings in .env file

   TORAGE_USERNAME=mongodb-user
   TORAGE_PASSWORD=XXXXXXXX
   TORAGE_NAME=mongodb-database
   

6. To install the stack

    ./install-radar-stack.sh
   

To stop RADAR-CNS stack on a single node setup, run

ADAR-Docker/dcompose-stack/radar-hadoop-cp-stack/
 ./stop-radar-stack.sh 

To reboot RADAR-CNS stack on a single node setup, run

ADAR-Docker/dcompose-stack/radar-hadoop-cp-stack/
 ./reboot-radar-stack.sh

To start RADAR-CNS stack on a single node setup after installing, run

ADAR-Docker/dcompose-stack/radar-hadoop-cp-stack/
 ./start-radar-stack.sh

cAdvisor

cAdvisor (Container Advisor) provides container users an understanding of the resource usage and performance characteristics of their running containers.

To view current resource performance,if running locally, try http://localhost:8181. This will bring up the built-in Web UI. Clicking on /docker in Subcontainers takes you to a new window with all of the Docker containers listed individually.

Portainer

Portainer provides simple interactive UI-based docker management. If running locally, try http://localhost:8182 for portainer's UI. To set-up portainer follow this link.

Logging

Set up logging by going to the dcompose-stack/logging directory and follow the README there.

Work in progress

The two following stacks will not work on with only Docker and docker-compose. For the Kerberos stack, the Kerberos image is not public. For the multi-host setup, also docker-swarm and Docker beta versions are needed.

Kerberized stack

In this setup, Kerberos is used to secure the connections between the Kafka brokers, Zookeeper and the Kafka REST API. Unfortunately, the Kerberos container from Confluent is not publicly available, so an alternative has to be found here.

 wip/radar-cp-sasl-stack/
cker-compose up

Multi-host setup

In the end, we aim to deploy the platform in a multi-host environment. We are currently aiming for a deployment with Docker Swarm. This setup uses features that are not yet released in the stable Docker Engine. Once they are, this stack may become the main Docker stack. See the wip/radar-swarm-cp-stack/ directory for more information.