unicef/devtrac

Name: devtrac

Owner: unicef

Description: null

Created: 2018-05-25 03:22:25.0

Updated: 2018-05-25 06:24:55.0

Pushed: 2018-05-25 06:25:15.0

Homepage: null

Size: 10

Language: Shell

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

DevTrac

Real-time integrated analytics stack.

WARNING!!: this repository is under heavy development, it should be considered in pre-alpha stage. Deep changes can happen frequently and no commits guarantee backward compatibility. Use at your risk :)

Quick start

Initialize the stack

make setup  # only first time

Start

make start

Services endpoints:

• Nifi http://127.0.0.1:8080/nifi
• Nifi Registry http://127.0.0.1:58080/
• Rancher http://127.0.0.1:8000
• Hbase REST Web http://127.0.0.1:28085/rest.jsp
• SuperSet http://127.0.0.1:48088

Main Components

• postgres
• nifi
• hdfs
• hbase
• spark
• superset
• kubernetes
• rancher

Extra packages and dependencies

Below some extra services provided by the stack, in some case they are required by some service and/or provided by the docker image

• Thrift
• Zookeeper

postgres

Postgres (Postgis) is here used to simulate eTools database.

To properly load the database, copy a database backup file as stack/db/backup.bz2 and run make init-db before you run the whole stack.

HDFS

Nifi

To move data from sources (postgres in this POC) to HBase

HBase

Big data store.

Spark

Superset

Rancher

Full port list

• 8000 Rancher

• 8080 NiFi

• 8443 NiFi

• 10000 NiFi

• 15432 Postgres

• 28080 Hbase REST API

• 28085 Hbase REST Web

• 29090 Thrift API

• 29095 Thrift Web UI

• 22181 HBase's Embedded zookeeper cluster

• 26010 HBase Master UI, HBase ZK

• 38080 Zeppelin

• 48088 SuperSet

• 58080 NiFi Registry