SoftwareDefinedBuildings/quasar-spark-connector

Name: quasar-spark-connector

Owner: Software Defined Buildings

Description: Quasar - Spark Connector

Created: 2015-05-23 20:49:02.0

Updated: 2015-05-25 00:52:03.0

Pushed: 2015-12-04 02:32:48.0

Homepage: null

Size: 151

Language: Scala

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

BTrDB - Spark Connector

BTrDB - Spark Connector

1. Pre-requisites

• Scala 2.10.4
  
• Spark 1.1.1
  
• Ceph 0.8
• BTrDB 3.0
  

2. Setting-up Rados-Java

† You should setup rados-java support across the entire cluster prior to running the connector.

Getting librados & libjna

Ubuntu

sudo apt-get install librados-dev libjna-java  

Centos

sudo yum install librados2-devel jna   

Build rados-java & install

git clone --recursive https://github.com/ceph/rados-java.git  
cd rados-java
ant  

sudo cp target/rados-0.1.3.jar /usr/share/java/rados-1.0-SNAPSHOT.jar
sudo ln -s /usr/share/java/jna-3.2.7.jar /usr/lib/jvm/default-java/jre/lib/ext/jna-3.2.7.jar
sudo ln -s /usr/share/java/rados-1.0-SNAPSHOT.jar  /usr/lib/jvm/default-java/jre/lib/ext/rados-1.0-SNAPSHOT.jar

# documentation
ant docs

• try run rados-java example and make sure it runs without extra java directives. Some platforms give you difficulties to link against librados or libjna.

2. Building

sbt assembly  

3. Examples

§ Current implementation only works when all Spark executors are Ceph monitor at the same time.

Statistical Aggregator

Start spark shell with the adapter

$ spark-shell --jars <path to the adapter jar>/quasar-spark-connector-assembly-1.0.jar  

sc.quasarStatQuery(“uuid”, start-time, end-time, “unit-time”, point-width)

scala> import edu.berkeley.eecs.btrdb.sparkconn._
scala> val qrdd = sc.quasarStatQuery("2e43475f-5359-5354-454d-5f5245414354", 1364823796L, 1398437046L, "ns", 16)
scala> qrdd.map(i => i.toString).collect.foreach(println)

TODO

1. Support creating RDD for unprocessed, raw time-series
   
2. Fix an issue of inability to retrieve slave nodes info from SparkContext
   
3. Support unified timeframe across multiple sensors
   
4. Support Spark Stream
   
5. Error handling