spotify/limbo

Name: limbo

Owner: Spotify

Description: null

Created: 2016-12-12 21:37:35.0

Updated: 2017-08-23 17:39:08.0

Pushed: 2017-01-02 16:34:35.0

Homepage: null

Size: 139

Language: Scala

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

Limbo

Raison d'être:

Note: Limbo is in a WIP state.

Limbo is a Scala API which allows to leverage best of different data processing frameworks by allowing seamless transition between framework specific data structures.

Features

• Itegration between Scio and Spark
• Programmatic Spark job submission to a Apache YARN cluster
• Scala API for Google Dataproc cluster

Example:

tart in Scio:
(sc, args) = ContextAndArgs(argv)
scol = sc.parallelize(1 to 10)

ove to Spark realm
.toRDD().map { rdd =>
d
.map(_ * 2)
.saveAsTextFile(args("output"))

Code of conduct

This project adheres to the Open Code of Conduct. By participating, you are expected to honor this code.

License

Copyright 2016 Spotify AB.

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0