Name: scio
Owner: Spotify
Description: A Scala API for Apache Beam and Google Cloud Dataflow.
Created: 2015-03-26 19:07:34.0
Updated: 2018-05-24 13:28:58.0
Pushed: 2018-05-24 09:32:14.0
Homepage: https://spotify.github.io/scio
Size: 10371
Language: Scala
GitHub Committers
User | Most Recent Commit | # Commits |
---|
Other Committers
User | Most Recent Commit | # Commits |
---|
Ecclesiastical Latin IPA: /??i.o/, [??i?.o], [??i.i?o]
Verb: I can, know, understand, have knowledge.
Scio is a Scala API for Apache Beam and Google Cloud Dataflow inspired by Apache Spark and Scalding.
Scio 0.3.0 and future versions depend on Apache Beam (org.apache.beam
) while earlier versions depend on Google Cloud Dataflow SDK (com.google.cloud.dataflow
). See this page for a list of breaking changes.
* provided by Google Cloud Dataflow
Use our giter8 template to quickly setup a project:
sbt new spotify/scio.g8
Compile it:
sbt pack
Run the included word count example:
target/pack/bin/word-count --output=wc
Inspect the results:
cat wc/part-00000-of-00001.txt
Getting Started is the best place to start with Scio. If you are new to Apache Beam and distributed data processing, check out the Beam Programming Guide first for a detailed explanation of the Beam programming model and concepts. If you have experience with other Scala data processing libraries, check out this comparison between Scio, Scalding and Spark. Finally check out this document about the relationship between Scio, Beam and Dataflow.
Example Scio pipelines and tests can be found under scio-examples. A lot of them are direct ports from Beam's Java examples. See this page for some of them with side-by-side explanation. Also see Big Data Rosetta Code for common data processing code snippets in Scio, Scalding and Spark.
Scio includes the following artifacts:
scio-core
: core libraryscio-test
: test utilities, add to your project as a “test” dependencyscio-avro
: add-on for Avro, included in scio-core
but can also be used standalonescio-bigquery
: add-on for BigQuery, included in scio-core
but can also be used standalonescio-bigtable
: add-on for Bigtablescio-cassandra2
: add-on for Cassandra 2.xscio-cassandra3
: add-on for Cassandra 3.xscio-elasticsearch2
: add-on for Elasticsearch 2.xscio-elasticsearch5
: add-on for Elasticsearch 5.xscio-extra
: extra utilities for working with collections, Breeze, etc.scio-hdfs
: add-on for HDFS IOscio-jdbc
: add-on for JDBC IOscio-parquet
: add-on for Parquetscio-tensorflow
: add-on for TensorFlow TFRecords IO and predictionCopyright 2016 Spotify AB.
Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0