appbaseio/transporter

Name: transporter

Owner: appbase.io

Description: Sync data between persistence engines, like ETL only not stodgy

Created: 2016-05-15 08:42:38.0

Updated: 2016-06-03 16:49:05.0

Pushed: 2016-06-10 10:09:40.0

Homepage: null

Size: 3715

Language: Go

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

Compose helps with database transformations from one store to another. It can also sync from one to another or several stores.

Transporter

Build

go get github.com/compose/transporter
cd $GOPATH/src/github.com/compose/transporter
git remote add fork https://github.com/appbaseio/transporter
git pull fork master

make sure godep is installed, go get github.com/tools/godep and then build with
godep restore
godep go build -a ./cmd/...

Install

Create a build
godep go install ./cmd/...

Configure

There is a sample config in test/config.yaml. The config defines the endpoints, (either sources or sinks) that are available to the application.

terval: 60s # time interval between metrics posts to the api endpoint
i: "http://requestb.in/1a0zlf11"
s:
calmongo:
type: mongo
uri: mongodb://localhost/boom
pernick:
type: elasticsearch
uri: http://10.0.0.1,10.0.0.2:9200/indexname
bug:
type: file
uri: stdout://
ofile:
type: file
uri: file:///tmp/foo
dout:
type: file
uri: stdout://

There is also a sample 'application.js' in test/application.js. The application is responsible for building transporter pipelines. Given the above config, this Transporter application.js will copy from a file (in /tmp/foo) to stdout.

ce({name:"foofile"}).save({name:"stdout"})

This application.js will copy from the local mongo to a file on the local disk

ce({name:"localmongo", namespace: "boom.foo"}).save({name:"tofile"})

Transformers can also be configured in the application.js as follows

pipeline = Source({name:"mongodb-production", namespace: "compose.milestones2"})
line = pipeline.transform("transformers/transform1.js").transform("transformers/transform2.js")
line.save({name:"supernick", namespace: "something/posts2"});

Run

• list transporter list --config ./test/config.yaml
• run transporter run --config ./test/config.yaml ./test/application.js
• eval transporter eval --config ./test/config.yaml 'Source({name:"localmongo", namespace: "boom.foo"}).save({name:"tofile"})'
• test transporter test --config ./test/config.yaml test/application.js

Complete beginners guide

OS X

• ensure you have mercurial installed as it is required for a dependency
  • using the homebrew package manager brew install hg Homebrew Guide/Install
• install the Mac OS X binary build from https://golang.org/dl/
• follow instructions on http://golang.org/doc/install
• VERY IMPORANT: Go has a required directory structure which the GOPATH needs to point to. Instructions can be found on http://golang.org/doc/code.html or by typing go help gopath in terminal.
• setup the directory structure in $GOPATH
  • cd $GOPATH; mkdir src pkg bin
  • create the github.com path and compose mkdir -p src/github.com/compose; cd src/github.com/compose
  • clone transporter git clone https://github.com/compose/transporter; cd transporter
  • run go get to get all the dependencies go get -a ./cmd/...
  • now you can build go build -a ./cmd/...

At this point you should be able to run transporter via $GOPATH/bin/transporter, you may need to add $GOPATH to your PATH environment variable. Something along the lines of export PATH="$GOPATH/bin:$PATH" should work.

Vagrant

• ensure vagrant is installed
• ensure ansible is installed
• ensure either virtual box or VMWare fusion or VMWare Workstation is installed

 transporter
grant up

grant ssh

ant> ./run-test

Windows

See READMEWINDOWS.md

Transporter in the Media

• Compose's articles

Contributing to Transporter

Want to help out with Transporter? Great! There are instructions to get you started here.

IRC

Want to talk about Transporter? Join us in #compose on irc.freenode.net!

Licensing

Transporter is licensed under the New BSD License. See LICENSE for full license text.