googlegenomics/pipelines-tools

Name: pipelines-tools

Owner: Google Genomics

Description: Tools for developing and running pipelines with the Genomics API

Created: 2018-03-02 19:20:13.0

Updated: 2018-03-12 14:43:26.0

Pushed: 2018-03-09 15:08:01.0

Homepage: null

Size: 29

Language: Go

GitHub Committers

User	Most Recent Commit	# Commits
Garret Kelly	2018-03-02 20:14:25.0	2
Aaron Kemp	2018-03-09 01:10:56.0	15

Other Committers

User	Email	Most Recent Commit	# Commits

README

Google Genomics Pipelines Tools

This repository contains various tools that are useful when running pipelines with the Google Genomics API.

Quick Start Using Cloud Shell

1. Enable the Genomics API and the Compute Engine API in a new or existing Google Cloud project.

2. Start a Cloud Shell inside your project.

3. Inside the Cloud Shell, run the command

   go get github.com/googlegenomics/pipelines-tools/...
   

   This command downloads and installs the pipelines tools. Note that to build these tools outside the Cloud Shell you will need the Go tool chain.

4. Make a bucket on GCS to store the output from the pipeline:

   export BUCKET=gs://${GOOGLE_CLOUD_PROJECT}-pipelines
   gsutil mb ${BUCKET}
   

5. Put some test data into the bucket:

   echo "Hello World" | gsutil cp - ${BUCKET}/input
   

6. Make a pipeline script that computes the SHA1 sum of a file:

   echo 'sha1sum ${INPUT0} > ${OUTPUT0}' > sha1.script
   

7. Run the script using the pipelines API:

   pipelines run --inputs=${BUCKET}/input --outputs=${BUCKET}/output sha1.script
   

8. Check the generated output file:

   gsutil cat ${BUCKET}/output
   

That's it: you've run your first pipeline. For more information about the input formats supported by the pipelines tool, check out the source code. To learn more about the Pipelines API, consult the reference documentation.

Usage

The pipelines tool

This tool provides support for running, cancelling and inspecting pipelines.

As a simple example, to run a pipeline that prints 'hello world':

t <<EOF > hello.script
 "hello world"

pelines --project=my-project run hello.script --output=gs://my-bucket/logs --wait

After the pipeline finishes, you can inspect the output using gsutil:

util cat gs://my-bucket/logs/output

The script file format is described in the source code for the command.

The migrate-pipeline tool

This tool takes a JSON encoded v1alpha2 run pipeline request and attempts to emit a v2alpha1 request that replicates the same behaviour.

For example, given a file v1.jsonpb that has a request containing a v1alpha2 ephemeral pipeline and arguments, running:

grate-pipeline < v1.jsonpb

will produce a v2alpha1 request that performs the same action on standard output.

Support

Please report problems using the issue tracker.