OpenGeoscience/geoingest

Name: geoingest

Owner: OpenGeoscience

Description: null

Created: 2017-02-21 19:16:41.0

Updated: 2017-06-19 19:50:58.0

Pushed: 2017-03-28 15:23:15.0

Homepage:

Size: 807

Language: Shell

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

Instructions

Clone this repository.

clone git@github.com:OpenGeoscience/geoingest.git
eoingest

Fill the credential information

Copy the config file.

ws-credentials.mk.example aws-credentials.mk

Edit the information in the aws-credentials.mk. That file will be ignored when you push.

(Optional) Build ingest assembly

Since we have an assembly on s3 we can skip the assembly part.

Get Geotrellis as a submodule

submodule init
submodule update

ash
eotrellis
t "project spark-etl" assembly

Once this succeeds your assembly will be

ngest/geotrellis/spark-etl/target/scala-2.11/geotrellis-spark-etl-assembly-1.1.0-SNAPSHOT.jar

If you don't have this assembly on s3 you can push it by doing

s3 cp geoingest/geotrellis/spark-etl/target/scala-2.11/geotrellis-spark-etl-assembly-1.1.0-SNAPSHOT.jar s3://my-bucket/

Create Json Specs For Geotrellis

Geotrellis ingest requires 3 json files for specifying the ingest job. There is a very very small utility that is written for that purpose.

rtualenv geoingest
install -r requirements.txt
install -e .

ash
st s3://locationOfS3BucketWithTiffs layerName s3://locationOfCatalog

This will create json specifications in the current directory for the ingest job.

Now we need to push those json files to s3 so that geotrellis can read it.

 copy-json-specs

Create Cluster

At this point it is very easy to launch our cluster.

 create-cluster

This will take roughly 10 minutes. After that we can ingest our layers.

Submit the ingest job

To submit the ingest job just do:

 submit-remote-ingest