twitter/scoot

Name: scoot

Owner: Twitter, Inc.

Description: null

Created: 2015-12-21 22:14:54.0

Updated: 2017-12-15 22:46:20.0

Pushed: 2018-01-17 00:17:25.0

Homepage:

Size: 9318

Language: Go

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

Scoot

Scoot is infrastructure to make developer tools smaller, simpler, and more distributed. The core concept of Scoot is the Snapshot, which represents an immutable filesystem state. Scoot allows for execution of commands against input Snapshots to create new output Snapshots.

Scoot Components

Scheduler

The Scheduler receives and distrubutes jobs to workers, and maintains state. It is also responsible for serving the Cloud Scoot API for clients.

Worker

The Worker (workerserver) receives information about jobs and runs them, and is responsible for all Snapshot-related functionality.

Daemon

The local daemon serves a Protobuf interface and can either send Scoot jobs to a remote scheduler or to embedded local workers.

Client APIs

• Cloud Scoot API
• Worker API
• Daemon API

Jobs and Tasks

Scoot work is broken down into independent jobs, which can consist of one or more tasks, which are executed sequentially.

Example

Setup a scheduler and worker nodes locally:

un ./binaries/setup-cloud-scoot/main.go --strategy local.local

Run a series of randomly generated tests against the local scheduler and workers:

un ./binaries/scootapi/main.go run_smoke_test

Scoot Integration Tests

Scoot has a few tests that exercise varying levels of common usages and workflows.

Smoketest/Swarmtest

Invokes a scootapi client directly to run jobs against a local cluster and waits for the scheduled jobs to complete.

(./scootapi/client/smoke_test_cmd.go)

Recoverytest

Invokes a scootapi client directly to run jobs against a local cluster, kills the cluster, attempts to spin up a new one, and waits for the originally scheduled jobs to complete.

(./binaries/recoverytest/main.go)

Integration

Invokes a scootapi and scoot-snapshot-db client via CLI to run a job against a local cluster and waits for the job to complete

(./tests/integration_test.go)

Scoot Thrift Code

Thrift Prerequisites Install the Thrift tool and golang thrift repository locally using the following section.

Generating thrift files (scootapi used as an example) See documentation in thrift definition files for specific generation instructions, or make thrift.

Scoot Protobuf Code

Generating go protobuf files (for local Scoot Daemon) cd to scoot's daemon/protocol

protoc -I . daemon.proto –go_out=plugins=grpc:.

Generating python client files (for client library accessing local Scoot Daemon) cd to scoot's daemon/protocol

on -m grpc.tools.protoc -I. --python_out=./python/scoot --grpc_python_out=./python/scoot daemon.proto

Installation Instructions

Install 3rd party tools:

Thrift (version >= 0.9.3)

macOS:

 install thrift

Thrift for go:

et github.com/apache/thrift/lib/go/thrift

Protobuf for the Scoot Daemon

• If necessary, remove any conflicting or older versions of Protobuf:

   uninstall protobuf
  

grpcio for python

install grpcio grpcio-tools

(If grpcio* is already installed you should probably update it by including '–upgrade' to the above pip command)

docopt for python

install docopt==0.6.2

Install/Access Scoot Executables and libraries

Scoot Local Daemon, Local Scheduler and Local Worker

cd to scoot directory (workspace/github.com/twitter/scoot) run: go install ./binaries/… the binaries will be installed in workspace/bin

Scoot Local Daemon Command Line Client

Python client library
be found at workspace/github.com/twitter/scoot/daemon/protocol/python/scoot/client_lib.py