Name: go-kafka-client-mesos
Owner: Elodina
Description: Go Kafka Client Mesos Framework
Created: 2016-01-25 16:43:37.0
Updated: 2016-02-17 08:33:40.0
Pushed: 2016-04-06 16:43:15.0
Homepage: http://www.elodina.net
Size: 3648
Language: Go
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
This framework aims to simplify running all sorts of software built on top of go_kafka_client on Mesos. Being actively developed right now.
Right now this framework supports running only the mirror maker.
Install go 1.5 (or higher) http://golang.org/doc/install
Install glide:
get -u github.com/Masterminds/glide
Clone and build the project
# git clone https://github.com/elodina/go-kafka-client-mesos.git
# cd go-kafka-client-mesos
# export GO15VENDOREXPERIMENT=1
# glide install
# go build cli.go
# go build executor.go
cli help scheduler
e: scheduler [options]
ons:
--master: Mesos Master addresses.
--api: API host:port for advertizing.
--user: Mesos user. Defaults to current system user.
--storage: Storage for cluster state. Examples: file:go_kafka_client_mesos.json; zk:master:2181/go-mesos.
--log.level: Log level. trace|debug|info|warn|error|critical. Defaults to info.
--framework.name: Framework name.
--framework.role: Framework role.
--framework.timeout: Framework failover timeout.
In order not to pass the API url to each CLI call lets export the URL as follows:
# export GM_API=http://master:6666
First lets start 1 mirror maker task with the default settings. Further in the readme you can see how to change these from the defaults.
cli add mirrormaker 0 --executor=executor
d tasks 0
ter:
sk:
type: mirrormaker
id: 0
state: inactive
cpu: 0.50
mem: 512.00
configs:
num.producers: 1
num.streams: 1
queue.size: 10000
You now have a cluster with 1 task that is not started.
cli status
ter:
sk:
type: mirrormaker
id: 0
state: inactive
cpu: 0.50
mem: 512.00
configs:
queue.size: 10000
num.producers: 1
num.streams: 1
Each mirror maker task requires some basic configuration.
cli update 0 --producer.config producer.config --consumer.config consumer.config --whitelist "^mytopic$"
iguration updated:
ter:
sk:
type: mirrormaker
id: 0
state: inactive
cpu: 0.50
mem: 512.00
configs:
num.producers: 1
num.streams: 1
queue.size: 10000
consumer.config: consumer.config
producer.config: producer.config
whitelist: ^mytopic$
Now lets start the task. This call to CLI will block until the task is actually started but will wait no more than a configured timeout. Timeout can be passed via –timeout flag and defaults to 30s. If a timeout of 0ms is passed CLI won't wait for tasks to start at all and will reply with “Scheduled tasks …” message.
cli start 0
ted tasks 0
ter:
sk:
type: mirrormaker
id: 0
state: running
cpu: 0.50
mem: 512.00
task id: mirrormaker-0-e5ab7581-1c31-d85d-ff14-77b5fb3de4d4
slave id: 20150914-091943-84125888-5050-10717-S1
executor id: mirrormaker-0
attributes:
hostname: slave0
configs:
consumer.config: consumer.config
producer.config: producer.config
whitelist: ^mytopic$
num.producers: 1
num.streams: 1
queue.size: 10000
By now you should have a single mirror maker instance running. Here's how you stop it:
cli stop 0
ped tasks 0
And remove:
cli remove 0
ved tasks 0
While the scheduler has a shutdown hook it doesn't actually finish the framework. To shutdown the framework completely (e.g. unregister it in Mesos) you may shoot a POST to /teardown specifying the framework id to shutdown:
rl -d frameworkId=20150807-094500-84125888-5050-14187-0005 -X POST http://master:5050/teardown
cli help
e:
lp: show this message
heduler: start scheduler
d: add task
date: update configuration
art: start task
atus: get current cluster status
op: stop task
move: remove task
detailed help from ./cli help <command>
cli help add
e: add <type> <id-expr> [options]
s:
mirrormaker
ons:
--executor: Executor binary name.
--api: API host:port for advertizing. Optional if GM_API env is set.
--cpu: CPUs per task. Defaults to 0.5.
--mem: Mem per task. Defaults to 512.
--constraints: Constraints (hostname=like:^master$,rack=like:^1.*$).
xpr examples:
0 - task 0
0,1 - tasks 0,1
0..2 - tasks 0,1,2
0,1..2 - tasks 0,1,2
* - all tasks in cluster
traint examples:
like:slave0 - value equals 'slave0'
unlike:slave0 - value is not equal to 'slave0'
like:slave.* - value starts with 'slave'
unique - all values are unique
cluster - all values are the same
cluster:slave0 - value equals 'slave0'
groupBy - all values are the same
groupBy:3 - all values are within 3 different groups
All file-related configs should be set as file names located . (e.g. setting –producer.config producer.properties) OR
http/https URLs (e.g. http://192.168.3.1:6666/producer.config). Please note that http URLs MUST end with file name to be downloaded as Mesos does not respect Content-Disposition header.
cli help update
e: update <id-expr> [options]
ons:
--api: API host:port for advertizing. Optional if GM_API env is set.
--cpu: CPUs per task.
--mem: Mem per task.
--constraints: Constraints (hostname=like:^master$,rack=like:^1.*$).
--whitelist: Regex pattern for whitelist. Providing both whitelist and blacklist is an error.
--blacklist: Regex pattern for blacklist. Providing both whitelist and blacklist is an error.
--producer.config: Producer config url or file name.
--consumer.config: Consumer config url or file name.
--num.producers: Number of producers.
--num.streams: Number of consumption streams.
--preserve.partitions: Preserve partition number. E.g. if message was read from partition 5 it'll be written to partition 5.
--preserve.order: E.g. message sequence 1, 2, 3, 4, 5 will remain 1, 2, 3, 4, 5 in destination topic.
--prefix: Destination topic prefix.")
--queue.size: Maximum number of messages that are buffered between the consumer and producer.
--options: Additional options for executor, separated by ';'.
xpr examples:
0 - task 0
0,1 - tasks 0,1
0..2 - tasks 0,1,2
0,1..2 - tasks 0,1,2
* - all tasks in cluster
traint examples:
like:slave0 - value equals 'slave0'
unlike:slave0 - value is not equal to 'slave0'
like:slave.* - value starts with 'slave'
unique - all values are unique
cluster - all values are the same
cluster:slave0 - value equals 'slave0'
groupBy - all values are the same
groupBy:3 - all values are within 3 different groups
cli help start
e: start <id-expr> [options]
ons:
--api: API host:port for advertizing. Optional if GM_API env is set.
--timeout: Timeout in seconds to wait until the task receives Running status.
xpr examples:
0 - task 0
0,1 - tasks 0,1
0..2 - tasks 0,1,2
0,1..2 - tasks 0,1,2
* - all tasks in cluster
cli help stop
e: stop <id-expr> [options]
ons:
--api: API host:port for advertizing. Optional if GM_API env is set.
xpr examples:
0 - task 0
0,1 - tasks 0,1
0..2 - tasks 0,1,2
0,1..2 - tasks 0,1,2
* - all tasks in cluster
cli help remove
e: remove <id-expr> [options]
ons:
--api: API host:port for advertizing. Optional if GM_API env is set.
xpr examples:
0 - task 0
0,1 - tasks 0,1
0..2 - tasks 0,1,2
0,1..2 - tasks 0,1,2
* - all tasks in cluster
cli help status
e: status [options]
ons:
--api: API host:port for advertizing. Optional if GM_API env is set.