Name: puppet-kafka
Owner: Mirantis Inc.
Description: Puppet module for Apache Kafka 0.8.
Forked from: wikimedia/puppet-kafka
Created: 2015-10-20 21:18:14.0
Updated: 2015-10-20 21:18:15.0
Pushed: 2015-10-23 17:37:23.0
Homepage: https://gerrit.wikimedia.org/r/#/admin/projects/operations/puppet/kafka
Size: 200
Language: Puppet
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
Table of Contents generated with DocToc
A Puppet module for installing and managing Apache Kafka brokers.
This module is currently being maintained by The Wikimedia Foundation in Gerrit at operations/puppet/kafka and mirrored here on GitHub. It was originally developed for 0.7.2 at https://github.com/wikimedia/puppet-kafka-0.7.2.
stall the kafka package.
s { 'kafka': }
This will install the Kafka package which includes /usr/sbin/kafka, useful for running client (console-consumer, console-producer, etc.) commands.
clude Kafka Broker Server.
s { 'kafka::server':
log_dirs => ['/var/spool/kafka/a', '/var/spool/kafka/b'],
brokers => {
'kafka-node01.example.com' => { 'id' => 1, 'port' => 12345 },
'kafka-node02.example.com' => { 'id' => 2 },
},
zookeeper_hosts => ['zk-node01:2181', 'zk-node02:2181', 'zk-node03:2181'],
zookeeper_chroot => '/kafka/cluster_name',
og_dirs``` defaults to a single ```['/var/spool/kafka]```, but you may
ify multiple Kafka log data directories here. This is useful for spreading
topic partitions across multiple disks.
```brokers``` parameter is a Hash keyed by ```$::fqdn```. Each value is another Hash
contains config settings for that kafka host. ```id``` is required and must
nique for each Kafka Broker Server host. ```port``` is optional, and defaults
092.
Kafka Broker Server's ```broker_id``` and ```port``` properties in server.properties
be set based by looking up the node's ```$::fqdn``` in the hosts
passed into the ```kafka``` base class.
h Kafka will store its metadata in Zookeeper. This is useful if you
to use a single Zookeeper cluster to manage multiple Kafka clusters.
below for information on how to create this Znode in Zookeeper.
ustom Zookeeper Chroot
afka will share a Zookeeper cluster with other users, you might want to
te a Znode in zookeeper in which to store your Kafka cluster's data.
can set the ```zookeeper_chroot``` parameter on the ```kafka``` class to do this.
t, you'll need to create the znode manually yourself. You can use
$ kafka zookeeper-shell
WATCHER::
WatchedEvent state:SyncConnected type:None path:null [zk: kraken-zookeeper(CONNECTED) 0] create /my_kafka kafka Created /my_kafka
can use whatever chroot znode path you like. The second argument
data```) is arbitrary. I used 'kafka' here.
:
class { 'kafka::server':
brokers => {
'kafka-node01.example.com' => { 'id' => 1, 'port' => 12345 },
'kafka-node02.example.com' => { 'id' => 2 },
},
zookeeper_hosts => ['zk-node01:2181', 'zk-node02:2181', 'zk-node03:2181'],
# set zookeeper_chroot on the kafka class.
zookeeper_chroot => '/kafka/clusterA',
}
afka Mirror
a MirrorMaker is usually used for inter data center Kafka cluster replication
aggregation. You can consume from any number of source Kafka clusters, and
uce to a single destination Kafka cluster.
class { 'kafka::mirror':
destination_brokers => {
'kafka-aggregator01.example.com' => { 'id' => 11 },
'kafka-aggregator02.example.com' => { 'id' => 12 },
},
topic_whitelist => 'webrequest.*',
}
kafka::mirror::consumer { 'clusterA':
zookeeper_hosts => ['zk-node01:2181', 'zk-node02:2181', 'zk-node03:2181'],
zookeeper_chroot => ['/kafka/clusterA'],
} kafka::mirror::consumer { 'clusterB':
zookeeper_hosts => ['zk-node01:2181', 'zk-node02:2181', 'zk-node03:2181'],
zookeeper_chroot => ['/kafka/clusterB'],
}
mxtrans monitoring
module contains a class called ```kafka::server::jmxtrans```. It contains
eful jmxtrans JSON config object that can be used to tell jmxtrans to send
ny output writer (Ganglia, Graphite, etc.). To you use this, you will need
[puppet-jmxtrans](https://github.com/wikimedia/puppet-jmxtrans) module.
class { '::kafka::server::jmxtrans':
ganglia => 'ganglia.example.com:8649',
}
will install jmxtrans and start render JSON config files for sending
and Kafka Broker stats to Ganglia.
[kafka-jmxtrans.json.md](kafka-jmxtrans.json.md) for a fully
ered jmxtrans Kafka JSON config file.