Name: doctorkafka
Owner: Pinterest
Description: DoctorKafka is a service for Kafka cluster auto healing and workload balancing
Created: 2017-08-21 15:30:24.0
Updated: 2018-05-23 16:15:24.0
Pushed: 2018-05-23 20:57:22.0
Homepage: null
Size: 216
Language: Java
GitHub Committers
User | Most Recent Commit | # Commits |
---|
Other Committers
User | Most Recent Commit | # Commits |
---|
DoctorKafka is a service for Kafka cluster auto healing and workload balancing. DoctorKafka can automatically detect broker failure and reassign the workload on the failed nodes to other nodes. DoctorKafka can also perform load balancing based on topic partitions's network usage, and makes sure that broker network usage does not exceed the defined settings. DoctorKafka sends out alerts when it is not confident on taking actions.
Design details are available in docs/DESIGN.md.
clone [git-repo-url] doctorkafka
octorkafka
package -pl kafkastats -am
Kafkastats is a kafka broker stats collector that runs on kafka brokers and reports broker stats to some kafka topic based on configuration. The following is the kafkastats usage.
-jmxport <kafka jmx port number> kafka process jmx port number
-kafka_config <arg> kafka server properties file path
-ostrichport <arg> ostrich port
-pollingintervalinseconds <arg> kafka broker stats polling interval in seconds
-topic <arg> the kafka topic that metric messages are sent to
-tsdhostport <arg> tsd host and port for tsdb
-uptimeinseconds <arg> how often kafkastats should restart itself.
-zookeeper <arg> zookeeper url for the metrics topic
The following is a sample command line for running kafkastats collector:
-server -cp /opt/kafkastats:/opt/kafkastats/*:/opt/kafkastats/lib/* -Dlog4j.configuration= com.pinterest.doctorkafka.stats.KafkaStatsMain -jmxport 9999 -topic brokerstats -zookeeper zookeeper001:2181/cluster1 -tsdhostport localhost:18126 -ostrichport 2051 -uptimeinseconds 3600 -pollingintervalinseconds 60 -kafka_config /etc/kafka/server.properties
Using the above command as an example, after the kafkastats process is up, we can check the process stats using `curl -s
` command, and view the logs under /var/log/kafkastats.
-s localhost:2051/stats.txt
The following is a sample upstart scripts for automatically restarting kafkastats if it fails:
tart on runlevel [2345]
espawn
espawn limit 20 5
nv NAME=kafkastats
nv JAVA_HOME=/usr/lib/jvm/java-8-oracle
nv STATSCOLLECTOR_HOME=/opt/kafkastats
nv LOG_DIR=/var/log/kafkastats
nv HOSTNAME=$(hostname)
cript
DAEMON=$JAVA_HOME/bin/java
CLASSPATH=$STATSCOLLECTOR_HOME:$STATSCOLLECTOR_HOME/*:$STATSCOLLECTOR_HOME/lib/*
DAEMON_OPTS="-server -Xmx800M -Xms800M -verbosegc -Xloggc:$LOG_DIR/gc.log \
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=20 -XX:GCLogFileSize=20M \
-XX:+UseG1GC -XX:MaxGCPauseMillis=250 -XX:G1ReservePercent=10 -XX:ConcGCThreads=4 \
-XX:ParallelGCThreads=4 -XX:G1HeapRegionSize=8m -XX:InitiatingHeapOccupancyPercent=70 \
-XX:ErrorFile=$LOG_DIR/jvm_error.log \
-cp $CLASSPATH"
exec $DAEMON $DAEMON_OPTS -Dlog4j.configuration=${LOG_PROPERTIES} \
com.pinterest.kafka.stats.KafkaStatsMain \
-jmxport 9999 -topic brokerstats \
-zookeeper zookeeper001:2181/cluster1 \
-tsdhostport localhost:18126 -ostrichport 2051 \
-uptimeinseconds 3600 -pollingintervalinseconds 60 \
-kafka_config /etc/kafka/server.properties
Edit drkafka/config/*.properties
files to specify parameters describing the environment. Those files contain comments describing the meaning of individual parameters.
package -pl drkafka -am
h
package
r ${DOCTORKAFKA_INSTALL_DIR} # directory to place Secor binaries in.
-zxvf target/doctorkafka-0.1.0-bin.tar.gz -C ${DOCTORKAFKA_INSTALL_DIR}
{DOCTORKAFKA_INSTALL_DIR}
-server -cp lib/*:doctorkafka-0.1.0.jar com.pinterest.doctorkafka.DoctorKafkaMain \
onfig drkafka/config/doctorkafka.prod.properties -topic brokerstats \
ookeeper zookeeper001:2181/cluster1 -ostrichport 2052 \
sdhostport localhost:18261 -uptimeinseconds 86400
Edit src/main/config/*.properties
files to specify parameters describing the environment.
Those files contain comments describing the meaning of individual parameters.
DoctorKafka comes with a number of tools implementing interactions with the environment.
{DOCTORKAFKA_INSTALL_DIR}
8 -Dlog4j.configurationFile=file:./log4j2.xml -cp lib/*:doctorkafka-0.1.0.jar \
com.pinterest.doctorkafka.tools.ClusterLoadBalancer \
-brokerstatstopic brokerstats -brokerstatszk zookeeper001:2181/cluster1 \
-clusterzk zookeeper001:2181,zookeeper002:2181,zookeeper003:2181/cluster2 \
-config ./drkafka/config/doctorkafka.prod.properties \
-inbound_limit_mb 40 -outbound_limit_mb 80
Cluster load balancer balances the workload among brokers to make sure the broker network usage does not exceed the threshold.
DoctorKafka is distributed under Apache License, Version 2.0.