skbkontur/carbonate

Name: carbonate

Owner: Kontur

Description: Utilities for managing graphite clusters

Created: 2014-11-21 14:54:33.0

Updated: 2016-01-28 10:01:56.0

Pushed: 2016-01-28 10:01:55.0

Homepage:

Size: 78

Language: Python

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

Carbonate

“Pop bottles.” – Birdman

Graphite clusters are pretty cool. Here are some primitive tools to help you manage your graphite clusters.

All of the tools support two common arguments; the path to a config file, and the name of the cluster. Using these tools alongside a config file that describes your graphite clusters you can build up scripts to manage your metrics. Some of the tools could easily be replaced with one-liners in shell, but exist here for convenience and readability. The goal is to provide fast, predictable utilities that can easily be composed into more advanced tooling.

Install

Carbonate is available from Python official third party repository (aka PyPi) and as such can be installed via regular Python package managers. Note that you might have to install a python package manager (e.g. apt-get install python-setuptools on a ubuntu host)

install carbonate

The Config

Carbonate expects a configuration file that defines the clusters in your environment. The default config file is located at /opt/graphite/conf/carbonate.conf or can be provided on the command line. The default cluster is named 'main'.

n]
INATIONS = 192.168.9.13:2004:carbon01, 192.168.9.15:2004:carbon02, 192.168.6.20:2004:carbon03
ICATION_FACTOR = 2
USER = carbon

You should take care to match the list of destination IPs or hostnames to the nodes in your cluster. Though its worth noting that the ports and labels are currently not used by carbonate. Order is important because of how the consistent hash ring is created.

The replication factor should match the replication factor for the cluster.

Finally, you can choose to provide a SSH user that will be used when carbonate requires connecting to another node in the cluster to perform an operation. If this is not provided, then the current user executing the command will be chosen.

The Tools

carbon-hosts

e: carbon-hosts [-h] [-c CONFIG_FILE] [-C CLUSTER]

rn the addresses for all nodes in a cluster

onal arguments:
, --help            show this help message and exit
 CONFIG_FILE, --config-file CONFIG_FILE
                    Config file to use (default:
                    /opt/graphite/conf/carbonate.conf)
 CLUSTER, --cluster CLUSTER
                    Cluster name (default: main)

carbon-lookup

e: carbon-lookup [-h] [-c CONFIG_FILE] [-C CLUSTER] [-s] METRIC

up where a metric lives in a carbon cluster

tional arguments:
TRIC                Full metric name to search for

onal arguments:
, --help            show this help message and exit
 CONFIG_FILE, --config-file CONFIG_FILE
                    Config file to use (default:
                    /opt/graphite/conf/carbonate.conf)
 CLUSTER, --cluster CLUSTER
                    Cluster name (default: main)
, --short           Only display the address, without port and cluster
                    name (default: False)

carbon-list

e: carbon-list [-h] [-c CONFIG_FILE] [-C CLUSTER] [-d STORAGE_DIR]

 the metrics this carbon node contains

onal arguments:
, --help            show this help message and exit
 CONFIG_FILE, --config-file CONFIG_FILE
                    Config file to use (default:
                    /opt/graphite/conf/carbonate.conf)
 CLUSTER, --cluster CLUSTER
                    Cluster name (default: main)
 STORAGE_DIR, --storage-dir STORAGE_DIR
                    Storage dir (default: /opt/graphite/storage/whisper)

carbon-sieve

e: carbon-sieve [-h] [-c CONFIG_FILE] [-C CLUSTER] [-f METRICS_FILE]
                [-n NODE] [-I]

n a list of metrics, output those that belong to a node

onal arguments:
, --help            show this help message and exit
 CONFIG_FILE, --config-file CONFIG_FILE
                    Config file to use (default:
                    /opt/graphite/conf/carbonate.conf)
 CLUSTER, --cluster CLUSTER
                    Cluster name (default: main)
 METRICS_FILE, --metrics-file METRICS_FILE
                    File containing metric names to filter, or '-' to read
                    from STDIN (default: -)
 NODE, --node NODE  Filter for metrics belonging to this node (default:
                    self)
, --invert          Invert the sieve, match metrics that do NOT belong to
                    a node (default: False)

carbon-sync

e: carbon-sync [-h] [-c CONFIG_FILE] [-C CLUSTER] [-f METRICS_FILE] -s
               SOURCE_NODE [-d STORAGE_DIR] [-b BATCH_SIZE]
               [--source-storage-dir SOURCE_STORAGE_DIR]
               [--rsync-options [RSYNC_OPTIONS [RSYNC_OPTIONS ...]]]

 local metrics using remote nodes in the cluster

onal arguments:
, --help            show this help message and exit
 CONFIG_FILE, --config-file CONFIG_FILE
                    Config file to use (default:
                    /opt/graphite/conf/carbonate.conf)
 CLUSTER, --cluster CLUSTER
                    Cluster name (default: main)
 METRICS_FILE, --metrics-file METRICS_FILE
                    File containing metric names to filter, or '-' to read
                    from STDIN (default: -)
 SOURCE_NODE, --source-node SOURCE_NODE
                    Override the source for metrics data (default: None)
 STORAGE_DIR, --storage-dir STORAGE_DIR
                    Storage dir (default: /opt/graphite/storage/whisper)
 BATCH_SIZE, --batch-size BATCH_SIZE
                    Batch size for the rsync job (default: 1000)
source-storage-dir SOURCE_STORAGE_DIR
                    Source storage dir (default:
                    /opt/graphite/storage/whisper)
rsync-options [RSYNC_OPTIONS [RSYNC_OPTIONS ...]]
                    Pass option(s) to rsync (default: -azpS)

carbon-path

e: carbon-path [-h] [-c CONFIG_FILE] [-C CLUSTER] [-f METRICS_FILE] [-r]
               [-p] [-d STORAGE_DIR]

sform metric paths to (or from) filesystem paths

onal arguments:
, --help            show this help message and exit
 CONFIG_FILE, --config-file CONFIG_FILE
                    Config file to use (default:
                    /opt/graphite/conf/carbonate.conf)
 CLUSTER, --cluster CLUSTER
                    Cluster name (default: main)
 METRICS_FILE, --metrics-file METRICS_FILE
                    File containing metric names to transform to file
                    paths, or '-' to read from STDIN (default: -)
, --reverse         Transform from file paths to metric paths (default:
                    False)
, --prepend         Prepend storage dir to file paths (default: False)
 STORAGE_DIR, --storage-dir STORAGE_DIR
                    Whisper storage directory to prepend when -p given
                    (default: /opt/graphite/storage/whisper)

whisper-aggregate

e: whisper-aggregate [-h] [-f METRICS_FILE] [-d STORAGE_DIR]

aggregation for whisper-backed metrics this carbon instance contains

onal arguments:
, --help            show this help message and exit
 METRICS_FILE, --metrics-file METRICS_FILE
                    File containing metric names and aggregation modes, or
                    '-' to read from STDIN (default: -)
 STORAGE_DIR, --storage-dir STORAGE_DIR
                    Whisper storage directory (default:
                    /opt/graphite/storage/whisper)

whisper-fill

e: whisper-fill [-h] SRC DST

fill datapoints from one whisper file into another

tional arguments:
C         Whisper source file
T         Whisper destination file

onal arguments:
, --help  show this help message and exit

Example usage

Resync a node in a cluster

in/sh

sync a node from other nodes in the cluster


L_IP="$1"

h in $(carbon-hosts) ; do

ssh $h -- carbon-list |
carbon-sieve -n $LOCAL_IP |
carbon-sync -s $h
&

Rebalance a cluster

in/sh

balance a cluster from one size to another. Remember to cleanup metrics
at no longer belong when all nodes are rebalanced!


L_IP="$1"
CLUSTER="old"
CLUSTER="main"

h in $(carbon-hosts -C "$OLD_CLUSTER") ; do
h $h -- carbon-list |
rbon-sieve -C "$NEW_CLUSTER" -n $LOCAL_IP |
rbon-sync -s $h

List metrics that don't belong

in/sh

st metrics from disk that don't belong


L_IP="$1"

on-list | carbon-sieve -I -n $LOCAL_IP

License and warnings

These tools should be considered beta quality right now. Tests exist for most functionality, but there is still significant work to be done to make them bullet-proof. However, instead of sitting on this code, I'd rather release it and allow others to provide input and help guide the development. So, expect a few bugs and please help me fix them!

The code is available under the MIT license.