HubSpot/collectd-mapreduce

Name: collectd-mapreduce

Owner: HubSpot

Description: Parses mapreduce metrics from the hadoop-metrics framework

Created: 2015-11-21 21:26:29.0

Updated: 2015-11-21 21:27:08.0

Pushed: 2015-11-21 21:35:02.0

Homepage: null

Size: 5

Language: Python

GitHub Committers

UserMost Recent Commit# Commits

Other Committers

UserEmailMost Recent Commit# Commits

README

Mapreduce CollectD plugin

A CollectD plugin to collect MapReduce stats and metrics. Uses CollectD's Python plugin.

Tested on CDH4 and CDH5 JobTrackers and TaskTrackers. Requires Collectd

Prepration

Hadoop metrics should be enabled. This is done by editing /etc/hadoop/conf/hadoop-metrics.properties.
For each class listed in the file, change to org.apache.hadoop.metrics.spi.NoEmitMetricsContext.
Restart the jobtracker or tasktracker to pick up new configuration.
Test by going to http://hostname:port/metrics for the hostname and port of your jobtracker/tasktracker.

Installation
  1. Ensure hadoop-metrics is enabled.
  2. Place all three .py files in /usr/share/collectd/python/collectd-mapreduce/
  3. Place mapreduce-jobtracker.conf and/or mapreduce-tasktracker.conf in your /etc/collectd.d/ or equivalent configuration directory.
  4. Restart collectd
This work is supported by the National Institutes of Health's National Center for Advancing Translational Sciences, Grant Number U24TR002306. This work is solely the responsibility of the creators and does not necessarily represent the official views of the National Institutes of Health.