Name: grafana-spark-dashboards
Owner: Hammer Lab
Description: Scripts for generating Grafana dashboards for monitoring Spark jobs
Created: 2015-01-25 06:42:38.0
Updated: 2018-01-12 07:21:46.0
Pushed: 2015-03-26 15:49:49.0
Size: 377
Language: JavaScript
GitHub Committers
User | Most Recent Commit | # Commits |
---|
Other Committers
User | Most Recent Commit | # Commits |
---|
This repository contains a Grafana “scripted dashboard”, spark.js
, designed to display metrics collected from Spark applications. You can read more about the background and motivation here.
Beautiful graphs of all of your Spark metrics!
Here's a diagram of most of the pieces involved in our Spark-on-YARN + Graphite + Grafana infrastructure that contributes to the above graphs:
There are several pieces that need to be installed and made to talk to each other here:
Each of these steps is at least briefly discussed below.
This can be an arduous process, but try following the instructions at the Graphite docs or in the various guides around the internet.
This StackOverflow answer that I wrote explains the process for configuring Spark to send metrics to Graphite.
Alternatively, you can modify your metrics.properties
under conf
folder on each node, then you can submit your application without append such --conf
--file
arguments.
The Grafana docs are pretty good, but a little lacking the “quick start” department. The basic steps you need to follow are:
clone git@github.com:grafana/grafana.git
rafana
s config.sample.js src/config.js # create src/config.js from the provided sample.
t src/config.js: uncomment Graphite section and set the hostname:port to your Graphite's.>
Here is an example src/config.js
that I use, with hostnames and ports redacted.
nginx
Again, primary docs are always a good place to go, but here is an example nginx.conf
that I use that serves my Grafana files.
If you want to use Grafana's dashboard-saving and -loading functionality, the easiest thing to do is to point it at an elasticsearch
instance.
Install Elasticsearch, run it on the default port 9200
, and don't delete the elasticsearch portion of the sample src/config.js
I showed you.
After the above steps, you should be able to go to you <grafana host>:8090
and see stub “random walk” graphs.
This is easy:
s $THIS_REPO/spark.js $GRAFANA_REPO/src/app/dashboards/spark.js
Now you should be able to go to http://
If your Spark cluster is a standalone cluster, you can simply go to http://
spark.js
URL APIHere are the URL parameters that you can pass to spark.js
:
&app=<YARN app ID>
Using this is highly recommended: any unique substring of a YARN application ID that you can see on your ResourceManager's web UI will do.
For example, to obtain graphs for my latest job shown here:
I can simply pass ?app=0006
to spark.js
.
This will hit your ResourceManager's JSON API (via the proxy you've set up on the same host, port 8091
), find the application that matches 0006
, and pull in:
If you are not specifying the app
parameter, then the next three parameters should be included:
&prefix=<metric prefix>
Pass the full application ID (which is the YARN application ID if you are running Spark on YARN, otherwise the spark.app.id
configuration param that your Spark job ran with) here if it is not fetched via the app
parameter documented above.
&from=YYYYMMDDTHHMMSS
, &to=YYYYMMDDTHHMMSS
These will be inferred from the YARN application if the app
param is used, otherwise they should be set manually; defaults are now-1h
and now
.
&maxExecutorId=<N>
Tell spark.js
how many per-executor graphs to draw, and how to initialize some sane values of the $executorRange
template variable.
&collapseExecutors=<bool>
Collapse the top row containing per-executor JVM statistics, which can commonly be quite large and take up many folds of screen-height.
Default: true
.
&executors=<ranges>
Comma-delimited list of dash-delimited pairs of integers denoting specific executors to show.
All ranges passed here, as well as their union, will be added as options to the $executorRange
template variable.
Example: 1-12,22-23
.
&sharedTooltip=<bool>
Toggle whether each graph's tooltip shows values for every plotted metric at a given x-axis value or for just a single metric that's being moused over.
Default: true
.
&executorLegends=<bool>
Show legends on per-executor graphs.
Default: true
.
&legends=<bool>
Show legends on graphs other than per-executor ones discussed above.
Default: false
. Many of these panels can plot 100s of executors at the same time, causing the legend to be cumbersome.
&percentilesAndTotals=<bool>
Render n
th-percentiles and sums on certain graphs; can slow down rendering.
Default: false
.
spark.js
Templated Variablesspark.js
exposes three templated variables that can be dynamically changed and cause dashboard updates:
$prefix
: the first piece of your Spark metrics' names; analogous to the prefix
URL param.$executorRange
: ranges of executors to restrict graphs that plot multiple executors' values of a given metric to.$driver
: typically unused; when sending metrics from Spark to Graphite via StatsD, the “driver” identifier can lose its angle-brackets. This variable provides an escape hatch in that situation.Please file issues if you run into any problems, as this is fairly “alpha”.