Name: prometheus-alertmanager
Owner: OpenShift
Description: Prometheus Alertmanager
Forked from: prometheus/alertmanager
Created: 2017-07-20 12:53:33.0
Updated: 2017-07-20 23:06:58.0
Pushed: 2018-02-15 16:26:33.0
Size: 20428
Language: Go
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integrations such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts.
There are various ways of installing Alertmanager.
Precompiled binaries for released versions are available in the download section on prometheus.io. Using the latest production release binary is the recommended way of installing Alertmanager.
Docker images are available on Quay.io.
You can either go get it:
15VENDOREXPERIMENT=1 go get github.com/prometheus/alertmanager/cmd/...
$GOPATH/src/github.com/prometheus/alertmanager
ertmanager --config.file=<your_file>
Or checkout the source code and build manually:
dir -p $GOPATH/src/github.com/prometheus
$GOPATH/src/github.com/prometheus
t clone https://github.com/prometheus/alertmanager.git
alertmanager
ke build
alertmanager --config.file=<your_file>
You can also build just one of the binaries in this repo by passing a name to the build function:
ke build BINARIES=amtool
This is an example configuration that should cover most relevant aspects of the new YAML configuration format. The full documentation of the configuration can be found here.
al:
The smarthost and SMTP sender used for mail notifications.
tp_smarthost: 'localhost:25'
tp_from: 'alertmanager@example.org'
e root route on which each incoming alert enters.
e:
The root route must not have any matchers as it is the entry point for
all alerts. It needs to have a receiver configured so alerts that do not
match any of the sub-routes are sent to someone.
ceiver: 'team-X-mails'
The labels by which incoming alerts are grouped together. For example,
multiple alerts coming in for cluster=A and alertname=LatencyHigh would
be batched into a single group.
oup_by: ['alertname', 'cluster']
When a new group of alerts is created by an incoming alert, wait at
least 'group_wait' to send the initial notification.
This way ensures that you get multiple alerts for the same group that start
firing shortly after another are batched together on the first
notification.
oup_wait: 30s
When the first notification was sent, wait 'group_interval' to send a batch
of new alerts that started firing for that group.
oup_interval: 5m
If an alert has successfully been sent, wait 'repeat_interval' to
resend them.
peat_interval: 3h
All the above attributes are inherited by all child routes and can
overwritten on each.
The child route trees.
utes:
This routes performs a regular expression match on alert labels to
catch alerts that are related to a list of services.
match_re:
service: ^(foo1|foo2|baz)$
receiver: team-X-mails
# The service has a sub-route for critical alerts, any alerts
# that do not match, i.e. severity != critical, fall-back to the
# parent node and are sent to 'team-X-mails'
routes:
- match:
severity: critical
receiver: team-X-pager
match:
service: files
receiver: team-Y-mails
routes:
- match:
severity: critical
receiver: team-Y-pager
This route handles all alerts coming from a database service. If there's
no team to handle it, it defaults to the DB team.
match:
service: database
receiver: team-DB-pager
# Also group alerts by affected database.
group_by: [alertname, cluster, database]
routes:
- match:
owner: team-X
receiver: team-X-pager
- match:
owner: team-Y
receiver: team-Y-pager
hibition rules allow to mute a set of alerts given that another alert is
ring.
use this to mute any warning-level notifications if the same alert is
ready critical.
bit_rules:
urce_match:
severity: 'critical'
rget_match:
severity: 'warning'
Apply inhibition if the alertname is the same.
ual: ['alertname']
ivers:
me: 'team-X-mails'
ail_configs:
to: 'team-X+alerts@example.org, team-Y+alerts@example.org'
me: 'team-X-pager'
ail_configs:
to: 'team-X+alerts-critical@example.org'
gerduty_configs:
routing_key: <team-X-key>
me: 'team-Y-mails'
ail_configs:
to: 'team-Y+alerts@example.org'
me: 'team-Y-pager'
gerduty_configs:
routing_key: <team-Y-key>
me: 'team-DB-pager'
gerduty_configs:
routing_key: <team-DB-key>
amtool is a cli tool for interacting with the alertmanager api. It is bundled with all releases of alertmanager.
Alternatively you can install with:
et github.com/prometheus/alertmanager/cmd/amtool
View all currently firing alerts
tool alert
tname Starts At Summary
_Alert 2017-08-02 18:30:18 UTC This is a testing alert!
_Alert 2017-08-02 18:30:18 UTC This is a testing alert!
k_Foo_Fails 2017-08-02 18:30:18 UTC This is a testing alert!
k_Foo_Fails 2017-08-02 18:30:18 UTC This is a testing alert!
View all currently firing alerts with extended output
tool -o extended alert
ls Annotations Starts At Ends At Generator URL
tname="Test_Alert" instance="node0" link="https://example.com" summary="This is a testing alert!" 2017-08-02 18:31:24 UTC 0001-01-01 00:00:00 UTC http://my.testing.script.local
tname="Test_Alert" instance="node1" link="https://example.com" summary="This is a testing alert!" 2017-08-02 18:31:24 UTC 0001-01-01 00:00:00 UTC http://my.testing.script.local
tname="Check_Foo_Fails" instance="node0" link="https://example.com" summary="This is a testing alert!" 2017-08-02 18:31:24 UTC 0001-01-01 00:00:00 UTC http://my.testing.script.local
tname="Check_Foo_Fails" instance="node1" link="https://example.com" summary="This is a testing alert!" 2017-08-02 18:31:24 UTC 0001-01-01 00:00:00 UTC http://my.testing.script.local
In addition to viewing alerts you can use the rich query syntax provided by alertmanager
tool -o extended alert query alertname="Test_Alert"
ls Annotations Starts At Ends At Generator URL
tname="Test_Alert" instance="node0" link="https://example.com" summary="This is a testing alert!" 2017-08-02 18:31:24 UTC 0001-01-01 00:00:00 UTC http://my.testing.script.local
tname="Test_Alert" instance="node1" link="https://example.com" summary="This is a testing alert!" 2017-08-02 18:31:24 UTC 0001-01-01 00:00:00 UTC http://my.testing.script.local
tool -o extended alert query instance=~".+1"
ls Annotations Starts At Ends At Generator URL
tname="Test_Alert" instance="node1" link="https://example.com" summary="This is a testing alert!" 2017-08-02 18:31:24 UTC 0001-01-01 00:00:00 UTC http://my.testing.script.local
tname="Check_Foo_Fails" instance="node1" link="https://example.com" summary="This is a testing alert!" 2017-08-02 18:31:24 UTC 0001-01-01 00:00:00 UTC http://my.testing.script.local
tool -o extended alert query alertname=~"Test.*" instance=~".+1"
ls Annotations Starts At Ends At Generator URL
tname="Test_Alert" instance="node1" link="https://example.com" summary="This is a testing alert!" 2017-08-02 18:31:24 UTC 0001-01-01 00:00:00 UTC http://my.testing.script.local
Silence an alert
tool silence add alertname=Test_Alert
e22e-ca14-4aa0-932c-ca2f3445f926
tool silence add alertname="Test_Alert" instance=~".+0"
b58a-0b17-49ba-b734-3585139b1d25
View silences
tool silence query
Matchers Ends At Created By Comment
e22e-ca14-4aa0-932c-ca2f3445f926 alertname=Test_Alert 2017-08-02 19:54:50 UTC kellel
tool silence query instance=~".+0"
Matchers Ends At Created By Comment
b58a-0b17-49ba-b734-3585139b1d25 alertname=Test_Alert instance=~.+0 2017-08-02 22:41:39 UTC kellel
Expire a silence
tool silence expire b3ede22e-ca14-4aa0-932c-ca2f3445f926
Expire all silences matching a query
tool silence query instance=~".+0"
Matchers Ends At Created By Comment
b58a-0b17-49ba-b734-3585139b1d25 alertname=Test_Alert instance=~.+0 2017-08-02 22:41:39 UTC kellel
tool silence expire $(amtool silence -q query instance=~".+0")
tool silence query instance=~".+0"
Expire all silences
tool silence expire $(amtool silence query -q)
Amtool allows a config file to specify some options for convenience. The default config file paths are $HOME/.config/amtool/config.yml or /etc/amtool/config.yml
An example configfile might look like the following:
fine the path that amtool can find your `alertmanager` instance at
tmanager.url: "http://localhost:9093"
erride the default author. (unset defaults to your username)
or: me@example.com
rce amtool to give you an error if you don't include a comment on a silence
ent_required: true
t a default output format. (unset defaults to simple)
ut: extended
Warning: High Availability is under active development
To create a highly available cluster of the Alertmanager the instances need to
be configured to communicate with each other. This is configured using the
-mesh.* flags.
--mesh.peer-id string: mesh peer ID (default “<hardware-mac-address>“)--mesh.listen-address string: mesh listen address (default “0.0.0.0:6783”)--mesh.nickname string: mesh peer nickname (default “<machine-hostname>“)--mesh.peer value: initial peers (repeat flag for each additional peer)The mesh.peer-id flag is used as a unique ID among the peers. It defaults to
the MAC address, therefore the default value should typically be a good option.
The same applies to the default of the mesh.nickname flag, as it defaults to
the hostname.
The chosen port in the mesh.listen-address flag is the port that needs to be
specified in the mesh.peer flag of the other peers.
To start a cluster of three peers on your local machine use goreman and the
Procfile within this repository.
goreman start
To point your Prometheus 1.4, or later, instance to multiple Alertmanagers, configure them
in your prometheus.yml configuration file, for example:
ting:
ertmanagers:
static_configs:
- targets:
- alertmanager1:9093
- alertmanager2:9093
- alertmanager3:9093
Important: Do not load balance traffic between Prometheus and its Alertmanagers, but instead point Prometheus to a list of all Alertmanagers. The Alertmanager implementation expects all alerts to be sent to all Alertmanagers to ensure high availability.
Refer to ui/app/CONTRIBUTING.md.
