Name: gotel
Owner: Crowdstrike
Description: GoTel - Scheduled job monitoring
Created: 2014-08-05 15:43:10.0
Updated: 2018-01-01 13:26:01.0
Pushed: 2017-10-01 02:36:11.0
Homepage: null
Size: 2453
Language: Go
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
Who monitors the monitors?
CrowdStrike Cloud Engineering is releasing GoTel which is an internal monitoring service that aims to ensure scheduled jobs, cronjobs, batch oriented work, or general scheduled tasks are completing successfully and within a set SLA time period.
GoTel is for monitoring scheduled operations.
Most companies have scheduled reports, cron jobs, backup jobs, random data process tasks, etc…, various things that are expected to run perfectly but sometimes silently fail. GoTel will let them make a “reservation” which means they have to check in during their allotted time frame or alerts will be sent out to the world. When they run they can “checkin” to GoTel which updates their last check-in time.
This was born from years of experience with various “cron” type jobs that suddenly stop working because the job they ran had a port blocked on it from a firewall config, data sets grow and something that takes 10 minutes now takes 2 hours, something in the environment changed or a myriad of various other failure conditions occurred. GoTel is for when things need to run and you need an independent monitor in your network that is not locked in to a specific vendor. We've also seen alerting failures from 3rd party vendors that are supposed to give us the warm fuzzy feeling they'll alert us when things stop working.
For a toy example take the case where you have a nightly job that removes old data from a data store. It?s a simple one liner that runs every night. One day you hit the inflection point and indexes turn useless and your script now does a full table scan. Now your 20 minute data clean up job takes 5 hours and you didn?t know about it as soon as it happened.
GoTel is for when you don't need the overhead of an “enterprise” grade schedule monitor. It's for the microservice world where you have apps running in various languages, platforms and locations. It's expected that you have two GoTel instances up for redundancy (across datacenters). The coordinator will monitor the worker and the worker will monitor the coordinator to ensure GoTel is always operational and if not that alerts are sent out to avoid silent failures.
GoTel has been running inside CrowdStrike and has already caught cases in production of scheduled operations silently failing, reducing customer impact.
0.1 expects a MySQL backend for storing jobs and leader election. Future plugins will allow direct integration with ZooKeeper for leader election but v1 keeps the minimum requirements for easier outside adoption.
Below is a screenshot showing what the admin UI looks like when you browse to
http://localhost:8080/status
in MySQL create a “gotel” database
mysql> create database gotel;
Grab the code:
mkdir -p gotel_github/src
export GOPATH=~/gotel_github
cd gotel_github
go get github.com/CrowdStrike/gotel
go get github.com/go-sql-driver/mysql
go get github.com/ParsePlatform/go.flagenv
cd $GOPATH/src/github.com/CrowdStrike/gotel/cmd/gotelweb
./run.sh
navigate to: http://127.0.0.1:8080/status
0.1
GoTel uses a number of hotel related concepts. It's imagined that a small hotel owner becomes fond of her regular guests and gets sad when they don't checkin when they say they're supposed to.
GoTel allows for configurable alerters to be set so when an application doesn't checkin over it's SLA then we fire off to one or more alert systems.
Currently configured alerts:
To Enable: edit gotel.gcfg and set enable=true under [smtp]
pass in the flag -GOTEL_SMTP_HOST=10.10.1.1 (or whatever your smtp server address is)
ake a reservation that tells GoTel testapp/requests will complete work every 5 minutes or alert me
upported time_units currently are seconds,minutes,hours
otify parameter supports a comma-separated list of recipients that will receive an alert when a job fails to checkin
-XPOST 'http://127.0.0.1:8080/reservation' -i -H "Content-type: application/json" -d '
pp": "testapp",
omponent": "requests",
otify": "jim@foo.com",
requency": 5,
ime_units": "minutes",
wner": "jim@foo.com"
heckin for a reservation to avoid having alerts sent
-XPOST 'http://127.0.0.1:8080/checkin' -i -H "Content-type: application/json" -d '
pp": "testapp",
omponent": "requests",
otes": "all is well"
ause (snooze your wakeup call) a job if you're going down for maintenance or testing
-XPOST 'http://127.0.0.1:8080/snooze' -i -H "Content-type: application/json" -d '
pp": "testapp",
omponent": "requests",
uration": 10,
ime_units": "hours"
heckout/delete reservation
-XPOST 'http://127.0.0.1:8080/checkout' -i -H "Content-type: application/json" -d '
pp": "testapp",
omponent": "requests"
iew all reservations
'http://127.0.0.1:8080/reservation'
iew the status in your browser
://127.0.0.1:8080/status