Name: bec-alerts
Owner: Mozilla Services
Description: null
Created: 2018-04-19 21:35:45.0
Updated: 2018-05-23 23:00:20.0
Pushed: 2018-05-24 17:31:21.0
Homepage: null
Size: 81
Language: Python
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
This is a prototype service that reads processed events from Sentry (exported via Amazon SQS) and evaluates several rules to determine if it should send email alerts.
Prerequisites:
Docker 18.03.0
docker-compose 1.21.0
Clone the repository:
clone https://github.com/mozilla/bec-alerts.git
ec-alerts
Build the Docker image:
er-compose build
Initialize Sentry and create an admin account (requires user input):
er-compose run sentry sentry upgrade
Initialize processor database:
er-compose up -d processor-postgres
er-compose run processor bec-alerts manage migrate
Start up services:
er-compose up
Visit http://localhost:9000 and finish Sentry setup via the web interface:
Use http://localhost:9000 as the base URL of the install.
Create a new project. It's easiest to select “Python” as the project type.
Once you see the “Getting Started” page that includes a code sample of sending an error via Python, copy the DSN (which looks like http://really_long_hash@localhost:9000/2) to a .env file at the root of your repo that should look like this:
ULATE_SENTRY_DSN=http://really_long_hash@sentry:9000/2
Important: You must change the localhost in the DSN to sentry, since it will be used in a Docker container where Sentry is not running on localhost.
After copying the DSN, go to the project settings for the project you just created. Select the All Integrations subsection, and click the link to configure the “Amazon SQS Standalone” plugin.
Enter the following config values:
asdfasdfSave the config and click the “Enable Plugin” button in the top right.
After configuring the plugin and enabling it, you should be able to submit errors to Sentry and see logging confirming that the processor received the post-processed events from Sentry.
To run the processor and related services:
er-compose up
If you've set SIMULATE_SENTRY_DSN in your .env file properly, you can simulate sending an error to your running processor instance with the simulate_error command:
er-compose run processor bec-alerts simulate_error
After you've processed an error, you can evaluate the configured alert triggers with the watcher:
er-compose run processor bec-alerts watcher --once --console-alerts --dry-run
--once instructs the watcher to only run once; normally it will continue to run once every 5 minutes after it starts up.--console-alerts enables logging alerts instead of sending them as emails.--dry-run disables saving records of the run and other persistent data. This helps avoid issues with triggers that don't notify users of issues they've already seen.You should see log output stating how many issues were evaluated, and messages
logged to the console if any alerts were triggered. It may be useful to set the
enabled property on the example triggers in bec_alerts/triggers.py, which
will trigger alerts whenever a new event is seen or when new events that the user has not seen before are seen.
You can run the test suite using pytest in the processor container:
er-compose run processor pytest /app
The service consists of a few AWS services and a Docker image that is intended to be run on EC2. The required AWS resources are:
sentry_errors by default) that Sentry can write to and the app can read from. The app will create the queue on startup itself.postgresql-hll extension installed and enabled.AWS credentials are pulled from IAM. The region is provided by the AWS_DEFAULT_REGION environment variable.
The Docker image defined in Dockerfile is used to run two separate processes: the processor, and the watcher, described in the following sections.
The following environment variables are available for all processes:
| Name | Required? | Default | Description |
| —- | ——— | ——- | ———– |
| AWS_DEFAULT_REGION | :white_check_mark: | | Region for connecting to AWS |
| DATABASE_URL | :white_check_mark: | | URL with connection data for the database. Typically a postgres URL of the form postgres://user:password@host/database_name. |
| DJANGO_SECRET_KEY | :white_check_mark: | | Secret key for Django's cryptographic signing. We don't really use it but Django fails to start without it. Set to a secret, random string. |
| SENTRY_DSN | :white_check_mark: | | DSN used to send errors to Sentry. |
| AWS_CONNECT_TIMEOUT | :x: | 30 | Timeout for connecting to AWS |
| AWS_READ_TIMEOUT | :x: | 30 | Timeout for reading a response from AWS |
| LOG_FORMAT | :x: | mozlog | Format to output logs in. Use mozlog for mozlog-compatible logs, or compose for a human-readable format suited for docker-compose output. |
| LOG_LEVEL | :x: | INFO | Minimum log level to output. One of CRITICAL, ERROR, WARNING, INFO, or DEBUG. |
Command: bec-alerts processor
Reads from SQS to fetch incoming events from Sentry, processes them, and saves the resulting data to Postgres. This process will launch and manage a pool of subprocesses for listening and processing data.
The following environment variables are available for the processor:
| Name | Required? | Default | Description |
| —- | ——— | ——- | ———– |
| SQS_QUEUE_URL | :x: | | URL of the queue to poll for events. If not provided, a queue will be created on startup with the default name sentry_errors. |
| SQS_ENDPOINT_URL | :x: | | Endpoint URL for connection to AWS. Only required for local development. |
| PROCESSOR_PROCESS_COUNT | :x: | System CPU count | Number of worker processes to start |
| PROCESSOR_WORKER_MESSAGE_COUNT | :x: | 200 | Number of messages a worker should process before terminating; a new worker process is started to take its place. Workers may process slightly more messages than this number before shutting down due to message batching. |
Command: bec-alerts watcher
Periodically checks for events that have been processed since the last run, and evaluates alert triggers to determine if we should send a notification to users. This process implements its own sleep timer.
The following environment variables are available for the watcher:
| Name | Required? | Default | Description |
| —- | ——— | ——- | ———– |
| SES_FROM_EMAIL | :white_check_mark: | notifications@sentry.prod.mozaws.net | Email to use in the From field for notifications |
| DATADOG_COUNTER_NAME | :x: | bec-alerts.watcher.health | Name of the DogStatsD counter to increment for health checks. |
| SES_VERIFY_EMAIL | :x: | False | If True, the watcher will attempt to verify the SES_VERIFY_EMAIL via API on startup. Should probably be False in production. |
| WATCHER_SLEEP_DELAY | :x: | 300 | Seconds to wait between checking for new events and alert triggers |
| SES_ENDPOINT_URL | :x: | | Endpoint URL for connection to AWS. Only required for local development. |
Each time the Watcher process evalutes triggers and potentially sends emails, it increments a counter in Datadog using DogStatsD. This behavior is controlled by the DATADOG_API_KEY and DATADOG_COUNTER_NAME environment variables.
This counter can be used to monitor if the watcher is still running.
A health check trigger exists to test the entire flow from an error being submitted to Sentry to an alert email being sent. To trigger it, send an error to Sentry with the string bec-alerts-healthcheck in the message field. Ops and dev will be sent an alert email confirming that the event was received and processed successfully.
Browser Error Collection Notifications is licensed under the MPL 2.0. See the LICENSE file for details.
The file sentry/sqs_plugin.py is a fork of an official Sentry plugin provided by Sentry. It is covered under the Apache License, version 2.0. See the file's comments for details.