cloudfoundry/garden-performance-acceptance-tests

Name: garden-performance-acceptance-tests

Owner: Cloud Foundry

Description: null

Created: 2016-11-28 10:39:17.0

Updated: 2016-11-29 10:02:21.0

Pushed: 2017-10-16 14:33:22.0

Homepage: null

Size: 1096

Language: Go

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

GPATs

These are the Garden Performance Acceptance Tests.

About

These tests run as part of the garden-runc pipeline and will gate us from shipping a new release if they fail. With this in mind, the goal is to keep this test suite as minimal and quick as possible.

Metrics generated by the test suite are currently sent to DataDog. This is achieved by wiring up the custom DataDog Reporter into Ginkgo. In the future, should we wish to send metrics to other services, we can extend the suite by writing and wiring up a new custom reporter for that service.

Note that the reporter package used here is based on this reporter package from the cloudfoundry/benchmarkbbs repo.

Conditionally Expect Metrics

This suite is used both to gate releases via expectations on performance but also to provide metrics when thresholds are succeeded.

To achieve this, metrics related expectations are wrapped in Conditionally() functions, to ensure they only fail tests when required.

To turn off metrics related expectations, set an environment variable: IGNORE_PERF_EXPECTATIONS=true.

Usage

DOG_API_KEY="x" DATADOG_APP_KEY="x" DATADOG_METRIC_PREFIX="x" GARDEN_ADDRESS="127.0.0.1:7777" ginkgo

To turn off metrics related expectations:

DOG_API_KEY="x" DATADOG_APP_KEY="x" DATADOG_METRIC_PREFIX="x" GARDEN_ADDRESS="127.0.0.1:7777" IGNORE_PERF_EXPECTATIONS=true ginkgo

NB: This test suite will destroy ALL containers on the Garden server as part of the run. You have been warned.