Name: pytest-services
Owner: Mozilla Services
Description: Unit testing framework for test driven security of AWS configurations, and more.
Created: 2017-10-04 20:00:35.0
Updated: 2018-05-24 11:37:55.0
Pushed: 2018-05-21 23:42:15.0
Size: 373
Language: Python
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
Clients and pytest tests for checking that third party services the @foxsec team uses are configured correctly.
We trust third party services to return their status correctly, but want to answer questions whether they are configured properly such as:
Note: other versions may work too these are the versions @g-k used for development
From the project root run:
install
This will:
Activate the venv in the project root:
ce venv/bin/activate
To fetch RDS resources from the cache or AWS API and check that
backups are enabled for DB instances for the configured aws profile
named default in the us-west-2 region we can run:
st --ignore aws/s3 --ignore aws/ec2 -k test_rds_db_instance_backup_enabled -s --aws-profiles default --debug-calls
The options include pytest options:
--ignore to skip fetching resources for non-RDS resources-k for selecting tests matching the substring test_rds_db_instance_backup_enabled for the one test we want to run-m not used but the marker filter can be useful for selecting all tests for specific services (e.g. -m rds)-s to disable capturing stdout so we can see the progress fetching AWS resourcesand options pytest-services adds:
--debug-calls for printing (with -s) API calls we make--aws-profiles for selecting one or more AWS profiles to fetch resources for or the AWS default profile / AWS_PROFILE environment variable--offline a flag to tell HTTP clients to not make requests and return empty params--config path to test custom config fileand produces output like the following showing a DB instance with backups disabled:
======================================================= test session starts ===========================================================
form darwin -- Python 3.6.2, pytest-3.3.2, py-1.5.2, pluggy-0.6.0
data: {'Python': '3.6.2', 'Platform': 'Darwin-15.6.0-x86_64-i386-64bit', 'Packages': {'pytest': '3.3.2', 'py': '1.5.2', 'pluggy': '0.6.
'Plugins': {'metadata': '1.5.1', 'json': '0.4.0', 'html': '1.16.1'}}
dir: /Users/gguthe/mozilla-services/pytest-services, inifile:
ins: metadata-1.5.1, json-0.4.0, html-1.16.1
ecting 0 items c
ng AWSAPICall(profile='default', region='us-west-2', service='rds', method='describe_db_instances', args=[], kwargs={})
ecting 4 items
rds/test_rds_db_instance_backup_enabled.py ...F [100%]
============================================================ FAILURES =================================================================
___________________________________ test_rds_db_instance_backup_enabled[test-db] ________________________________________
db_instance = {'AllocatedStorage': 50, 'AutoMinorVersionUpgrade': True, 'AvailabilityZone': 'us-west-2c', 'BackupRetentionPeriod': 0, .
@pytest.mark.rds
@pytest.mark.parametrize('rds_db_instance',
rds_db_instances(),
ids=lambda db_instance: db_instance['DBInstanceIdentifier'])
def test_rds_db_instance_backup_enabled(rds_db_instance):
assert rds_db_instance['BackupRetentionPeriod'] > 0, \
'Backups disabled for {}'.format(rds_db_instance['DBInstanceIdentifier'])
AssertionError: Backups disabled for test-db
assert 0 > 0
rds/test_rds_db_instance_backup_enabled.py:12: AssertionError
======================================================= 72 tests deselected ===========================================================
======================================== 1 failed, 3 passed, 72 deselected in 3.12 seconds ============================================
The below policy will allow you to run all AWS tests in pytest-services against all resources in your account.
ersion": "2012-10-17",
tatement": [
{
"Sid": "PytestServicesReadOnly",
"Action": [
"autoscaling:DescribeLaunchConfigurations",
"ec2:DescribeInstances",
"ec2:DescribeSecurityGroups",
"ec2:DescribeVolumes",
"elasticache:DescribeCacheClusters",
"elasticloadbalancing:DescribeLoadBalancers",
"es:DescribeElasticsearchDomains",
"es:ListDomainNames",
"iam:GenerateCredentialReport",
"iam:GetCredentialReport",
"iam:GetLoginProfile",
"iam:ListAttachedGroupPolicies",
"iam:ListAttachedRolePolicies",
"iam:ListAttachedUserPolicies",
"iam:ListGroupPolicies",
"iam:ListGroupsForUser",
"iam:ListMFADevices",
"iam:ListRolePolicies",
"iam:ListRoles",
"iam:ListUserPolicies",
"iam:ListUsers",
"rds:DescribeDbInstances",
"rds:DescribeDbSecurityGroups",
"rds:DescribeDbSnapshotAttributes",
"rds:DescribeDbSnapshots",
"rds:ListTagsForResource",
"redshift:DescribeClusterSecurityGroups",
"redshift:DescribeClusters",
"s3:GetBucketAcl",
"s3:GetBucketCORS",
"s3:GetBucketLogging",
"s3:GetBucketPolicy",
"s3:GetBucketVersioning",
"s3:GetBucketWebsite",
"s3:ListAllMyBuckets",
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": "*"
}
Make sure to have an OAuth2 app created and have the client_secret.json file in ~/.credentials and then run:
setup_gsuite
The AWS client will use AWS API JSON responses when available and save them using AWS profile, region, service name, service method, botocore args and kwargs in the cache key to filenames with the format .cache/v/pytest_aws:<aws profile>:<aws region>:<aws service>:<service method>:<args>:<kwargs>.json e.g.
.cache/v/pytest_aws:cloudservices-aws-stage:us-west-2:rds:describe_db_instances::.json
"DBInstances": [
{
"AllocatedStorage": 5,
"AutoMinorVersionUpgrade": true,
"AvailabilityZone": "us-west-2c",
"BackupRetentionPeriod": 1,
"CACertificateIdentifier": "rds-ca-2015",
"CopyTagsToSnapshot": false,
"DBInstanceArn": "arn:aws:rds:us-west-2:123456678901:db:test-db",
These files can be removed individually or all at once with the pytest –cache-clear option.
pytest-services adds a --config cli option for passing in a custom config file specific to tests within pytest-services.
The example config in repo (config.yaml.example):
ptions:
test_name: test_ec2_instance_has_required_tags
test_param_id: i-0123456789f014c162
expiration_day: 2019-01-01
reason: ec2 instance has no owner
test_name: test_ec2_security_group_opens_specific_ports_to_all
test_param_id: '*HoneyPot'
expiration_day: 2020-01-01
reason: purposefully insecure security group
rities:
test_name: test_ec2_instance_has_required_tags
severity: INFO
test_name: '*'
severity: ERROR
essions:
test_name: test_ec2_security_group_opens_all_ports_to_all
test_param_id: '*mycustomgroup'
comment: this was remediated by ops team
er_is_inactive:
no_activity_since:
years: 1
months: 0
created_after:
weeks: 1
quired_tags:
- Name
- Type
- App
- Env
itelisted_ports_global:
- 25
itelisted_ports:
- test_param_id: '*bastion'
ports:
- 22
- 2222
pytest-services custom config format adds support for marking test and test resource IDs as expected failures.
The keys for each exemption rule is:
* to turn into a regex matcher)The config looks like:
ptions:
test_name: test_ec2_instance_has_required_tags
test_param_id: i-0123456789f014c162
expiration_day: 2019-01-01
reason: ec2 instance has no owner
test_name: test_ec2_security_group_opens_specific_ports_to_all
test_param_id: '*HoneyPot'
expiration_day: 2020-01-01
reason: purposefully insecure security group
You can prefix the test ID with a * to enable regex matching for the test ID. The * prefix will be stripped
off, and the rest will be used as a regex.
For example:
*foobar becomes foobar*foo\w+ becomes foo\w+For more information on Python's regex syntax see: Regular Expression HOWTO.
Note: All regex rules are applied first. As well, the ordering of both regex and non-regex rules is top to bottom and the first one wins.
When a json report is generated, the exemptions will show up in the json metadata as serialized markers:
on -m json.tool report.json | grep -C 20 xfail
"markers": {
"ec2": {
"name": "ec2",
"args": [],
"kwargs": {}
},
"parametrize": {
"name": "parametrize",
"args": [
"...skipped..."
],
"kwargs": [
"...skipped..."
]
},
"xfail": {
"name": "xfail",
"args": [],
"kwargs": {
"reason": "ec2 instance has no owner",
"strict": true,
"expiration": "2019-01-01"
}
}
},
pytest-services custom config format adds support for marking the severity of a certain test. A severity can be INFO, WARN, or ERROR.
These do not modify pytest results (pass, fail, xfail, skip, etc.).
The config looks like:
rities:
test_name: test_ec2_instance_has_required_tags
severity: INFO
test_name: '*'
severity: ERROR
And results in a severity and severity marker being included in the json metadata:
st --ignore aws/s3 --ignore aws/rds --ignore aws/iam -s --aws-profiles stage --aws-require-tags Name Type App Stack -k test_ec2_instance_has_required_tags --config config.yaml.example --json=report.json
son
on -m json.tool report.json
"report": {
"environment": {
"Python": "3.6.2",
"Platform": "Darwin-15.6.0-x86_64-i386-64bit"
},
"tests": [
{
"metadata": [
{
"markers": {
"severity": {
"name": "severity",
"args": [
"INFO"
],
"kwargs": {}
}
},
"severity": "INFO",
"unparametrized_name": "test_ec2_instance_has_required_tags"
}
pytest-services custom config format adds support for marking specific tests on specific resources as regressions.
As with severity this does not modify the pytest results, but rather adds a marker that can be used when analyzing the results.
The config looks like:
essions:
test_name: test_ec2_security_group_opens_all_ports_to_all
test_param_id: '*mycustomgroup'
comment: this was remediated by ops team
pytest-services has a suite of AWS tests. This section of the custom config includes configuration options specific to these tests.
The config looks like:
Relative time delta for test_iam_user_is_inactive. no_activity_since will be used as the failure marker,
so in this example any user that hasn't had any activity for a year will be marked as a "failure". created_after
is used as a grace period, so in this case any user that was created within the last week will be automatically
pass this test.
er_is_inactive:
no_activity_since:
years: 1
months: 0
created_after:
weeks: 1
Required tags used within the test_ec2_instance_has_required_tags test
quired_tags:
- Name
- Type
- App
- Env
Whitelsited ports for the test_ec2_security_group_opens_specific_ports_to_all
test for all instances
itelisted_ports_global:
- 25
Whitelsited ports for the test_ec2_security_group_opens_specific_ports_to_all
test for specific instances. In this example, we are whitelisting ports 22
and 2222 for all security groups that include the word 'bastion' in them.
itelisted_ports:
- test_param_id: '*bastion'
ports:
- 22
- 2222
pytest-services has a suite of GSuite tests. This section of the custom config includes configuration options specific to these tests.
Make sure to setup GSuite before running GSuite tests
The config looks like:
te:
The specific GSuite domain to test.
main: 'example.com'
Relative time delta for test_admin_user_is_inactive. no_activity_since will be used as the failure marker,
so in this example any user that hasn't had any activity for a year will be marked as a "failure".
er_is_inactive:
no_activity_since:
years: 1
months: 0
There are two important things to note about pytest-services tests that may be different from your expectations.
First, the focus is on “actionable results”. This plays out as an attempt to reduce false positives by trying to filter out unused resources. An example of this can be seen by looking at any of the security group tests, where we are skipping any security groups that are not attached to a resource.
Second, there are some tests that make naive assumptions instead of trying to capture the complexities of the system. The current best example of this is all IAM tests that relate to “admin” users. How we are determining what an user or role is an admin is based simply off substring matching on the policies attached. This obviously has a high chance of false negatives.
q is $bad_value?“)Currently this is a monolithic pytest package, but should eventually be extracted into a pytest plugin and with separate dependent pytest plugins for each service.
API responses should fit on disk and in memory (i.e. don't use this for log processing or checking binaries for malware), and be safe to cache for minutes, hours, or days (i.e. probably don't use this for monitoring a streaming API) (NB: bug for specifying data freshness).
Additionally we want:
resources.pyhelpers.pytest_helpers.py, resources.py, client.py)meta_test_ (and probably mock or pytest.monkeypatch)example_cache/v/st-services
example_cache
??? v
??? cache
? ??? lastfailed
??? pytest_aws:example-account:us-east-1:ec2:describe_instances::.json
??? pytest_aws:example-account:us-east-1:ec2:describe_security_groups::.json
<third party service A>
??? client.py
??? meta_test_client.py
??? <subservice A (optional)>
? ??? __init__.py
? ??? helpers.py
? ??? resources.py
? ??? ...
? ??? test_ec2_security_group_all_ports.py
??? <subservice b (optional)>
? ??? __init__.py
? ??? resources.py
? ??? ...
? ?? test_s3_bucket_web_hosting_disabled.py
<third party service B>
??? __init__.py
??? helpers.py
??? resources.py
??? test_user_has_escalation_policy.py
Let's write a test to check that http://httpbin.org/ip returns an AWS IP:
httpbin/test_httpbin_ip.py with the contents:rt itertools
rt ipaddress
rt pytest
rt json
rt urllib.request
get_httpbin_ips():
# IPs we always want to test
ips = [
'127.0.0.1',
'13.58.0.0',
]
req = urllib.request.Request('http://httpbin.org/ip')
with urllib.request.urlopen(req) as response:
body = response.read().decode('utf-8')
ips.append(json.loads(body).get('origin', None))
return ips
get_aws_ips():
req = urllib.request.Request('https://ip-ranges.amazonaws.com/ip-ranges.json')
with urllib.request.urlopen(req) as response:
body = response.read().decode('utf-8')
return json.loads(body)['prefixes']
est.mark.httpbin
est.mark.aws_ip_ranges
est.mark.parametrize(
['ip', 'aws_ip_ranges'],
zip(get_httpbin_ips(), itertools.repeat(get_aws_ips())))
test_httpbin_ip_in_aws(ip, aws_ip_ranges):
for aws_ip_range in aws_ip_ranges:
assert ipaddress.IPv4Address(ip) not in ipaddress.ip_network(aws_ip_range['ip_prefix']), \
"{0} is in AWS range {1[ip_prefix]} region {1[region]} service {1[service]}".format(ip, aws_ip_range)
Notes:
we add two data fetching functions that return lists that we can zip into tuples for the pytest parametrize decorator
we add markers for the services we're fetching data from
Running it we see that one of the IPs is an AWS IP:
st --ignore aws/
form darwin -- Python 3.6.2, pytest-3.3.2, py-1.5.2, pluggy-0.6.0
data: {'Python': '3.6.2', 'Platform': 'Darwin-15.6.0-x86_64-i386-64bit', 'Packages': {'pytest': '3.3.2', 'py': '1.5.2', 'pluggy': '0.6.0'}, 'Plugins': {'metadata': '1.5.1', 'json': '0.4.0', 'html': '1.16.1'}}
dir: /Users/gguthe/mozilla-services/pytest-services, inifile:
ins: metadata-1.5.1, json-0.4.0, html-1.16.1
ected 3 items
bin/test_httpbin_ip_in_aws.py .F. [100%]
============================================================ FAILURES =================================================================
________________________________________ test_httpbin_ip_in_aws[13.58.0.0-aws_ip_ranges1] _____________________________________________
'13.58.0.0'
ip_ranges = [{'ip_prefix': '13.32.0.0/15', 'region': 'GLOBAL', 'service': 'AMAZON'}, {'ip_prefix': '13.35.0.0/16', 'region': 'GLOB...on': 'us-west-1', 'service': 'AMAZON'}, {'ip_prefix': '13.57.0.0/16', 'region': 'us-west-1', 'service': 'AMAZON'}, ...]
@pytest.mark.httpbin
@pytest.mark.aws_ip_ranges
@pytest.mark.parametrize(
['ip', 'aws_ip_ranges'],
zip(get_httpbin_ips(), itertools.repeat(get_aws_ips())),
# ids=lambda ip: ip
)
def test_httpbin_ip_in_aws(ip, aws_ip_ranges):
for aws_ip_range in aws_ip_ranges:
assert ipaddress.IPv4Address(ip) not in ipaddress.ip_network(aws_ip_range['ip_prefix']), \
"{0} is in AWS range {1[ip_prefix]} region {1[region]} service {1[service]}".format(ip, aws_ip_range)
AssertionError: 13.58.0.0 is in AWS range 13.58.0.0/15 region us-east-2 service AMAZON
assert IPv4Address('13.58.0.0') not in IPv4Network('13.58.0.0/15')
+ where IPv4Address('13.58.0.0') = <class 'ipaddress.IPv4Address'>('13.58.0.0')
+ where <class 'ipaddress.IPv4Address'> = ipaddress.IPv4Address
+ and IPv4Network('13.58.0.0/15') = <function ip_network at 0x107cf66a8>('13.58.0.0/15')
+ where <function ip_network at 0x107cf66a8> = ipaddress.ip_network
bin/test_httpbin_ip_in_aws.py:43: AssertionError
=============================================== 1 failed, 2 passed in 15.69 seconds ===================================================
Note: marking tests as expected failures with @pytest.mark.xfail can hide data fetching errors
To improve this we could:
As we add more tests we can:
<service name>/resources.py files and import them into the test<service name>/client.py and save to the pytest cache