CD2H gitForager

reddit/kube2iam

Name: kube2iam

Owner: Reddit

Description: kube2iam provides different AWS IAM roles for pods running on Kubernetes

Forked from: jtblin/kube2iam

Created: 2017-08-31 02:58:51.0

Updated: 2017-08-31 02:58:52.0

Pushed: 2017-12-21 18:40:29.0

Homepage:

Size: 139

Language: Go

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

kube2iam

Provide IAM credentials to containers running inside a kubernetes cluster based on annotations.

Context

Traditionally in AWS, service level isolation is done using IAM roles. IAM roles are attributed through instance profiles and are accessible by services through the transparent usage by the aws-sdk of the ec2 metadata API. When using the aws-sdk, a call is made to the ec2 metadata API which provides temporary credentials that are then used to make calls to the AWS service.

Problem statement

The problem is that in a multi-tenanted containers based world, multiple containers will be sharing the underlying nodes. Given containers will share the same underlying nodes, providing access to AWS resources via IAM roles would mean that one needs to create an IAM role which is a union of all IAM roles. This is not acceptable from a security perspective.

Solution

The solution is to redirect the traffic that is going to the ec2 metadata API for docker containers to a container running on each instance, make a call to the AWS API to retrieve temporary credentials and return these to the caller. Other calls will be proxied to the ec2 metadata API. This container will need to run with host networking enabled so that it can call the ec2 metadata API itself.

Usage

IAM roles

It is necessary to create an IAM role which can assume other roles and assign it to each kubernetes worker.


ersion": "2012-10-17",
tatement": [
{
  "Action": [
    "sts:AssumeRole"
  ],
  "Effect": "Allow",
  "Resource": "*"
}

The roles that will be assumed must have a Trust Relationship which allows them to be assumed by the kubernetes worker role. See this StackOverflow post for more details.


ersion": "2012-10-17",
tatement": [
{
  "Sid": "",
  "Effect": "Allow",
  "Principal": {
    "Service": "ec2.amazonaws.com"
  },
  "Action": "sts:AssumeRole"
},
{
  "Sid": "",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::123456789012:role/kubernetes-worker-role"
  },
  "Action": "sts:AssumeRole"
}

kube2iam daemonset

Run the kube2iam container as a daemonset (so that it runs on each worker) with hostNetwork: true. The kube2iam daemon and iptables rule (see below) need to run before all other pods that would require access to AWS resources.

ersion: extensions/v1beta1
: DaemonSet
data:
me: kube2iam
bels:
app: kube2iam
:
mplate:
metadata:
  labels:
    name: kube2iam
spec:
  hostNetwork: true
  containers:
    - image: jtblin/kube2iam:latest
      name: kube2iam
      args:
        - "--base-role-arn=arn:aws:iam::123456789012:role/"
      ports:
        - containerPort: 8181
          hostPort: 8181
          name: http

iptables

To prevent containers from directly accessing the ec2 metadata API and gaining unwanted access to AWS resources, the traffic to 169.254.169.254 must be proxied for docker containers.

bles \
append PREROUTING \
protocol tcp \
destination 169.254.169.254 \
dport 80 \
in-interface docker0 \
jump DNAT \
table nat \
to-destination `curl 169.254.169.254/latest/meta-data/local-ipv4`:8181

This rule can be added automatically by setting --iptables=true, setting the HOST_IP environment variable, and running the container in a privileged security context.

Note that the interface --in-interface above or using the --host-interface cli flag may be different than docker0 depending on which virtual network you use e.g.

for Calico, use cali+ (the interface name is something like cali1234567890
for kops (on kubenet), use cbr0
for CNI, use cni0
for weave use weave
for flannel use cni0

ersion: extensions/v1beta1
: DaemonSet
data:
me: kube2iam
bels:
app: kube2iam
:
mplate:
metadata:
  labels:
    name: kube2iam
spec:
  hostNetwork: true
  containers:
    - image: jtblin/kube2iam:latest
      name: kube2iam
      args:
        - "--base-role-arn=arn:aws:iam::123456789012:role/"
        - "--iptables=true"
        - "--host-ip=$(HOST_IP)"
      env:
        - name: HOST_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
      ports:
        - containerPort: 8181
          hostPort: 8181
          name: http
      securityContext:
        privileged: true

kubernetes annotation

Add an iam.amazonaws.com/role annotation to your pods with the role that you want to assume for this pod.

ersion: v1
: Pod
data:
me: aws-cli
bels:
name: aws-cli
notations:
iam.amazonaws.com/role: role-arn
:
ntainers:
image: fstab/aws-cli
command:
  - "/home/aws/aws/env/bin/aws"
  - "s3"
  - "ls"
  - "some-bucket"
name: aws-cli

You can use --default-role to set a fallback role to use when annotation is not set.

ReplicaSet, CronJob, Deployment, etc.

When creating higher-level abstractions than pods, you need to pass the annotation in the pod template of the
resource spec.

Example for a Deployment:

ersion: extensions/v1beta1
: Deployment
data:
me: nginx-deployment
:
plicas: 3
mplate:
metadata:
  annotations:
    iam.amazonaws.com/role: role-arn
  labels:
    app: nginx
spec:
  containers:
  - name: nginx
    image: nginx:1.9.1
    ports:
    - containerPort: 80

Example for a CronJob:

ersion: batch/v2alpha1
: CronJob
data:
me: my-cronjob
:
hedule: "00 11 * * 2"
ncurrencyPolicy: Forbid
artingDeadlineSeconds: 3600
bTemplate:
spec:
  template:
    metadata:
      annotations:
        iam.amazonaws.com/role: role-arn
    spec:
      restartPolicy: OnFailure
      containers:
      - name: job
        image: my-image

Namespace Restrictions

By using the flag –namespace-restrictions you can enable a mode in which the roles that pods can assume is restricted by an annotation on the pod's namespace. This annotation should be in the form of a json array.

To allow the aws-cli pod specified above to run in the default namespace your namespace would look like the following.

ersion: v1
: Namespace
data:
notations:
iam.amazonaws.com/allowed-roles: |
  ["role-arn"]
me: default

Note: You can also use glob-based matching for namespace restrictions, which works nicely with the path-based namespacing supported for AWS IAM roles.

Example: to allow all roles prefixed with my-custom-path/ to be assuemd by pods in the default namespace, the default namespace would be annotated as follows:

ersion: v1
: Namespace
data:
notations:
iam.amazonaws.com/allowed-roles: |
  ["my-custom-path/*"]
me: default

Debug

By using the –debug flag you can enable some extra features making debugging easier:

/debug/store endpoint enabled to dump knowledge of namespaces and role association.

Base ARN auto discovery

By using the --auto-discover-base-arn flag, kube2iam will auto discover the base arn via the ec2 metadata service.

Using ec2 instance role as default role

By using the --auto-discover-default-role flag, kube2iam will auto discover the base arn and the iam role attached to the instance and use it as the fallback role to use when annotation is not set.

Options

By default, kube2iam will use the in-cluster method to connect to the kubernetes master, and use the iam.amazonaws.com/role annotation to retrieve the role for the container. Either set the base-role-arn option to apply to all roles and only pass the role name in the iam.amazonaws.com/role annotation, otherwise pass the full role ARN in the annotation.

be2iam --help
e of ./build/bin/darwin/kube2iam:
  --api-server string                   Endpoint for the api server
  --api-token string                    Token to authenticate with the api server
  --app-port string                     Http port (default "8181")
  --auto-discover-base-arn              Queries EC2 Metadata to determine the base ARN
  --auto-discover-default-role          Queries EC2 Metadata to determine the default Iam Role and base ARN, cannot be used with --default-role, overwrites any previous setting for --base-role-arn
  --backoff-max-elapsed-time duration   Max elapsed time for backoff when querying for role. (default 2s)
  --backoff-max-interval duration       Max interval for backoff when querying for role. (default 1s)
  --base-role-arn string                Base role ARN
  --debug                               Enable debug features
  --default-role string                 Fallback role to use when annotation is not set
  --host-interface string               Host interface for proxying AWS metadata (default "docker0")
  --host-ip string                      IP address of host
  --iam-role-key string                 Pod annotation key used to retrieve the IAM role (default "iam.amazonaws.com/role")
  --insecure                            Kubernetes server should be accessed without verifying the TLS. Testing only
  --iptables                            Add iptables rule (also requires --host-ip)
  --log-level string                    Log level (default "info")
  --metadata-addr string                Address for the ec2 metadata (default "169.254.169.254")
  --namespace-key string                Namespace annotation key used to retrieve the IAM roles allowed (value in annotation should be json array) (default "iam.amazonaws.com/allowed-roles")
  --namespace-restrictions              Enable namespace restrictions
  --verbose                             Verbose
  --version                             Print the version and exits

Development loop

Use minikube to run cluster locally
Build and push dev image to docker hub: make docker-dev DOCKER_REPO=<your docker hub username>
Update deployment.yaml as needed
Deploy to local kubernetes cluster: kubectl create -f deployment.yaml or kubectl delete -f deployment.yaml && kubectl create -f deployment.yaml
Expose as service: kubectl expose deployment kube2iam --type=NodePort
Retrieve the services url: minikube service kube2iam --url
Test your changes e.g. curl -is $(minikube service kube2iam --url)/healthz

Author

Jerome Touffe-Blin, @jtblin, About me

License

This work is supported by the National Institutes of Health's National Center for Advancing Translational Sciences, Grant Number U24TR002306. This work is solely the responsibility of the creators and does not necessarily represent the official views of the National Institutes of Health.