namshi/s3-sidecar

Name: s3-sidecar

Owner: Namshi

Description: A container that works as a volume sidecar within a k8s pod.

Created: 2017-03-08 07:47:52.0

Updated: 2017-03-09 10:28:15.0

Pushed: 2017-03-27 07:43:02.0

Homepage: null

Size: 9

Language: Shell

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

s3-sidecar

A container downloads a compressed file from S3 and extract the files into a local directory /data which can be a shared volume with other containers

Installing and Running

You need to set the env variables and run it as contaner, and here are the required variables:

AWS_ACCESS_KEY_ID: 'access key'
AWS_SECRET_ACCESS_KEY: 'access secret'
AWS_DEFAULT_REGION: 'aws region'
AWS_BUCKET: 'bucket name'

Build the container docker-compose build

Then you can run it docker-compose up

How it works

Once you run the container, it will start a script which will run in a loop every 3 seconds.

It will get the new file name we need to download from CONFIG_PATH as the last part after / and then it will check that the content of this file /data/{file name} is different from the file name we are downloading.

If they are different it will download the file from S3 and extract it in teh /data/ directory.

It will create a file /data/{file name} contains the name of the extracted file,

then it creates a lock file /data/update.lock, this file can be used from other containers to clear their cache.

If the file name in /data/{file name} is the same as the file name we are downloading, it won't do anything.