spotify/talos

Name: talos

Owner: Spotify

Description: Hiera secrets distribution over HTTP

Created: 2015-12-16 12:12:40.0

Updated: 2018-05-18 18:09:20.0

Pushed: 2018-05-18 18:09:21.0

Homepage:

Size: 18

Language: Ruby

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

Talos

Talos is a rack application which serves Hiera yaml files over HTTP. It authorizes clients based on the SSL certificates issued by the Puppet CA and returns only the files in the Hiera scope.

Talos is used to store and distribute secrets via Hiera to the masterless puppet clients.

How it works

Talos listens for incoming HTTP requests and returns compressed hiera tree based on the client's SSL certificate.

To determine the list of files to send, Talos matches the certificate common name against a list of regular expressions.

Fetching the tree

It's possible to run a cron task or create a wrapper around the puppet agent. Here's an example of the client-side code which uses local puppet SSL key to authenticate:

ire 'puppet'
et[:confdir] = '/etc/puppetlabs/puppet/'
r/bin/curl -s --fail -X GET -k https://talos.internal}/ \
cert #{Puppet[:hostcert]} --key #{Puppet[:hostprivkey]} \
data-urlencode pool=#{Facter.value(:pool)} > /etc/talos/tree.tar.gz`
n/tar xzf /etc/talos/tree.tar.gz -C /etc/talos/hiera_secrets`

In this example the client also passes pool variable which will be included in the Hiera scope if unsafe_scopes option is enabled.

The received copy of the tree could be included in the local hiera config and used in the normal puppet runs.

Configuration

Talos configuration is stored in /etc/talos/talos.yaml:

es:
lon-puppet-a1: site = lon, role = puppet, pool = a
match: '(?<site>[[:alpha:]]+)-(?<role>[a-z0-9]+)-(?<pool>[[:alpha:]]+)'
facts:
  environment: production
match: 'cloud\.example\.com'
facts:
  environment: testing

fe_scopes: true
 true

When receiving a request, Talos iterates over scopes list and matches the client certificate against the match blocks. If the match is successful, Talos does 2 things:

1. Adds all the named captures from the regexp to the Hiera scope
2. Adds all the facts to the Hiera scope

Talos will iterate over all the regexps updating the Hiera scope, meaning that the later matches will override the existing scope on collision.

If unsafe_scopes option is enabled, Talos will also add all the parameters passed by the client to the Hiera scope.

The ssl option defaults to enabled. When disabled, the fqdn query parameter is used to determine scopes rather than the client certificate.

Hiera

You need to provide /etc/talos/hiera.yaml file to configure Hiera backend on the Talos server:

kends:
yaml
rarchy:
'hiera-secrets/fqdn/%{fqdn}'
'hiera-secrets/role/%{role}/%{pod}/%{pool}'
'hiera-secrets/role/%{role}/%{pod}'
'hiera-secrets/role/%{role}'
'hiera-secrets/pod/%{pod}'
'hiera-secrets/common'
l:
atadir: '/etc/puppet'
ge_behavior: :deeper

Talos will use the datadir option to search for YAML files and it will return only the files that match the Hiera scope of the clients.

Installing

You can use spotify/talos puppet module to install Talos.

Manual installation

First, install talos using rubygems:

$ gem install talos

Create a separate user and Document Root for the Rack application:

$ useradd talos --system --create-home --home-dir /var/lib/talos
$ mkdir -p /var/lib/talos/public /var/lib/talos/tmp /etc/talos
$ chown -R talos:talos /var/lib/talos/ /etc/talos

Then copy config.ru to /var/lib/talos/ directory.

You also need to copy and adjust hiera.yaml and talos.yaml configs in /etc/talos directory.

Hiera repository

You need to have a copy of the hiera-secrets repository available on the talos server. Make sure it's located at the datadir specified in /etc/talos/hiera.yaml

Apache

You can run Talos using Passenger or any other application server. Make sure you use Puppet SSL keys to validate the client certificates and to forward SSL_CLIENT_S_DN_CN header:

tualHost *:443>
cumentRoot "/var/lib/talos/public"

irectory "/var/lib/talos/public">
Require all granted
Directory>

LEngine on
LCertificateFile "/etc/puppetlabs/puppet/ssl/certs/talos.internal.pem"
LCertificateKeyFile "/etc/puppetlabs/puppet/ssl/private_keys/talos.internal.pem"
LCertificateChainFile "/etc/puppetlabs/puppet/ssl/certs/ca.pem"
LCACertificatePath "/etc/ssl/certs"
LCACertificateFile "/etc/puppetlabs/puppet/ssl/certs/ca.pem"
LCARevocationFile "/etc/puppetlabs/puppet/ssl/crl.pem"
LVerifyClient require
LOptions +StdEnvVars +FakeBasicAuth
questHeader set SSL_CLIENT_S_DN_CN "%{SSL_CLIENT_S_DN_CN}s"
rtualHost>

Contributing

1. Fork the project on github
2. Create your feature branch
3. Open a Pull Request

This project adheres to the [Open Code of Conduct][code-of-conduct]. By participating, you are expected to honor this code.

[code-of-conduct]: https://github.com/spotify/code-of-conduct/blob/master/code-of-conduct.md

License

right 2013-2016 Spotify AB

nsed under the Apache License, Version 2.0 (the "License");
may not use this file except in compliance with the License.
may obtain a copy of the License at

ttp://www.apache.org/licenses/LICENSE-2.0

ss required by applicable law or agreed to in writing, software
ributed under the License is distributed on an "AS IS" BASIS,
OUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
the License for the specific language governing permissions and
tations under the License.