CD2H gitForager

hybris/spruce

Name: spruce

Owner: hybris GmbH

Description: A BOSH template merge tool

Forked from: geofffranks/spruce

Created: 2016-10-11 16:21:38.0

Updated: 2016-10-11 16:21:40.0

Pushed: 2016-10-06 08:22:07.0

Homepage: null

Size: 2170

Language: Go

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

      *          .---. ,---.  ,---.  .-. .-.  ,--,  ,---.         *
     /.\        ( .-._)| .-.\ | .-.\ | | | |.' .')  | .-'        /.\
    /..'\      (_) \   | |-' )| `-'/ | | | ||  |(_) | `-.       /..'\
    /'.'\      _  \ \  | |--' |   (  | | | |\  \    | .-'       /'.'\
   /.''.'\    ( `-'  ) | |    | |\ \ | `-')| \  `-. |  `--.    /.''.'\
   /.'.'.\     `----'  /(     |_| \)\`---(_)  \____\/( __.'    /.'.'.\
""/'.''.'.\""'"'""""""(__)""""""""(__)"""""""""""""(__)""'""""/'.''.'.\""'"'"
  ^^^[_]^^^                                                   ^^^[_]^^^

Questions? Pop in our slack channel!

Introducing Spruce

spruce is a domain-specific YAML merging tool, for generating BOSH manifests.

It was written with the goal of being the most intuitive solution for merging BOSH templates. As such, it pulls in a few semantics that may seem familiar to those used to merging with the other merging tool, but there are a few key differences.

Installation

Spruce is now available via Homebrew, just brew tap starkandwayne/cf; brew install spruce

Alternatively, you can download a prebuilt binaries for 64-bit Linux, or Mac OS X, or you can install via go get (provided you have installed go):

et github.com/geofffranks/spruce/...

Merging Rules

Merging in spruce is designed to be pretty intuitive. Files to merge are listed in-order on the command line. The first file serves as the base to the file structure, and subsequent files are merged on top, adding when keys are new, replacing when keys exist. This differs slightly in mentality from spiff, but hopefully the results are more predictable.

A word on 'meta'

meta was a convention used quite often in templates merged with spiff. This convention is not necessary with spruce. If you want to merge two hashes together, simply include the new keys in the file merged on top of the original.

What about arrays?

Arrays can be modified in multiple ways: prepending data, appending data, inserting data, merging data onto the existing data, or completely replacing data.

To append data to an existing array, ensure that the first element in the new array is
(( append ))
To prepend the data to an existing array, ensure that the first element in the new array is
(( prepend ))
To insert new elements either after or before a specific position of an existing array, you can use insert with a hint to the respective insertion point in the target list
```
:
```
name: consul instances: 1
name: nats instances: 2
name: ccdb instances: 2
name: uaadb instances: 2
name: dea instances: 8
name: api instances: 2
jobs:
(( insert after “dea” ))
name: dea_v2 instances: 2
```
br>
```
(( insert after “” ))

: new-kid-on-the-block

 The first `insert` is using `name` as the default key to determine the target position in the target array.
second is used to customize which key to use. In any case, instead of `after`, you can also use `before`. This will
end the entries (relative to the specified insertion point).

Similar to the insert operation, you can also use a (( delete ... )) operation multiple times in a list. The delete will remove a map from the list
```
:
```
(( delete “dea” ))
(( delete “api” ))
```
br>
```

(( delete “” ))

 The array modification operations `(( append ))`, `(( prepend ))`, `(( delete ... ))`, and `(( insert ... ))` can be
 multiple times in one list. Entries that follow `(( append ))`, `(( prepend ))`, and `(( insert ... ))` belong to the
 operation. This however does not apply to `(( delete ... ))`, which always stands alone.

To replace the first array with the second, ensure that the first element in the new array is
(( replace ))
To merge two arrays by way of their index, just make the first element
(( inline ))
To merge two arrays of maps together (using a specific key for identifying like objects), ensure that the first element in the new array is either
(( merge ))
```
 or <br>
```

(( merge on ))

 The first merges using `name` as the key to determine
 objects in the array elements. The second is used to customize which key to use.

If you don't specify a specific merge strategy, the array will be merged automatically; using keys if they exist (i.e. (( merge )), and array indices otherwise ((( inline ))).

Do you want to learn about array modifications in more detail? See modifying arrays for examples and explanations.

Cleaning Up After Yourself

To prune a map key from the final output, you can either use the --prune flag:

ce merge --prune key.1.to.prune --prune key.2.to.prune file1.yml file2.yml

or you can use the (( prune )) operator:

to_prune: (( prune ))

Referencing Other Data

Need to reference existing data in your datastructure? No problem! spruce will wait until all the data is merged together before dereferencing anything, but to handle this, you can use the (( grab <thing> )) syntax:

:
lor: blue


lor: (( grab data.color ))

You can even reference multiple values at once, getting back an array of their data, for things like getting all IPs of multi-AZ jobs in a BOSH manifest, just do it like so:

rab jobs.myJob_z1.networks.myNet1.static_ips jobs.myJob_z2.networks.myNet2.static_ips ))

You can also provide alternatives to your grab operation, by using the || (or) operator:

      (( grab site.key || nil ))
in:   (( grab global.domain || "example.com" ))
ocol: (( grab site.protocol || global.protocol || "http" ))

In these examples, if the referenced key does not exist, the next reference is attempted, or the literal value (nil, numbers or strings) is used. Spruce recognizes the following keywords and uses the appropriate literal value:

nil, null and ~ map to the YAML null value
true is the YAML boolean value for truth
false is the YAML boolean value for non-truth

Other types of literals include double-quoted strings (with embedded double quotes escaped with a single backslash - \), integer literals (a string of digits) and floating point literals (a string of digits, a period, and another string of digits). Scientific notation is not currently supported.

Accessing the Environment

Want to pull in secret credentials from your environment? No problem!

ets:
s:
access_key: (( grab $AWS_ACCESS_KEY ))
secret_key: (( grab $AWS_SECRET_KEY ))

spruce will try to pull the named environment variables value from the environment, and fail if the value is not set, or is empty. You can use the || syntax to provide defaults, � la:

:
vironment: (( grab $ENV_NAME || "default-env" ))

Hmm.. How about auto-calculating static IPs for a BOSH manifest?

spruce supports that too! Just use the same (( static_ips(x, y, z) )) syntax that you're used to with spiff, to specify the offsets in the static IP range for a job's network.

Behind the scenes, there are a couple behavior improvements upon spiff. First, since all the merging is done first, then post-processing, there's no need to worry about getting the instances + networks defined before (( static_ips() )) is merged in. Second, the error messaging output should be a lot better to aid in tracking down why static_ips() calls fail.

Check out the static_ips() example

But I Want To Make Strings!!

Yeah, spruce can do that!

 production
ter:
me: mjolnir
t: (( concat cluster.name "//" env ))

Which will give you an ident: key of “mjolnir/production”

But what if I have a list of strings that I want as a single line? Like a users list, authorities, or similar. Do I have to concat that piece by piece? No, you can use join to concatenate a list into one entry.

:
thorities:
password.write
clients.write
clients.read
scim.write

erties:
a:
clients:
  admin:
    authorities: (( join "," meta.authorities ))

This will give you a concatenated list for authorities:

erties:
a:
clients:
  admin:
    authorities: password.write,clients.write,clients.read,scim.write

How About Some Examples?

Basic Example

Here's a pretty broad example, that should cover all the functionality of spruce, to be used as a reference.

If I start with this data:

amples/basic/main.yml

ig_key: This is a string attached to a key
mber: 50
ray1:
first element
second element
third element
p:
key1: v1
key2: v2
key3:
  subkey1: vv1
  subkey2: vv2
  subkey3:
  - nested element 1
  - nested element 2

 430.0
 this starts as a string
ray2:
1
2
3
4
line_array_merge:
will be overwritten
this: will
be: merged

And want to merge in this:

amples/basic/merge.yml

w_key: this is added
ig_key: this is replaced
p:
key4: added key
key1: replaced key
key2: ~
key3:
  subkey3:
  - (( append ))
  - nested element 3
ray1:
(( prepend ))
prepend this
ray2:
over
ridden
array
 You can change types too

even: drastically
to:   from scalars to maps/lists
line_array_merge:
(( inline ))
this has been overwritten
be: overwritten
merging: success!
rtop: you can add new top level keys too

I would use spruce like this:

ruce merge main.yml merge.yml
rtop: you can add new top level keys too

 You can change types too

even: drastically
to: from scalars to maps/lists
ray1:
prepend this
first element
second element
third element
ray2:
over
ridden
array
4
line_array_merge:
this has been overwritten
be: overwritten
merging: success!
this: will
p:
key1: replaced key
key2: null
key3:
  subkey1: vv1
  subkey2: vv2
  subkey3:
  - nested element 1
  - nested element 2
  - nested element 3
key4: added key
w_key: this is added
mber: 50
ig_key: this is replaced

Map Replacement

One of spiff's quirks was that it quite easily allowed you to completely replace an entire map, with new data (rather than merging by default). That result is still possible with spruce, but it takes a little bit more work, since the primary use case is to merge two maps together:

We start with this yaml:

amples/map-replacement/original.yml
uched:
p: stays
e: same
to_replace:
s: upstream
ta: that
: do
t: want

Next, create a YAML file to clear out the map:

amples/map-replacement/delete.yml
to_replace: ~

Now, create a YAML file to insert the data you want in the end:

amples/map-replacement/insert.yml
to_replace:
: special
ta: here

And finally, merge it all together:

ruce merge original.yml delete.yml insert.yml
to_replace:
: special
ta: here
uched:
p: stays
e: same

Key Removal

How about deleting keys outright? Use the –prune flag to the merge command:

amples/key-removal/original.yml
teme:
ing:
foo: 1
bar: 2

ml
amples/key-removal/things.yml
gs:
me: first-thing
o: (( grab deleteme.thing.foo ))
me: second-thing
r: (( grab deleteme.thing.bar ))


ruce merge --prune deleteme original.yml things.yml

The deleteme key is only useful for holding a temporary value, so we'd really rather not see it in the final output. --prune drops it.

Lists of Maps

Let's say you have a list of maps that you would like to merge into another list of maps, while preserving as much data as possible.

Given this original.yml:

amples/list-of-maps/original.yml
:
me: concatenator_z1
stances: 5
source_pool: small
operties:
spruce: is cool
me: oldjob_z1
stances: 4
source_pool: small
operties:
this: will show up in the end

And this new.yml:

amples/list-of-maps/new.yml
:
me: newjob_z1
stances: 3
source_pool: small
operties:
this: is a job defined solely in new.yml
me: concatenator_z1
operties:
this: is a new property added to an existing job

You would get this when merged:

ruce merge original.yml new.yml
:
stances: 5
me: concatenator_z1
operties:
spruce: is cool
this: is a new property added to an existing job
source_pool: small
stances: 4
me: oldjob_z1
operties:
this: will show up in the end
source_pool: small
stances: 3
me: newjob_z1
operties:
this: is a job defined solely in new.yml
source_pool: small

Pretty sweet, huh?

Static IPs

Lets define our jobs.yml:

amples/static-ips/jobs.yml
:
me: staticIP_z1
stances: 3
tworks:
name: net1
static_ips: (( static_ips(0, 2, 4) ))
me: api_z1
stances: 3
tworks:
name: net1
static_ips: (( static_ips(1, 3, 5) ))

Next, we'll define our properties.yml:

amples/static-ips/properties.yml
erties:
aticIP_servers: (( grab jobs.staticIP_z1.networks.net1.static_ips ))
i_servers: (( grab jobs.api_z1.networks.net1.static_ips ))

And lastly, define our networks.yml:

amples/static-ips/networks.yml
orks:
me: net1
bnets:
cloud_properties: random
static:
- 192.168.0.2 - 192.168.0.10

Merge it all together, and see what we get:

ruce merge jobs.yml properties.yml networks.yml
:
stances: 3
me: staticIP_z1
tworks:
name: net1
static_ips:
- 192.168.0.2
- 192.168.0.4
- 192.168.0.6
stances: 3
me: api_z1
tworks:
name: net1
static_ips:
- 192.168.0.3
- 192.168.0.5
- 192.168.0.7
orks:
me: net1
bnets:
cloud_properties: random
static:
- 192.168.0.2 - 192.168.0.10
erties:
i_servers:
192.168.0.3
192.168.0.5
192.168.0.7
aticIP_servers:
192.168.0.2
192.168.0.4
192.168.0.6

Static IPs with Availability Zones

Lets define our jobs.yml:

amples/availability-zones/jobs.yml
ance_groups:
me: staticIP
stances: 3
s: [z1,z2]
tworks:
name: net1
static_ips: (( static_ips(0, "z2:2", "z1:3") ))
me: api
stances: 3
s: [z1]
tworks:
name: net1
static_ips: (( static_ips(1, "z1:4", 5) ))
me: web
stances: 3
tworks:
name: net1
static_ips: (( static_ips(9, 10, 11) ))

Next, we'll define our properties.yml:

amples/availability-zones/properties.yml
erties:
aticIP_servers: (( grab instance_groups.staticIP.networks.net1.static_ips ))
i_servers: (( grab instance_groups.api.networks.net1.static_ips ))
b_servers: (( grab instance_groups.web.networks.net1.static_ips ))

And lastly, define our networks.yml:

amples/availability-zones/networks.yml
orks:
me: net1
bnets:
cloud_properties: random
az: z1
static:
- 192.168.0.1 - 192.168.0.10
cloud_properties: random
az: z2
static:
- 192.168.2.1 - 192.168.2.10

Merge it all together, and see what we get:

ruce merge jobs.yml properties.yml networks.yml
ance_groups:
me: staticIP
stances: 3
s: [z1,z2]
tworks:
name: net1
static_ips:
- 192.168.0.1
- 192.168.2.3
- 192.168.0.4
me: api
stances: 3
s: [z1]
tworks:
name: net1
static_ips:
- 192.168.0.2
- 192.168.0.5
- 192.168.0.6
me: web
stances: 3
tworks:
name: net1
static_ips:
- 192.168.0.10
- 192.168.2.1
- 192.168.2.2
orks:
me: net1
bnets:
cloud_properties: random
az: z1
static:
- 192.168.0.1 - 192.168.0.10
cloud_properties: random
az: z2
static:
- 192.168.2.1 - 192.168.2.10
erties:
i_servers:
192.168.0.2
192.168.0.5
192.168.0.6
aticIP_servers:
192.168.0.1
192.168.2.3
192.168.0.4
b_servers:
192.168.0.10
192.168.2.1
192.168.2.2

Injecting Subtrees

One of the great things about YAML is the oft-overlooked << inject operator, which lets you start with a copy of another part of the YAML tree and override keys, like this:

amples/inject/all-in-one.yml
:
mplate: &template
color: blue
size: small

n:
: *template
lor: green

Here, $.green.size will be small, but $.green.color stays as green:

ruce merge --prune meta all-in-one.yml
n:
lor: green
ze: small

That works great if you are in the same file, but what if you want to inject data from a different file into your current map and then override some things?

That's where (( inject ... )) really shines.

amples/inject/templates.yml
:
mplate:
color: blue
size: small

ml
amples/inject/green.yml
n:
ot: (( inject meta.template ))
lor: green

ml
ruce merge --prune meta templates.yml green.yml
n:
lor: green
ze: small

Note: The key used for the (( inject ... )) call (in this case, woot) is removed from the final tree as part of the injection operator.

File

Sometimes you need to include large blocks of text in your YAML, such as the body of a configuration file, or a script block. However, the indentation can cause issues when that block needs to be edited later, and there's no easy way to use tools to validate the block.

Using the (( file ... )) operator allows you to keep the block in its natural state to allow for easy viewing, editing and processing, but then add it to YAML file as needed. It supports specifying the file either by a string literal or a reference.

The relative path to the file is based on where spruce is run from. Alternatively, you can set the SPRUCE_FILE_BASE_PATH environment variable to the desired root that your YAML file uses as the reference to the relative file paths specified. You can also specify an absolute path in the YAML

RUCE_FILE_BASE_PATH=$HOME/myproj/configs

# Source file
er:
inx:
name: nginx.conf
config_file: (( file server.nginx.name ))
proxy:
name: haproxy.cfg
config_file: (( file "/haproxy/haproxy.cfg" ))

The server.nginx.config_file will contain the contents of $HOME/myproj/configs/nginx.conf, while the server.haproxy.config_file will contain the contents of /haproxy/haproxy.cfg

Parameters

Sometimes, you may want to start with a good starting-point template, but require other YAML files to provide certain values. Parameters to the rescue!

amples/params/global.yml
s:
all: 4096
dium: 8192
rge:  102400
tworks: (( param "please define the networks" ))
:
- ubuntu
- centos
- fedora

And then combine that with these local definitions:

amples/params/local.yml
s:
dium: 16384
tworks:
- name: public
  range: 10.40.0.0/24
- name: inside
  range: 10.60.0.0/16

This works, but if local.yml forgot to specify the top-level networks key, or an error should be emitted.

Author

Written By Geoff Franks and James Hunt, inspired by spiff

Thanks to Long Nguyen for breaking it repeatedly in the interest of improvement and quality assurance.

License

Licensed under the MIT License

This work is supported by the National Institutes of Health's National Center for Advancing Translational Sciences, Grant Number U24TR002306. This work is solely the responsibility of the creators and does not necessarily represent the official views of the National Institutes of Health.