Name: oxidized
Owner: Netflix-Skunkworks
Description: configuration backup software (IOS, JunOS) - silly attempt at rancid
Forked from: ytti/oxidized
Created: 2017-02-10 17:50:26.0
Updated: 2017-09-28 13:20:46.0
Pushed: 2017-11-09 22:12:54.0
Homepage: null
Size: 1140
Language: Ruby
GitHub Committers
User | Most Recent Commit | # Commits |
---|
Other Committers
User | Most Recent Commit | # Commits |
---|
Oxidized is a network device configuration backup tool. It's a RANCID replacement!
Youtube Video: Oxidized TREX 2014 presentation
Vendor
OS model
A10 Networks
Alcatel-Lucent
Wireless
Alvarion
APC
Arista
Arris
Aruba
Brocade
Casa
Check Point
Ciena
Cisco
Citrix
Coriant (former Tellabs)
Cumulus
DataCom
DELL
D-Link
Ericsson/Redback
Extreme Networks
F5
Force10
FortiGate
Fujitsu
Hatteras
HP
Huawei
Infinera
Juniper
Mellanox
Mikrotik
Motorola
MRV
Netonix
Nokia (formerly TiMetra, Alcatel, Alcatel-Lucent)
Opengear
Palo Alto
Quanta
Supermicro
Trango Systems
TPLink
Ubiquiti
Watchguard
Zhone
Zyxel
Install all required packages and gems.
get install ruby ruby-dev libsqlite3-dev libssl-dev pkg-config cmake libssh2-1-dev
install oxidized
install oxidized-script oxidized-web # if you don't install oxidized-web, make sure you remove "rest" from your config
On CentOS 6 / RHEL 6, install Ruby greater than 1.9.3 (for Ruby 2.1.2 installation instructions see “Installing Ruby 2.1.2 using RVM”), then install Oxidized dependencies
install cmake sqlite-devel openssl-devel libssh2-devel
RHEL 7 / CentOS 7 will work out of the box with the following package list:
install cmake sqlite-devel openssl-devel libssh2-devel ruby gcc ruby-devel
Now let's install oxidized via Rubygems:
install oxidized
install oxidized-script oxidized-web
clone https://github.com/ytti/oxidized.git
xidized/
build *.gemspec
install pkg/*.gem
Oxidized configuration is in YAML format. Configuration files are subsequently sourced from `/etc/oxidized/config
then ``
~/.config/oxidized/config`. The hashes will be merged, this might be useful for storing source information in a system wide file and user specific configuration in the home directory (to only include a staff specific username and password). Eg. if many users are using
oxs``
, see Oxidized::Script.
It is recommended practice to run Oxidized using its own username. This username can be added using standard command-line tools:
add oxidized
It is recommended not to run Oxidized as root.
To initialize a default configuration in your home directory `~/.config/oxidized/config
, simply run ``
oxidized` once. If you don't further configure anything from the output and source sections, it'll extend the examples on a subsequent
oxidized``
execution. This is useful to see what options for a specific source or output backend are available.
You can set the env variable OXIDIZED_HOME
to change its home directory.
IZED_HOME=/etc/oxidized
ee -L 1 /etc/oxidized
/oxidized/
config
log-router-ssh
log-router-telnet
pid
router.db
repository.git
Oxidized supports `CSV
, ``
SQLite` and
HTTP``
as source backends. The CSV backend reads nodes from a rancid compatible router.db file. The SQLite backend will fire queries against a database and map certain fields to model items. The HTTP backend will fire queries against a http/https url. Take a look at the Cookbook for more details.
Possible outputs are either `file
or ``
git```. The file backend takes a destination directory as argument and will keep a file per device, with most recent running version of a device. The GIT backend (recommended) will initialize an empty GIT repository in the specified path and create a new commit on every configuration change. Take a look at the Cookbook for more details.
Maps define how to map a model's fields to model model fields. Most of the settings should be self explanatory, log is ignored if use_syslog
(requires Ruby >= 2.0) is set to true
.
First create the directory where the CSV `output
` is going to store device configs and start Oxidized once.
r -p ~/.config/oxidized/configs
ized
Now tell Oxidized where it finds a list of network devices to backup configuration from. You can either use CSV or SQLite as source. To create a CSV source add the following snippet:
Note: If gpg is set to anything other than false it will attempt to decrypt the file contents
ce:
fault: csv
v:
file: ~/.config/oxidized/router.db
delimiter: !ruby/regexp /:/
gpg: false
gpg_password: 'password'
map:
name: 0
model: 1
Now lets create a file based device database (you might want to switch to SQLite later on). Put your routers in `~/.config/oxidized/router.db
` (file format is compatible with rancid). Simply add an item per line:
er01.example.com:ios
ch01.example.com:procurve
er02.example.com:ios
Run `oxidized
` again to take the first backups.
Install Ruby 2.1.2 build dependencies
install curl gcc-c++ patch readline readline-devel zlib zlib-devel
install libyaml-devel libffi-devel openssl-devel make cmake
install bzip2 autoconf automake libtool bison iconv-devel libssh2-devel
Install RVM
-L get.rvm.io | bash -s stable
Setup RVM environment and compile and install Ruby 2.1.2 and set it as default
ce /etc/profile.d/rvm.sh
install 2.1.2
use --default 2.1.2
clone git repo:
clone https://github.com/ytti/oxidized
build container locally:
er build -q -t oxidized/oxidized:latest oxidized/
create config directory in main system:
r /etc/oxidized
run container the first time: _Note: this step in only needed for creating Oxidized's configuration file and can be skipped if you already have it
er run --rm -v /etc/oxidized:/root/.config/oxidized -p 8888:8888/tcp -t oxidized/oxidized:latest oxidized
If the RESTful API and Web Interface are enabled, on the docker host running the container edit /etc/oxidized/config and modify 'rest: 127.0.0.1:8888' by 'rest: 0.0.0.0:8888' this will bind port 8888 to all interfaces then expose port out. (Issue #445)
You can also use docker-compose to launch oxidized container:
cker-compose.yml
cker-compose file example for oxidized that will start along with docker daemon
ized:
start: always
age: oxidized/oxidized:latest
rts:
- 8888:8888/tcp
vironment:
CONFIG_RELOAD_INTERVAL: 600
lumes:
- /etc/oxidized:/root/.config/oxidized
create the /etc/oxidized/router.db
/etc/oxidized/router.db
run container again:
er run -v /etc/oxidized:/root/.config/oxidized -p 8888:8888/tcp -t oxidized/oxidized:latest
ized[1]: Oxidized starting, running as pid 1
ized[1]: Loaded 1 nodes
2.13.4 starting...
n threads: 0, max threads: 16
vironment: development
stening on tcp://0.0.0.0:8888
If you want to have the config automatically reloaded (e.g. when using a http source that changes)
er run -v /etc/oxidized:/root/.config/oxidized -p 8888:8888/tcp -e CONFIG_RELOAD_INTERVAL=3600 -t oxidized/oxidized:latest
In case a model plugin doesn't work correctly (ios, procurve, etc.), you can enable live debugging of SSH/Telnet sessions. Just add a `debug
option containing the value true to the ``
input``` section. The log files will be created depending on the parent directory of the logfile option.
The following example will log an active ssh/telnet session `/home/oxidized/.config/oxidized/log/<IP-Adress>-<PROTOCOL>
. The file will be truncated on each consecutive ssh/telnet session, so you need to put a ``
tailf` or
tail -f``
on that file!
/home/oxidized/.config/oxidized/log
t:
fault: ssh, telnet
bug: true
h:
secure: false
To start privileged mode before pulling the configuration, Oxidized needs to send the enable command. You can globally enable this, by adding the following snippet to the global section of the configuration file.
:
nable: S3cre7
To strip out secrets from configurations before storing them, Oxidized needs the the remove_secrets flag. You can globally enable this by adding the following snippet to the global sections of the configuration file.
:
move_secret: true
Device models can contain substitution filters to remove potentially sensitive data from configs.
As a partial example from ios.rb:
d :secret do |cfg|
cfg.gsub! /^(snmp-server community).*/, '\\1 <configuration removed>'
(...)
cfg
d
The above strips out snmp community strings from your saved configs.
NOTE: Removing secrets reduces the usefulness as a full configuration backup, but it may make sharing configs easier.
Oxidized uses exec channels to make information extraction simpler, but there are some situations where this doesn't work well, e.g. configuring devices. This feature can be turned off by setting the `ssh_no_exec
`
variable.
:
h_no_exec: true
One line per device, colon seperated. If ip
isn't present, a DNS lookup will be done against name
. For large installations, setting ip
will dramatically reduce startup time.
ce:
fault: csv
v:
file: /var/lib/oxidized/router.db
delimiter: !ruby/regexp /:/
map:
name: 0
ip: 1
model: 2
username: 3
password: 4
vars_map:
enable: 5
Oxidized can ssh
through a proxy as well. To do so we just need to set ssh_proxy
variable.
me: 0
del: 1
_map:
able: 2
h_proxy: 3
Oxidized uses the sequel
ruby gem. You can use a variety of databases that aren't explicitly listed. For more information visit https://github.com/jeremyevans/sequel Make sure you have the correct adapter!
values correspond to your fields in the DB such that ip, model, etc are field names in the DB
source: default: sql sql:
adapter: mysql2
database: oxidized
table: nodes
username: root
password: rootpass
map:
name: ip
model: model
username: username
password: password
vars_map:
enable: enable
Source: SQLite
row per device, filtered by hostname.
source: default: sql sql:
adapter: sqlite
database: "/var/lib/oxidized/devices.db"
table: devices
map:
name: fqdn
model: model
username: username
password: password
vars_map:
enable: enable
Source: HTTP
object per device.
Supports basic auth, configure the user and pass you want to use under the http: section.
source: default: http http:
url: https://url/api
scheme: https
delimiter: !ruby/regexp /:/
user: username
pass: password
map:
name: hostname
model: os
username: username
password: password
vars_map:
enable: enable
headers:
X-Auth-Token: 'somerandomstring'
can also pass `secure: false` if you want to disable ssl certificate verification:
source: default: http http:
url: https://url/api
scheme: https
secure: false
Output: File
nt directory needs to be created manually, one file per device, with most recent running config.
output: file:
directory: /var/lib/oxidized/configs
Output: Git
uses the rugged/libgit2 interface. So you should remember that normal Git hooks will not be executed.
a single repositories for all devices:
output: default: git git:
user: Oxidized
email: o@example.com
repo: "/var/lib/oxidized/devices.git"
for groups repositories:
output: default: git git:
user: Oxidized
email: o@example.com
repo: "/var/lib/oxidized/git-repos/default.git"
ized will create a repository for each group in the same directory as the `default.git`. For
ple:
host1:ios:first host2:nxos:second
will generate the following repositories:
$ ls /var/lib/oxidized/git-repos
default.git first.git second.git
ou would like to use groups and a single repository, you can force this with the `single_repo` config.
output: default: git git:
single_repo: true
repo: "/var/lib/oxidized/devices.git"
Output: Http
a config to the specified URL
output: default: http http:
user: admin
password: changeit
url: "http://192.168.162.50:8080/db/coll"
Output types
ou prefer to have different outputs in different files and/or directories, you can easily do this by modifying the corresponding model. To change the behaviour for IOS, you would edit `lib/oxidized/model/ios.rb` (run `gem contents oxidized` to find out the full file path).
example, let's say you want to split out `show version` and `show inventory` into separate files in a directory called `nodiff` which your tools will not send automated diffstats for. You can apply a patch along the lines of
cmd 'show version' do |cfg|
comment cfg.lines.first
cmd 'show version' do |state|
state.type = 'nodiff'
state
cmd 'show inventory' do |cfg|
comment cfg
cmd 'show inventory' do |state|
state.type = 'nodiff'
state
end
cmd 'show running-config' do |cfg|
cfg = cfg.each_line.to_a[3..-1].join
cfg.gsub! /Current configuration : [\n]*\n/, ''
cfg.sub! /^(ntp clock-period).*/, '! \1'
cfg.gsub! /\ tunnel\ mpls\ traffic-eng\ bandwidth[\n]\n(
cmd 'show running-config' do |state|
state = state.each_line.to_a[3..-1].join
state.gsub! /Current configuration : [\n]*\n/, ''
state.sub! /^(ntp clock-period).*/, '! \1'
state.gsub! /\ tunnel\ mpls\ traffic-eng\ bandwidth[\n]\n(
(?:\ [^\n]*\n*)*
tunnel\ mpls\ traffic-eng\ auto-bw)/mx, '\1'
cfg
state = Oxidized::String.new state
state.type = 'nodiff'
state
which will result in the following layout
/$FQDN--show_running_config
ff/$FQDN--show_version
ff/$FQDN--show_inventory
The RESTful API and Web Interface is enabled by configuring the rest:
parameter in the config file. This parameter can optionally contain a relative URI.
sten on http://127.0.0.1:8888/
: 127.0.0.1:8888
sten on http://10.0.0.1:8000/oxidized/
: 10.0.0.1:8000/oxidized
Below is an advanced example configuration. You will be able to (optionally) override options per device. The router.db format used is `hostname:model:username:password:enable_password
`. Hostname and model will be the only required options, all others override the global configuration sections.
name: oxidized
word: S3cr3tx
l: junos
rval: 3600
~/.config/oxidized/log
g: false
ads: 30
out: 20
ies: 3
pt: !ruby/regexp /^([\w.@-]+[#>]\s?)$/
:
able: S3cr3tx
ps: {}
: 127.0.0.1:8888
~/.config/oxidized/oxidized.pid
t:
fault: ssh, telnet
bug: false
h:
secure: false
ut:
fault: git
t:
user: Oxidized
email: oxidized@example.com
repo: "~/.config/oxidized/oxidized.git"
ce:
fault: csv
v:
file: ~/.config/oxidized/router.db
delimiter: !ruby/regexp /:/
map:
name: 0
model: 1
username: 2
password: 3
vars_map:
enable: 4
l_map:
sco: ios
niper: junos
For group specific credentials
ps:
krotik:
username: admin
password: blank
iquiti:
username: ubnt
password: ubnt
and add group mapping
del: 0
me: 1
oup: 2
A node can be moved to head-of-queue via the REST API GET/POST /node/next/[NODE]
.
In the default configuration this node will be processed when the next job worker becomes available, it could take some time if existing backups are in progress. To execute moved jobs immediately a new job can be added:
_adds_job: true
You can define arbitrary number of hooks that subscribe different events. The hook system is modular and different kind of hook types can be enabled.
Following configuration keys need to be defined for all hooks:
events
: which events to subscribe. Needs to be an array. See below for the list of available events.type
: what hook class to use. See below for the list of available hook types.node_success
: triggered when configuration is succesfully pulled from a node and right before storing the configuration.node_fail
: triggered after retries
amount of failed node pulls.post_store
: triggered after node configuration is stored (this is executed only when the configuration has changed).The exec
hook type allows users to run an arbitrary shell command or a binary when triggered.
The command is executed on a separate child process either in synchronous or asynchronous fashion. Non-zero exit values cause errors to be logged. STDOUT and STDERR are currently not collected.
Command is executed with the following environment:
VENT
ODE_NAME
ODE_FROM
ODE_MSG
ODE_GROUP
OB_STATUS
OB_TIME
EPO_COMMITREF
EPO_NAME
Exec hook recognizes following configuration keys:
timeout
: hard timeout for the command execution. SIGTERM will be sent to the child process after the timeout has elapsed. Default: 60async
: influences whether main thread will wait for the command execution. Set this true for long running commands so node pull is not blocked. Default: falsecmd
: command to run.s:
me_for_example_hook1:
type: exec
events: [node_success]
cmd: 'echo "Node success $OX_NODE_NAME" >> /tmp/ox_node_success.log'
me_for_example_hook2:
type: exec
events: [post_store, node_fail]
cmd: 'echo "Doing long running stuff for $OX_NODE_NAME" >> /tmp/ox_node_stuff.log; sleep 60'
async: true
timeout: 120
This hook configures the repository remote
and push the code when the specified event is triggerd. If the username
and password
are not provided, the Rugged::Credentials::SshKeyFromAgent
will be used.
githubrepo
hook recognizes following configuration keys:
remote_repo
: the remote repository to be pushed to.username
: username for repository auth.password
: password for repository auth.publickey
: publickey for repository auth.privatekey
: privatekey for repository auth.When using groups repositories, each group must have its own remote
in the remote_repo
config.
s:
sh_to_remote:
remote_repo:
routers: git@git.intranet:oxidized/routers.git
switches: git@git.intranet:oxidized/switches.git
firewalls: git@git.intranet:oxidized/firewalls.git
s:
sh_to_remote:
type: githubrepo
events: [post_store]
remote_repo: git@git.intranet:oxidized/test.git
username: user
password: pass
The awssns
hook publishes messages to AWS SNS topics. This allows you to notify other systems of device configuration changes, for example a config orchestration pipeline. Multiple services can subscribe to the same AWS topic.
Fields sent in the message:
event
: Event type (e.g. node_success
)group
: Group namemodel
: Model name (e.g. eos
)node
: Device hostnameConfiguration example:
s:
ok_script:
type: awssns
events: [node_fail,node_success,post_store]
region: us-east-1
topic_arn: arn:aws:sns:us-east-1:1234567:oxidized-test-backup_events
AWS SNS hook requires the following configuration keys:
region
: AWS Region nametopic_arn
: ASN Topic referenceYour AWS credentials should be stored in ~/.aws/credentials
.
The slackdiff
hook posts colorized config diffs to a Slack channel of your choice. It only triggers for post_store
events.
You will need to manually install the slack-api
gem on your system:
install slack-api
Configuration example:
s:
ack:
type: slackdiff
events: [post_store]
token: SLACK_BOT_TOKEN
channel: "#network-changes"
Note the channel name must be in quotes.
The following objects exist in Oxidized.
Copyright 2013-2015 Saku Ytti saku@ytti.fi
2013-2015 Samer Abdel-Hafez <sam@arahant.net>
Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.