reddit/puppet-fluentd

Name: puppet-fluentd

Owner: Reddit

Description: Generic Puppet module to manage Fluentd

Created: 2018-05-08 22:33:05.0

Updated: 2018-05-14 22:12:27.0

Pushed: 2018-05-14 22:12:25.0

Homepage: https://forge.puppetlabs.com/wywygmbh/fluentd

Size: 117

Language: Ruby

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

pilchkinstein-gtv/fluentd Puppet Module

Table of Contents

1. Description
2. Setup - The basics of getting started with fluentd
3. Usage
4. Configuration
   • Parameters
   • Source
   • Filter
   • Match
   • Match Store
5. Plugin Installation
6. Requirements
7. Limitations - OS compatibility, etc.
8. Development - Guide for contributing to the module
9. License

Description

The fluentd module sets up fluentd (td-agent) and manages configuration files.

This module only supports fluentd from version 2.

Setup

This will install the latest version of fluentd

ude '::fluentd'

Usage

By default this module doesn't configure any sources, matches or filters. The section below describes how to configure these.

Configuration

Parameters

repo_manage

Include repository to install recent fluentd (td-agent) from. Default: 'true'

package_ensure

Package ensure. Default: 'installed'

package_name

Package name. Default: 'td-agent'

package_install_options

Package install options. Default: '[]'

service_manage

Defines if the service should be managed by puppet. Default: 'true'

service_name

Name of the service. Default: 'td-agent'

service_ensure

Service ensure. Default: 'running'

service_enable

Defines if the service should be enabled. Default: 'true'

config_path

Path to configuration files Default: '/etc/td-agent'

config_dir

Configuration directory name Default: '${config_path}/conf.d'

config_file

Default configuration file name Default: '${config_path}/td-agent.conf'

conf_dir_manage

Exclusively handle config files into fluentd::conf_dir. Other files not created by puppet, will be deleted Default: 'false'

user_manage

Defines if the user should be manage, which will add the user to groups defined in user_groups. For example to be able to view the /var/log directory with group adm Default: 'true'

user_name

Default: 'td-agent'

user_group

Default: 'td-agent'

user_groups

Default: '[“adm”]'

Source

uentd::source { 'test':
iority => 10,
nfig   => {
'type'   => 'tail',
'format' => 'json',
'path'   => '/var/log/test-application/*.json',
'parse'  => {
  'type'       => 'regexp',
  'expression' => '^(?<name>[^ ]*) (?<user>[^ ]*) (?<age>\d*)$'
},
'storage'=> {
  'type'   => 'local',
  'path'   => 'test.pos'
},
'tag'    => 'application.test'

creates:

/td-agent/conf.d/10-source-test.conf
rce>
pe tail
rmat json
th /var/log/test-application/*.json
arse>
type regexp
expression ^(?<name>[^ ]*) (?<user>[^ ]*) (?<age>\d*)$
parse>
torage>
type local
path test.pos
storage>
g application.test
urce>

Filter

uentd::filter { 'test':
iority => 20,
ttern  => '*.test',
nfig   => {
'type'   => 'record_transformer',
'record' => {
  'hostname' => '${hostname}'
}

creates:

/td-agent/conf.d/20-filter-test.conf
ter *.test>
pe record_transformer
ecord>
hostname ${hostname}
record>
lter>

Match

uentd::match { 'test':
iority => 30,
ttern  => '*.test',
nfig   => {
'flush_interval' => '30s',
'type'           => 'secure_forward',
'secure'         => 'yes',
'shared_key'     => 'my_shared_key',
'self_hostname'  => 'instance.test.com',
'ca_cert_path'   => '/path/to/ca.cert',
'server'        => [{
  'host' => 'test.server.com',
}]

creates:

/td-agent/conf.d/30-match-test.conf
ch *.test>
ush_interval 30s
pe secure_forward
cure yes
ared_key my_shared_key
lf_hostname instance.test.com
_cert_path /path/to/ca.cert
erver>
host test.server.com
server>
tch>

Match Store

uentd::match { 'test':
iority => 30,
ttern  => '*.test',
nfig   => {
'type'  => 'copy',
'store' => [
  {
    'type'           => 'elasticsearch',
    'logstashformat' => true,
    'hosts'          => '172.20.10.17:9200',
    'flush_interval' => '30s',
  },
  {
    'type' => 'file',
    'path' => '/tmp/td-agent-debug.log',
  }
]

creates:

/td-agent/conf.d/30-match-test.conf
ch *.test>
pe copy
tore>
type elasticsearch
logstash_format true
hosts 172.20.10.17:9200
flush_interval 30s
store>
tore>
type file
path /tmp/crs
store>
tch>

Plugin Installation

This module gives you the possibility to install plugins as gem or files.

gem installation

uentd::plugin { 'fluent-plugin-elasticsearch':
pe => 'gem',
sure => '0.1.3'

file installation

uentd::plugin { 'fluent-plugin-elasticsearch':
pe   => 'file',
urce => 'puppet:///path/to/plugin'

Requirements

Modules:

• puppetlabs/apt >= 4.0.0

Limitations

This module has been built on and tested against Puppet 4.6 and higher.

The module has been tested on:

• Ubuntu 16.04

Testing on other platforms has been light and cannot be guaranteed.

Development

Running the test suite

To run the tests install the ruby dependencies with bundler and execute rake:

le install --path vendor/bundle
le exec rake spec
le exec rake lint
le exec rake beaker BEAKER_set=ubuntu-server-1404-x64
le exec rake beaker BEAKER_set=ubuntu-server-1604-x64
le exec rake beaker BEAKER_set=debian-60-x64
le exec rake beaker BEAKER_set=debian-78-x64
le exec rake beaker BEAKER_set=debian-82-x64
le exec rake beaker BEAKER_set=centos-66-x64
le exec rake beaker BEAKER_set=centos-70-x64

License

Copyright 2017 ATVAG GmbH

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

This Puppet module is being actively maintained by some fellow puppeteers at wywy.