Name: openshift-fluentd-forwarder
Owner: Centers for Disease Control and Prevention Surveillance Strategy
Description: null
Created: 2017-06-21 18:08:16.0
Updated: 2018-05-17 18:08:26.0
Pushed: 2018-05-17 18:08:25.0
Homepage: null
Size: 36
Language: Shell
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
OpenShift can be configured to host an EFK stack that stores and indexes log data but at some sites a log aggregation system is already in place. A forwarding fluentd can be configured to forward log data to a remote collection point. Using a containerized version that runs within OCP both simplifies some of the infrastructure and certificate management and allows rapid deployment with resiliancy.
This project constitutes a work of the United States Government and is not subject to domestic copyright protection under 17 USC § 105. This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication. All contributions to this project will be released under the CC0 dedication. By submitting a pull request you are agreeing to comply with this waiver of copyright interest.
The project utilizes code licensed under the terms of the Apache Software License and therefore is licensed under ASL v2 or later.
This program is free software: you can redistribute it and/or modify it under the terms of the Apache Software License version 2, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the Apache Software License for more details.
You should have received a copy of the Apache Software License along with this program. If not, see http://www.apache.org/licenses/LICENSE-2.0.html
This project contains only non-sensitive, publicly available data and information. All material and community participation is covered by the Surveillance Platform Disclaimer and Code of Conduct. For more information about CDC's privacy policy, please visit http://www.cdc.gov/privacy.html.
This quickstart should be run on an installation of OpenShift Enterprise V3 with an existing EFK deployment.
Have the [fluentd-forwarder-build-config-template](./fluentd-forwarder-build-config-template.yaml) and the [fluentd-forwarder-template](./fluentd-forwarder-template.yaml) available for adding to the cluster. These templates will be needed for creating builds and deploying the application.
The EFK stack should already be configured in the “logging” namespace.
Choose the RHEL (default) or CentOS (-centos) flavor of build configuration. Add the build configuration template to the logging namespace.
roject logging
pply -f fluentd-forwarder-build-config-template.yaml
For CentOS use the -centos template.
roject logging
pply -f fluentd-forwarder-centos-build-config-template.yaml
Process the template to create a build, using any relevant variables. In the general case the defaults are fine.
roject logging
rocess fluentd-forwarder | oc apply -f -
For CentOS process the -centos template.
roject logging
rocess fluentd-forwarder-centos | oc apply -f -
By default the build will disable all repositories in the base image, enabling only the ones required for installing the required packages. If you want to use the build process to use the existing repository config as is (e.g. if you're using a custom base image) then set the USE_SYSTEM_REPOS environment variable to any value in the BuildConfig object.
roject logging
et env bc/fluentd-forwarder USE_SYSTEM_REPOS=1
On CentOS:
roject logging
et env bc/fluentd-forwarder-centos USE_SYSTEM_REPOS=1
Build the fluentd-forwarder
roject logging
tart-build fluentd-forwarder-build
To build with CentOS use the -centos build configuration.
roject logging
tart-build fluentd-forwarder-centos-build
Add the template to the logging namespace:
roject logging
pply -f fluentd-forwarder-template.yaml
Create the new logging forwarder application deployment:
roject logging
ew-app fluentd-forwarder \
p "P_TARGET_TYPE=remote_syslog" \
p "P_TARGET_HOST=rsyslog.internal.company.com" \
p "P_TARGET_PORT=514" \
p "P_SHARED_KEY=changeme"
To do the same for CentOS you need to reference the ImageStream created by that build.
roject logging
ew-app fluentd-forwarder \
p "P_IMAGE_NAME=fluentd-forwarder-centos"
p "P_TARGET_TYPE=remote_syslog" \
p "P_TARGET_HOST=rsyslog.internal.company.com" \
p "P_TARGET_PORT=514" \
p "P_SHARED_KEY=changeme"
A full list of parameters can be found in the template. Additional non-parameterized parameters and environment variables can be found in the Dockerfile.
The “logging-fluentd” configmap's “data.secure-forward.conf” key needs to be edited as well.
dit configmap -n logging logging-fluentd
Edit the following YAML:
:
cure-forward.conf: |
<store>
@type secure_forward
self_hostname ${HOSTNAME}
shared_key changeme
secure yes
enable_strict_verification yes
ca_cert_path /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
<server>
host fluentd-forwarder.logging.svc.cluster.local
port 24284
</server>
</store>
This will cause each individual fluentd logger to begin forwarding to the service address fluentd-forwarder.logging.svc.cluster.local which was created with the new-app command. That service has it's own cluster-generated certificates and the “ca_cert_path” value here is used to trust the cluster's service signer CA.
After saving the above changes the logging-fluentd pods need to be restarted. Delete them and they will be recreated.
elete pod -l component=fluentd
After creating the application you can edit the configuration for the logging forwarder in a more direct manner by manipulating the configuration map.
dit configmap -n logging fluentd-forwarder
This will allow you to edit a copy of the configuration template and override the one provided in the Docker container without performing a rebuild.
Any environment variables (like SHARED_KEY or TARGET_TYPE) will be substituted during the Pod startup just as with the built-in template using the envsubst command. Additional parameters can be added to the deployment config or directly edited here.
:
uentd.conf: |
<source>
@type secure_forward
self_hostname "#{ENV['HOSTNAME']}"
bind 0.0.0.0
port 24284
shared_key ${SHARED_KEY}
secure ${IS_SECURE}
enable_strict_verification ${STRICT_VERIFICATION}
ca_cert_path ${CA_PATH}
cert_path ${CERT_PATH}
private_key_path ${KEY_PATH}
private_key_passphrase ${KEY_PASSPHRASE}
</source>
<filter **>
@type record_transformer
<record>
forwarded_by "#{ENV['HOSTNAME']}"
source_component "OCP"
</record>
</filter>
<match **>
type ${TARGET_TYPE}
host ${TARGET_HOST}
port ${TARGET_PORT}
output_format json
</match>
If you save changes to this configuration map you will need to delete the pods for the deployment so they can be recreated.
elete pods -l name=fluentd-forwarder
In some use cases it might be necessary to perform filtering at the external fluentd process. This would be done to reduce the number or type of messages that are forwared.
Using the fluentd.conf file from above a new record will be added to the json message. The record kubernetes_namespace_name will be set to the OpenShift namespace from where the messages originated.
Using the appened records, a filter is applied to all messages. Messages where kubernetes_namespace_name match the specified regex pattern null|devnull|logging|default|kube-public|kube-service-catalog|kube-system|logging|management-infra|openshift|openshift-ansible-service-broker|openshift-infra|openshift-metrics|openshift-node are dropped and not forwared on.
:
uentd.conf: |
<source>
@type secure_forward
self_hostname "#{ENV['HOSTNAME']}"
bind 0.0.0.0
port 24284
shared_key ${SHARED_KEY}
secure ${IS_SECURE}
enable_strict_verification ${STRICT_VERIFICATION}
ca_cert_path ${CA_PATH}
cert_path ${CERT_PATH}
private_key_path ${KEY_PATH}
private_key_passphrase ${KEY_PASSPHRASE}
</source>
<filter **>
@type record_transformer
enable_ruby yes
auto_typecast yes
<record>
kubernetes_namespace_name ${record["kubernetes"].nil? ? 'devnull' : record["kubernetes"]["namespace_name"].nil? ? 'devnull' : record["kubernetes"]["namespace_name"]}
forwarded_by "#{ENV['HOSTNAME']}"
source_component "OCP"
</record>
</filter>
#Run filter on kube messages
<filter **>
@type grep
#Always filter out the restricted namespaces
exclude1 kubernetes_namespace_name (null|devnull|logging|default|kube-public|kube-service-catalog|kube-system|logging|management-infra|openshift|openshift-ansible-service-broker|openshift-infra|openshift-metrics|openshift-node)
</filter>
<match **>
@type ${TARGET_TYPE}
#host ${TARGET_HOST}
#port ${TARGET_PORT}
${ADDITIONAL_OPTS}
</match>
All system level messages would be dropped in the example above. To filter system messages filter on the system.** tag.
:
uentd.conf: |
<source>
@type secure_forward
self_hostname "#{ENV['HOSTNAME']}"
bind 0.0.0.0
port 24284
shared_key ${SHARED_KEY}
secure ${IS_SECURE}
enable_strict_verification ${STRICT_VERIFICATION}
ca_cert_path ${CA_PATH}
cert_path ${CERT_PATH}
private_key_path ${KEY_PATH}
private_key_passphrase ${KEY_PASSPHRASE}
</source>
<filter system.**>
#Add system filtering logic here.
</filter>
<match system.**>
@type ${TARGET_TYPE}
#host ${TARGET_HOST}
#port ${TARGET_PORT}
${ADDITIONAL_OPTS}
</match>
#Toss the rest of the records.
<match **>
@type null
</match>
The best verification is that logs are showing up in the remote location. The application sets two tags “forwarded_by” which is set to the pod's hostname and “source_component” which is always set to “OCP”. You can use those tags to search the logging collection facility for the logs being produced.
If VERBOSE is set as an environment variable in the deployment config (oc edit dc fluentd-forwarder) then you can tail the logs of the fluentd-forwarder container and you should see a lot of information about reads. This is not the most reliable test but it will at least point in the right direction.
ogs fluentd-forwarder-1-a3zdf
-06-19 21:05:20 +0000 [debug]: plugin/input_session.rb:122:on_read: on_read
-06-19 21:05:23 +0000 [debug]: plugin/input_session.rb:122:on_read: on_read
-06-19 21:05:24 +0000 [debug]: plugin/input_session.rb:122:on_read: on_read
-06-19 21:05:25 +0000 [debug]: plugin/input_session.rb:122:on_read: on_read
-06-19 21:05:25 +0000 [debug]: plugin/input_session.rb:122:on_read: on_read
-06-19 21:05:25 +0000 [debug]: plugin/input_session.rb:122:on_read: on_read
-06-19 21:05:25 +0000 [debug]: plugin/input_session.rb:122:on_read: on_read
-06-19 21:05:26 +0000 [debug]: plugin/input_session.rb:122:on_read: on_read
-06-19 21:05:26 +0000 [debug]: plugin/input_session.rb:122:on_read: on_read
This project contains only non-sensitive, publicly available data and information. All material and community participation is covered by the Surveillance Platform Disclaimer and Code of Conduct. For more information about CDC's privacy policy, please visit http://www.cdc.gov/privacy.html.
Anyone is encouraged to contribute to the project by forking and submitting a pull request. (If you are new to GitHub, you might start with a basic tutorial.) By contributing to this project, you grant a world-wide, royalty-free, perpetual, irrevocable, non-exclusive, transferable license to all users under the terms of the Apache Software License v2 or later.
All comments, messages, pull requests, and other submissions received through CDC including this GitHub page are subject to the Presidential Records Act and may be archived. Learn more at http://www.cdc.gov/other/privacy.html.
This project is not a source of government records, but is a copy to increase collaboration and collaborative potential. All government records will be published through the CDC web site.