Name: syslog-migration-release
Owner: Pivotal Cloud Foundry
Description: Deprecated BOSH release to facilitate PCF tiles migrating from metron_agent to syslog-release for forwarding logs.
Forked from: cloudfoundry/syslog-release
Created: 2017-03-07 09:59:31.0
Updated: 2018-05-07 15:27:09.0
Pushed: 2018-05-07 15:27:07.0
Homepage: https://github.com/cloudfoundry/syslog-release
Size: 1544
Language: Go
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
This release has been depricated in favor of syslog-release.
Existing major versions of this release will only continue to receive security updates, and those only to the extent that they are included in currently supported versions of PCF.
This is a fork of [syslog-release]. It was intended to help tiles which provided their own log forwarding implementation transition into a world where syslog responsibilities are more centralized.
Changes were made in the upstream syslog release,
with the only exception of migration-related changes which must have a specific end of life goal.
If you do want to make changes to syslog-migration-release,
you should add tests to the ruby tests and the acceptance tests.
All customizations must be configured via syslog.migration.* properties)
and those defaults must be equivalent to the upstream defaults.
Once a tile is no longer configuring syslog.migration.* properties, they should switch to the upstream syslog release. Available migration properties:
syslog.migration.disabled - If true, do not reconfigure rsyslog or forward /var/vcap/sys/log files (nearly equivalent to not installing the job). This is the only syslog-migration-release feature still supported in syslog-release.
syslog.migration.insistent_custom_rule - Rule will be applied even if the above property is true
syslog.migration.message_format - The message format used to send messages to remote endpoints. If no value is set, uses the upstream default.
rfc5424 - structured data of [instance@47450 director="%s" deployment="%s" group="%s" az="%s" id="%s"]. This is the default setting.job_index - structured data of - and [job=%s index=%d] prefixed in the messagejob_index_id - structured data of - and [job=%s index=%d id=%s] prefixed in the messagesyslog.migration.cleanup_conf_files - List of filenames to remove from /etc/rsyslog.d/
If you need any of these capabilities in syslog-release (other than disabled, which is already supported),
please contact the syslog team using the information below, (or in syslog-release).
This is a BOSH release to forward local syslog events in RFC5424 format to a remote syslog endpoint. It currently uses RSYSLOG which is pre-installed by the stemcell.
Download the latest release from bosh.io and include it in your manifest:
ases:
me: syslog
rsion: latest
If you are deploying the Cloud Foundry Application Runtime
using cf-deployment,
there is an ops-file available
that will add the syslog release and syslog_forwarder job,
and expose configuration variables.
Otherwise, you can co-locate
and configure
the syslog_forwarder yourself.
Add the syslog_forwarder
to forward all local syslog messages
from an instance
to a syslog endpoint.
Configure address and,
optionally,
port and transport:
ance_groups:
me: some-instance-group
bs:
name: syslog_forwarder
release: syslog
operties:
syslog:
address: <IP or hostname>
By default,
if the syslog endpoint is unavailable,
messages will be queued.
Alternatively, configure fallback_servers
for higher availability.
Only TCP or RELP are supported
for fallback functionality:
erties:
slog:
address: 10.10.10.100
fallback_servers:
- address: 10.10.10.101
- address: 10.10.10.102
TLS is supported with additional properties. The following example would forward syslog messages to papertrail:
erties:
slog:
address: logs4.papertrailapp.com
port: 12345
transport: tcp
tls_enabled: true
permitted_peer: "*.papertrailapp.com"
ca_cert: |
-----BEGIN CERTIFICATE-----
MIIFdDCCBFygAwIBAgIQJ2buVutJ846r13Ci/ITeIjANBgkqhkiG9w0BAQwFADBv
...
pu/xO28QOG8=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIENjCCAx6gAwIBAgIBATANBgkqhkiG9w0BAQUFADBvMQswCQYDVQQGEwJTRTEU
...
mnkPIAou1Z5jJh5VkpTYghdae9C8x49OhgQ=
-----END CERTIFICATE-----
Alternatively, if the intended syslog recipient's certificate
is signed by any Certificate Authority
in the BOSH instances' cert store
(most common CAs are included on the stemcell),
you can omit the ca_cert field entirely.
If you do include ca_cert,
please note that the standard
cert store will no longer be referenced.
This necessitates including
the entire certificate chain.
This release allows a custom rule to be inserted before the rule that accomplishes log forwarding. This can be useful if you only wish to forward certain logs, or if there is a certain type of log you wish to exclude from forwarding.
We have some simple documentation
with a few example rules in
example-custom-rules.md.
Please note: if your custom rule is invalid, no logs will be forwarded.
The syslog_storer is meant for testing.
Deploy it and configure your instances to forward logs to it.
It should not be co-located
with other jobs which also try to configure syslog.
Received logs are stored in /var/vcap/store/syslog_storer/syslog.log.
You can add it to a deployment manifest very simply:
ance_groups:
me: syslog_storer
bs:
name: syslog_storer
release: syslog
Remember to allow inbound traffic on TCP port 514 in your IaaS security groups.
This release forwards messages using the RFC5424 standard (natively supported by most log platforms). Forwarded messages are annotated with structured data that identify the originating BOSH instance (director, deployment, availability zone, instance group, and instance ID). Forwarded messages are also tagged with our private enterprise number, 47450.
<$PRI>$VERSION $TIMESTAMP $HOST $APP_NAME $PROC_ID $MSG_ID [instance@47450 director="$DIRECTOR" deployment="$DEPLOYMENT" group="$INSTANCE_GROUP" az="$AVAILABILITY_ZONE" id="$ID"] $MESSAGE
An example message from diego is transmitted as…
<14>1 2017-01-25T13:25:03.18377Z 192.0.2.10 etcd - - [instance@47450 director="test-env" deployment="cf" group="diego_database" az="us-west1-a" id="83bd66e5-3fdf-44b7-bdd6-508deae7c786"] [INFO] the leader is [https://diego-database-0.etcd.service.cf.internal:4001]
<14>1 2017-01-25T13:25:03.184491Z 192.0.2.10 bbs - - [instance@47450 director="test-env" deployment="cf" group="diego_database" az="us-west1-a" id="83bd66e5-3fdf-44b7-bdd6-508deae7c786"] {"timestamp":"1485350702.539694548","source":"bbs","message":"bbs.request.start-actual-lrp.starting","log_level":1,"data":{"actual_lrp_instance_key":{"instance_guid":"271f71c7-4119-4490-619f-4f44694717c0","cell_id":"diego_cell-2-41f21178-d619-4976-901c-325bc2d0d11d"},"actual_lrp_key":{"process_guid":"1545ce89-01e6-4b8f-9cb1-5654a3ecae10-137e7eb4-12de-457d-8e3e-1258e5a74687","index":5,"domain":"cf-apps"},"method":"POST","net_info":{"address":"192.0.2.12","ports":[{"container_port":8080,"host_port":61532},{"container_port":2222,"host_port":61533}]},"request":"/v1/actual_lrps/start","session":"418.1"}}
A sample logstash config with filters
to extract instance metadata is in
scripts/logstash-filters.conf.
RSYSLOG is a system for log processing; it is a drop-in replacement for the UNIX's venerable syslog, which logs messages to various files and/or log hosts. RSYSLOG can be configured as a storer (i.e. it receives log messages from other hosts) or a forwarder (i.e. it forwards system log messages to RSYSLOG storers, syslog servers, or log aggregation services).
The RSYSLOG configuration file is /etc/rsyslog.conf.
The RSYSLOG forwarder's customizations
are rendered into /etc/rsyslog.d/rsyslog.conf,
which is included by the configuration file.
To run the ruby tests run
le install
le exec rspec
Important: If you want to run the acceptance tests after you have run the ruby tests you are going to have to run
uninstall bosh_cli
to remove the v1.x version of bosh from your path. Then ensure your version of bosh is 2.x
To run the acceptance tests, first ensure you have a bosh deployment(preferably bosh-lite), set your BOSH_ENVIROMENT, ensure that syslog-migration-release is located in your GOPATH and then run
ripts/test
from the tests directory.