CD2H gitForager

pantheon-systems/journal-2-logstash

Name: journal-2-logstash

Owner: Pantheon

Description: ship the journal to logstash servers

Created: 2016-02-03 03:38:43.0

Updated: 2018-04-08 23:53:44.0

Pushed: 2018-04-09 01:44:13.0

Homepage: null

Size: 103

Language: Go

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

journal-2-logstash

Securely ship JSON formatted logs from systemd's journald to logstash (ELK).

Features

Reads JSON formatted logs from systemd-journal-gatewayd (aka s-j-gatewayd) over HTTP on a unix socket. Local unix socket is used for extra security on the localhost.
Ships logs to logstash server using TLS with mutual authentication of client and server.
Saves the journal cursor periodically and on shutdown. Restarts from last log message on restarts. Reducing message loss.

Usage

Config

systemd-journal-gatewayd

The default configuration for s-j-gatewayd is to listen on a TCP socket. Override to listen on a unix sock in a protected path that journal-2-logstash will have access to:

/etc/systemd/system/systemd-journal-gatewayd.socket

t]
ription=Journal Gateway Service Socket

ket]
enStream=/run/systemd-journal-gatewayd.sock

tall]
edBy=sockets.target

journal-2-logstash

/etc/systemd/system/journal-2-logstash.service:

t]
ription=Journal 2 Logstash shipper

vice]
nd this service's output to the console instead of the journal to avoid a logging loop.
dardOutput=tty
dardError=tty

ronment=JOURNAL2LOGSTASH_URL=logstash:4000
ronment=JOURNAL2LOGSTASH_DEBUG=true
ronment=JOURNAL2LOGSTASH_SOCKET=/run/systemd-journal-gatewayd.sock
ronment=JOURNAL2LOGSTASH_STATE_FILE=/etc/journal2logstash.state

S keys and certs are not included in the container and should be mounted
to /etc/certs at runtime.
ronment=JOURNAL2LOGSTASH_TLS_KEY=/etc/certs/logger.key
ronment=JOURNAL2LOGSTASH_TLS_CERT=/etc/certs/logger.crt
ronment=JOURNAL2LOGSTASH_TLS_CA=/etc/certs/ca.crt

Start=/opt/journal-2-logstash/journal-2-logstash
art=on-failure
artSec=2s

tall]
edBy=multi-user.target

Logstash Receiver Config

Use the following configuration for the logstash receiver. This configuration requires all clients to present a certificate trusted by a CA in the cacert bundle.

Filtering rules convert journald's JSON format into Logstash JSON format.

t {
tcp {
        port => "4000"
        codec => "json"
        ssl_enable => "true"
        ssl_key => "/path/to/server.pem"
        ssl_cert => "/path/to/server.pem"
        ssl_cacert => "/path/to/ca_bundle.pem"
        ssl_verify => "true"
        tags => ["systemd_journal_json"]
     }



er {

convert timestamp and message fields from journald format into logstash format

 ("systemd_journal_json" in [tags]) {
### -- convert microsecond to millisecond timestamp for logstash
### -- extract into logstash `@timestamp` field.
### -- remove `__REALTIME_TIMESTAMP` field.
ruby   { code   => "event['__REALTIME_TIMESTAMP'] = event['__REALTIME_TIMESTAMP'].to_i / 1000" }
date   { match  => ["__REALTIME_TIMESTAMP", "UNIX_MS"] }
mutate { remove => ["__REALTIME_TIMESTAMP"] }

### -- convert journald `MESSAGE` to logstash's `message` field.
mutate { rename => { "MESSAGE" => "message" } }

### -- remove systemd fields you don't care about here
mutate { remove => ["__CURSOR", "_BOOT_ID"] }



put {
tdout { codec => "rubydebug" }

Developing

Building:

Run make build to build the journal-2-logstash binary.

Testing:

Unit tests: make test
Generate test coverage report: make cov
Open test coverage report in a browser: make cov_html

Docker demoing / testing

Requirements:

docker
docker-compose

Included in the repo is a docker-compose.yml in the test/ directory that will spinup two containers:

logger a fedora container running systemd as pid 1 and systemd-journald, systemd-journal-gatewayd, and the journal-2-logstash binary running.
logstash a logstash server instance configured to listen for JSON over TLS from the logger container.

Run make docker_up to start the containers. After the containers start (the logstash container takes the longest) you should immediately see messages flow into the logstash container and will be printed to stdout with the rubydebug logstash output plugin.

ke docker_up

er_1   |
er_1   | Welcome to Fedora 22 (Twenty Two)!
er_1   |
er_1   | Set hostname to <5ef5a7ea14ee>.

er_1   | 2016/02/12 18:11:31 Could not load cursor (open /etc/journal2logstash.state: no such file or directory). Will start reading from 'last boot time'.
er_1   | 2016/02/12 18:11:31 Error connecting to logstash: dial tcp 172.17.0.2:4000: getsockopt: connection refused
er_1   | 2016/02/12 18:11:33 Could not load cursor (open /etc/journal2logstash.state: no such file or directory). Will start reading from 'last boot time'.
er_1   | 2016/02/12 18:11:33 Error connecting to logstash: dial tcp 172.17.0.2:4000: getsockopt: connection refused



tash_1 | {
tash_1 |          "__MONOTONIC_TIMESTAMP" => "59420873093",
tash_1 |                       "PRIORITY" => "6",
tash_1 |                           "_UID" => "0",
tash_1 |                           "_GID" => "0",
tash_1 |                    "_MACHINE_ID" => "9d52a846ee1a4be2b2d6e563162c6aa3",
tash_1 |                      "_HOSTNAME" => "a586fdbd6d5b",
tash_1 |                "SYSLOG_FACILITY" => "3",
tash_1 |                     "_TRANSPORT" => "journal",
tash_1 |                      "CODE_FILE" => "../src/core/unit.c",
tash_1 |                      "CODE_LINE" => "1412",
tash_1 |                  "CODE_FUNCTION" => "unit_status_log_starting_stopping_reloading",
tash_1 |              "SYSLOG_IDENTIFIER" => "systemd",
tash_1 |                     "MESSAGE_ID" => "7d4958e842da4a758f6c1cdc7b36dcc5",
tash_1 |                           "_PID" => "1",
tash_1 |                          "_COMM" => "systemd",
tash_1 |                           "_EXE" => "/usr/lib/systemd/systemd",
tash_1 |                       "_CMDLINE" => "/usr/sbin/init",
tash_1 |                 "_CAP_EFFECTIVE" => "3fffffffff",
tash_1 |                "_SYSTEMD_CGROUP" => "/",
tash_1 |                           "UNIT" => "systemd-journal-gatewayd.service",
tash_1 |     "_SOURCE_REALTIME_TIMESTAMP" => "1455301048170788",
tash_1 |                       "@version" => "1",
tash_1 |                     "@timestamp" => "2016-02-12T18:17:28.171Z",
tash_1 |                           "host" => "172.17.0.3",
tash_1 |                           "tags" => [
tash_1 |         [0] "journal_json"
tash_1 |     ],
tash_1 |                        "message" => "Starting Journal Gateway Service..."
tash_1 | }

From another terminal you can generate log messages inside the logger container:

ho "hello there" | make docker_log

Observe the log message flow from journald -> journal-2-logstash -> logstash:

er_1   | 2016/02/12 18:19:15 [DEBUG] Received from journal: { "__CURSOR" : "s=aae5e906525c4c00be5e9d5026bbf40a;i=35c;b=7f0ea6b7d19f47f9a9bc928f32512ae7;m=ddc296aab;t=52b96b46b549a;x=e92c6b237a40211b", "__REALTIME_TIMESTAMP" : "1455301155574938", "__MONOTONIC_TIMESTAMP" : "59528276651", "_BOOT_ID" : "7f0ea6b7d19f47f9a9bc928f32512ae7", "PRIORITY" : "6", "_UID" : "0", "_GID" : "0", "_MACHINE_ID" : "9d52a846ee1a4be2b2d6e563162c6aa3", "_HOSTNAME" : "63c73cd1ec2e", "_TRANSPORT" : "stdout", "MESSAGE" : "hello there", "_PID" : "97", "_COMM" : "cat" } }

tash_1 | {
tash_1 |     "__MONOTONIC_TIMESTAMP" => "59528276651",
tash_1 |                  "PRIORITY" => "6",
tash_1 |                      "_UID" => "0",
tash_1 |                      "_GID" => "0",
tash_1 |               "_MACHINE_ID" => "9d52a846ee1a4be2b2d6e563162c6aa3",
tash_1 |                 "_HOSTNAME" => "63c73cd1ec2e",
tash_1 |                "_TRANSPORT" => "stdout",
tash_1 |                      "_PID" => "97",
tash_1 |                     "_COMM" => "cat",
tash_1 |                  "@version" => "1",
tash_1 |                "@timestamp" => "2016-02-12T18:19:15.574Z",
tash_1 |                      "host" => "172.17.0.3",
tash_1 |                      "tags" => [
tash_1 |         [0] "journal_json"
tash_1 |     ],
tash_1 |                   "message" => "hello there"
tash_1 | }

TODO

Make TLS client cert optional. This will be Useful for connecting to logz.io's TCP/TLS receiver which doesn't use client certs. workaround: specify an arbitrary client key and cert and logz.io will ignore it.
also check TODOs in source files

Misc Notes

curl testing of s-j-gatewayd:

curl -H'Accept: application/event-stream' -H 'Range: entries=:-1:1' 'localhost:19531/entries'

This work is supported by the National Institutes of Health's National Center for Advancing Translational Sciences, Grant Number U24TR002306. This work is solely the responsibility of the creators and does not necessarily represent the official views of the National Institutes of Health.