internetitem/logback-elasticsearch-appender

Name: logback-elasticsearch-appender

Owner: internetitem

Description: Logback Elasticsearch Appender

Created: 2015-04-08 13:07:13.0

Updated: 2018-04-28 10:09:45.0

Pushed: 2017-12-12 17:42:55.0

Homepage: null

Size: 104

Language: Java

GitHub Committers

UserMost Recent Commit# Commits

Other Committers

UserEmailMost Recent Commit# Commits

README

Logback Elasticsearch Appender

Build Status

Send log events directly from Logback to Elasticsearch. Logs are delivered asynchronously (i.e. not on the main thread) so will not block execution of the program. Note that the queue backlog can be bounded and messages can be lost if Elasticsearch is down and either the backlog queue is full or the producer program is trying to exit (it will retry up to a configured number of attempts, but will not block shutdown of the program beyond that). For long-lived programs, this should not be a problem, as messages should be delivered eventually.

This software is dual-licensed under the EPL 1.0 and LGPL 2.1, which is identical to the Logback License itself.

Usage

Include slf4j and logback as usual (depending on this library will not automatically pull them in).

In your pom.xml (or equivalent), add:

 <dependency>
    <groupId>com.internetitem</groupId>
    <artifactId>logback-elasticsearch-appender</artifactId>
    <version>1.6</version>
 </dependency>

In your logback.xml:

    <appender name="ELASTIC" class="com.internetitem.logback.elasticsearch.ElasticsearchAppender">
        <url>http://yourserver/_bulk</url>
        <index>logs-%date{yyyy-MM-dd}</index>
        <type>tester</type>
        <loggerName>es-logger</loggerName> <!-- optional -->
        <errorLoggerName>es-error-logger</errorLoggerName> <!-- optional -->
        <connectTimeout>30000</connectTimeout> <!-- optional (in ms, default 30000) -->
        <errorsToStderr>false</errorsToStderr> <!-- optional (default false) -->
        <includeCallerData>false</includeCallerData> <!-- optional (default false) -->
        <logsToStderr>false</logsToStderr> <!-- optional (default false) -->
        <maxQueueSize>104857600</maxQueueSize> <!-- optional (default 104857600) -->
        <maxRetries>3</maxRetries> <!-- optional (default 3) -->
        <readTimeout>30000</readTimeout> <!-- optional (in ms, default 30000) -->
        <sleepTime>250</sleepTime> <!-- optional (in ms, default 250) -->
        <rawJsonMessage>false</rawJsonMessage> <!-- optional (default false) -->
        <includeMdc>false</includeMdc> <!-- optional (default false) -->
        <maxMessageSize>100</maxMessageSize> <!-- optional (default -1 -->
        <authentication class="com.internetitem.logback.elasticsearch.config.BasicAuthentication" /> <!-- optional -->
        <properties>
            <property>
                <name>host</name>
                <value>${HOSTNAME}</value>
                <allowEmpty>false</allowEmpty>
            </property>
            <property>
                <name>severity</name>
                <value>%level</value>
            </property>
            <property>
                <name>thread</name>
                <value>%thread</value>
            </property>
            <property>
                <name>stacktrace</name>
                <value>%ex</value>
            </property>
            <property>
                <name>logger</name>
                <value>%logger</value>
            </property>
        </properties>
        <headers>
            <header>
                <name>Content-Type</name>
                <value>text/plain</value>
            </header>
        </headers>
    </appender>

    <root level="info">
        <appender-ref ref="FILELOGGER" />
        <appender-ref ref="ELASTIC" />
    </root>

    <logger name="es-error-logger" level="INFO" additivity="false">
        <appender-ref ref="FILELOGGER" />
    </logger>

    <logger name="es-logger" level="INFO" additivity="false">
        <appender name="ES_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
            <!-- ... -->
            <encoder>
                <pattern>%msg</pattern> <!-- This pattern is important, otherwise it won't be the raw Elasticsearch format anyomre -->
            </encoder>
        </appender>
    </logger>

Configuration Reference

The fields @timestamp and message are always sent and can not currently be configured. Additional fields can be sent by adding <property> elements to the <properties> set.

Groovy Configuration

If you configure logback using logback.groovy, this can be configured as follows:

  import com.internetitem.logback.elasticsearch.ElasticsearchAppender

  appender("ELASTIC", ElasticsearchAppender){
    url = 'http://yourserver/_bulk'
    index = 'logs-%date{yyyy-MM-dd}'
    type = 'log'
    rawJsonMessage = true
    errorsToStderr = true
    authentication = new BasicAuthentication()
    def configHeaders = new HttpRequestHeaders()
    configHeaders.addHeader(new HttpRequestHeader(name: 'Content-Type', value: 'text/plain'))
    headers = configHeaders
  }

  root(INFO, ["ELASTIC"])

Authentication

Authentication is a pluggable mechanism. You must specify the authentication class on the XML element itself. The currently supported classes are:

Logback Access

Included is also an Elasticsearch appender for Logback Access. The configuration is almost identical, with the following two differences:

This work is supported by the National Institutes of Health's National Center for Advancing Translational Sciences, Grant Number U24TR002306. This work is solely the responsibility of the creators and does not necessarily represent the official views of the National Institutes of Health.