cloudfoundry/log-cache

Name: log-cache

Owner: Cloud Foundry

Description: null

Created: 2017-11-14 18:25:08.0

Updated: 2018-05-23 22:47:07.0

Pushed: 2018-05-23 22:47:05.0

Homepage: null

Size: 451

Language: Go

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

Log Cache

Log Cache persists data in memory from the Loggregator System.

Usage

This repository should be imported as:

import logcache "code.cloudfoundry.org/log-cache"

Source IDs

Log Cache indexes everything by the source_id field on the Loggregator Envelope. The source ID should distinguish the cluster from other clusters. It should not distinguish a specific instance.

Cloud Foundry

In terms of Cloud Foundry, the source ID can either represent an application guid (e.g. cf app <app-name> --guid), or a component name (e.g. doppler).

Each request must have the Authorization header set with a UAA provided token. If the token contains the doppler.firehose scope, the request will be able to read data from any source ID. If the source ID is an app guid, the Cloud Controller is consulted to verify if the provided token has the appropriate app access.

Shard Groups

Shard groups can be used to split up consumption among multiple readers. With a shard group and set of requester IDs, Log Cache will do its best to evenly spread results among the requesters.

Restful API via Gateway

Log Cache implements a restful interface for getting data.

GET /v1/read/<source-id>

Get data from Log Cache for the given source-id.

Request

Query Parameters:

• start_time is a UNIX timestamp in nanoseconds. It defaults to the start of the cache (e.g. date +%s). Start time is inclusive. [starttime..endtime)
• end_time is a UNIX timestamp in nanoseconds. It defaults to current time of the cache (e.g. date +%s). End time is exclusive. [starttime..endtime)
• envelope_types is a filter for Envelope Type. The available filters are: LOG, COUNTER, GAUGE, TIMER, and EVENT. If set, then only those types of envelopes will be emitted. This parameter may be specified multiple times to include more types.
• limit is the maximum number of envelopes to request. The max limit size is 1000 and defaults to 100.

 "http://<log-cache-addr>:8080/v1/read/<source-id>?start_time=<start-time>&end_time=<end-time>"

Response Body

nvelopes": {"batch": [...] }

GET /v1/meta

Lists the available source IDs that Log Cache has persisted.

Response Body

eta":{
"source-id-0":{"count":"100000","expired":"129452","oldestTimestamp":"1524071322998223702","newestTimestamp":"1524081739994226961"},
"source-id-1":{"count":"2114","oldestTimestamp":"1524057384976840476","newestTimestamp":"1524081729980342902"},
...

Response fields

• count contains the number of envelopes held in Log Cache
• expired, if present, is a count of envelopes that have been pruned
• oldestTimestamp and newestTimestamp are the oldest and newest entries for the source, in nanoseconds since the Unix epoch.

GET /v1/shard_group/<group-name>

Reads from the given shard-group. The shard-group's source-ids are merged and sorted.

Query Parameters:

• requester_id is a string used to shard data across multiple clients. This string should be unique for each client reading from the group. If multiple clients use the same requester ID, sharding may not be reliable.
• start_time is UNIX timestamp in nanoseconds. It defaults to the start of the cache (e.g. date +%s). Start time is inclusive. [starttime..endtime)
• end_time is UNIX timestamp in nanoseconds. It defaults to current time of the cache (e.g. date +%s). End time is exclusive. [starttime..endtime)
• envelope_types is a filter for Envelope Type. The available filters are: LOG, COUNTER, GAUGE, TIMER, and EVENT. If set, then only those types of envelopes will be emitted. This parameter may be specified multiple times to include more types.
• limit is the maximum number of envelopes to request. The max limit size is 1000 and defaults to 100.

 "http://<log-cache-addr>:8080/v1/shard_group/<group-name>?start_time=<start-time>&end_time=<end-time>&requester_id=<requester-id>"

Response Body

nvelopes": {"batch": [...] },
rgs": [...]

PUT /v1/shard_group/<group-name>/

Adds the given source ids to the given shard-group. If the shard-group does not exist, then it gets created. Each shard-group may contain many sub-groups. Each sub-group may contain many source-ids. Each requester (identified by a requester_id) will receive an equal subset of the shard group. A sub-group ensures that each requester receives envelopes for the given source-ids grouped together (and not spread across other requesters). When setting a shard-group, an arg can also be provided. This arg will be returned when reading back from the group. This gives context as to what data has been read.

Request Body

ourceIds": [
"source-id-1",
"source-id-2"

rg": "some-arg"

 "http://<log-cache-addr>:8080/v1/shard_group/<group-name>" -XPUT -d'{"sourceIds":["source-id-1","source-id-2"]}'

Response Body

GET /v1/shard_group/<group-name>/meta

Gets meta information about the shard-group.

 "http://<log-cache-addr>:8080/v1/shard_group/<group-name>/meta"

Response Body

ource_ids": [...],
equester_ids": [...],
rgs": [...]

GET /v1/promql

Issues a PromQL query against Log Cache data.

 -G "http://<log-cache-addr>:8080/v1/promql" --data-urlencode 'query=metrics{source_id="source-id-1"}'

Response Body

ector": {
"samples": [{ "metric": {...}, "point": {...} }]

Cloud Foundry CLI Plugin

Log Cache provides a plugin for the Cloud Foundry command line tool, which makes interacting with the API simpler.