GoogleCloudPlatform/gcsbeat

Name: gcsbeat

Owner: Google Cloud Platform

Description: An Elastic Beat to ingest data from Google Cloud Storage

Created: 2018-03-30 19:30:15.0

Updated: 2018-05-23 20:32:04.0

Pushed: 2018-05-23 20:32:03.0

Homepage:

Size: 7621

Language: Go

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

GCSBeat

GCSBeat is an Elastic Beat to read logs/data from Google Cloud Storage (GCS) buckets. The beat reads JSON objects or raw text from files in a bucket and forwards them to a beats output.

Example use-cases:

• Read Stackdriver logs from a GCS bucket into Elastic.
• Read gzipped logs from cold-storage into Elastic.
• Restore data from an Elastic dump.
• Watch files on a local path (possibly a mounted GCS bucket) and upload them.
• Parse JSON logs and upload them.

Note: While this project is partially maintained by Google, this is not an official Google product.

Configuration Options

See the _meta/beat.yml file for a list of configuration options.

Make sure your user has permissions to read files on the bucket and write metadata. GCSBeat marks objects as being processed using metadata keys.

Example Configurations

Read archived redis logs from the local filesystem hourly and delete them after upload:

eat:
terval: 60m
cket_id: "file:///var/log/redis"
lete: true
le_matches: "*.log.gz"
dec: "text"
pack_gzip: true

Read Stackdriver logs from a bucket:

eat:
cket_id: my_log_bucket
on_key_file: /path/to/key.json
le_matches: "*.json"
dec: "json-stream"

Read files into two separate Elastic clusters:

uster 1 beat
eat:
cket_id: my_log_bucket
on_key_file: /region-one-key.json
tadata_key: "region-one-beat"

uster 2 beat
eat:
cket_id: my_log_bucket
on_key_file: /disaster-recovery-key.json
le_matches: "*.log"
tadata_key: "disaster-recovery-beat"

Read data from a read-only bucket:

eat:
cket_id: read_only_log_bucket
on_key_file: /path/to/key.json
le_matches: "*.log"
ocessed_db_path: "processed_file_list.db"

Getting Started with GCSBeat

You can either download GCSBeat compiled binaries or build them yourself.

Download

1. Download a binary for your system on the releases page.
2. Extract the archive and edit gcsbeat.yml for your environment. See above for examples.
3. Run it. For testing use: ./gcsbeat -e -v -c gcsbeat.yml. You can mock publishing for testing purposes using the -N argument.
4. (Optional) Run ./gcsbeat setup to install predefined indexes.

Build it Yourself

You can build GCSBeat yourself using the instructions in the DEVELOPING.md file.

Run

To run gcsbeat with info level logging configured:

sbeat -c gcsbeat.yml -e -v

Normal Mode:

sbeat -c gcsbeat.yml

Debug

It can sometimes be difficult to figure out why the plugin is or isn't picking up particular files. You can enable explain mode by running the beat with the -d "Explain" flag:

sbeat -d "Explain" -e

Here's an example and how to read it:

bucket gcsbeat-test has five files it can process.
 that Cloud Storage lists directories as files.
    [Explain]   storage/explain.go:55   Source "gs://gcsbeat-test" found 5 files
G   [Explain]   storage/explain.go:58    - "bak-backup-log.log"
G   [Explain]   storage/explain.go:58    - "new.log"
G   [Explain]   storage/explain.go:58    - "old.log"
G   [Explain]   storage/explain.go:58    - "test-folder/"
G   [Explain]   storage/explain.go:58    - "test-folder/test.log"

of the files have already been marked as processed using the x-gcsbeat-processed metadata key.
G   [Explain]   storage/explain.go:32   Test: has key "x-gcsbeat-processed"?
G   [Explain]   storage/explain.go:43    - "bak-backup-log.log" (pass)
G   [Explain]   storage/explain.go:43    - "new.log" (pass)
G   [Explain]   storage/explain.go:46    - "old.log" (fail)
G   [Explain]   storage/explain.go:43    - "test-folder/" (pass)
G   [Explain]   storage/explain.go:46    - "test-folder/test.log" (fail)
    [Explain]   storage/explain.go:50   Test: has key "x-gcsbeat-processed"? passed 3 of 5 files

 of the remaining files are already in the pending queue.
G   [Explain]   storage/explain.go:32   Test: already pending?
G   [Explain]   storage/explain.go:43    - "bak-backup-log.log" (pass)
G   [Explain]   storage/explain.go:43    - "new.log" (pass)
G   [Explain]   storage/explain.go:43    - "test-folder/" (pass)
    [Explain]   storage/explain.go:50   Test: already pending? passed 3 of 3 files

of the files match the include filter.
G   [Explain]   storage/explain.go:32   Test: matches "*.log"?
G   [Explain]   storage/explain.go:43    - "bak-backup-log.log" (pass)
G   [Explain]   storage/explain.go:43    - "new.log" (pass)
G   [Explain]   storage/explain.go:46    - "test-folder/" (fail)
    [Explain]   storage/explain.go:50   Test: matches "*.log"? passed 2 of 3 files

backup file matches the exclusion filter so it is skipped.
G   [Explain]   storage/explain.go:32   Test: does not match "bak-*"?
G   [Explain]   storage/explain.go:46    - "bak-backup-log.log" (fail)
G   [Explain]   storage/explain.go:43    - "new.log" (pass)
    [Explain]   storage/explain.go:50   Test: does not match "bak-*"? passed 1 of 2 files

tly one file remained.
    [GCS:gcsone]    beater/gcsbeat.go:131   Added 1 files to queue

License

right (c) 2018 Google Inc.

nsed under the Apache License, Version 2.0 (the "License");
may not use this file except in compliance with the License.
may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

ss required by applicable law or agreed to in writing, software
ributed under the License is distributed on an "AS IS" BASIS,
OUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
the License for the specific language governing permissions and
tations under the License.