aws-samples/aws-geotagging-logs

Name: aws-geotagging-logs

Owner: AWS Samples

Description: Enriches AWS logs with geo-location information and uses Elasticsearch and AWS Athena to analyze.

Created: 2017-11-22 21:46:27.0

Updated: 2017-11-30 15:29:29.0

Pushed: 2017-11-24 23:43:05.0

Homepage: null

Size: 31093

Language: JavaScript

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

GeoTaging Log Files in AWS

Enrich log files with geo-location information and analyze using AWS Athena and/or ElasticSearch

Overview

This repo provides a CloudFormation template to enrich AWS log files with geo-location information. VPC Flowlogs, ELB access logs, CloudFront access logs, and CloudTrail logs are streamed through lambda and saved in an S3 Bucket. The Enriched logs are then sent to ElasticSearch or Athena for analysis. To securely access the Kibana dashboard a Cognito login page is provided.

Prerequisites

• Run Linux. (tested on Amazon Linux)
• Install npm >3 and node >6. (instructions)
• Clone this repo.
• Set up an AWS account. (instructions)
• Configure AWS CLI and local credentials. (instructions)
  
• Copy /config.example.json to /config.json and set the aws-region and aws credential profile used to deploy CloudFormation templates. Also for testing and automated deployments set the AWS Certificate ARN and Route53 ZoneId.

Getting Started

First, install all prerequisites:

install 

Next, use the following command to launch a CloudFormation template to create the S3 bucket to be used for Lambda code and CloudFormation templates. Wait for this template to complete (you can watch progress from the AWS CloudFormation console)

run stack dev/bootstrap up

After the template has launched, use the following command to build all assets and upload to the S3 bucket created in the previous step:

run upload

Finally, run the following command to get the url to launch your template.

run url

Components

CloudFormation Templates

The templates are found in the /templates directory. To build all templates run:

run cfn

The built templates will be in the /build/templates directory.

Lambda Functions

Lambda functions are found in the /lambda directory. To build all lambdas run:

run lambda

the built lambda zip files will be in the /build/lambdas

Proxy Server

The code for the ElasticSearch Proxy server is in the /proxy directory. It will be deployed to EC2 instances using CodeDeploy.

Running Tests

The following will launch a CloudFormation template to create AWS resources in your account that are used in the Lambda and CloudFormation tests.

run stack dev up

Once the template has completed you can run the tests in the following sections.

CloudFormation tests

The CloudFormation test templates are in the templates/test folder. Run a template test with:

run stack test/{template-name}

For example, if you want to launch a template with filename “es.json” run the following command:

run check test/es

You also can check a template's syntax with:

run check {template-name}

Running Lambda Function tests

Each lambda directory has its own tests that can be run by executing the following command in that directory:

run test

Some tests may require additional configuration to run properly.

License

See the LICENSE.md file for details