mapzen/api-traffic-processors

Name: api-traffic-processors

Owner: Mapzen

Description: null

Created: 2016-04-06 18:01:08.0

Updated: 2018-05-12 15:13:16.0

Pushed: 2017-09-15 19:02:31.0

Homepage: null

Size: 128

Language: JavaScript

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

Package for processing api traffic

Installing

clone package and run npm install

Testing

ackage Structure

 Processors
y points are called "processors" and are exposed in index.js.

ocessor obtains raw input, and sends it through parser -> formatter -> exporter

 Parsers
ers convert raw input into parsed records for a formatter to handle.

 parser has a matching fixture file in test/fixtures and its test file will make sure each line is parsed into the correct format.

 Formatters
atters convert parsed records into a specific format for logging/importing to redshift/etc.

 Exporters
rters should be created with "new" and have .add(record) and .addBatch(records) methods

piaxle Usage

 apiaxle to 'require' apiaxleKinesis.js or apiaxleLog.js by adding to apiaxle.traffic_processors in the stack.json in https://github.com/mapzen/opsworks-apiaxle

ou're running apiaxle locally, you can accomplish the same thing by adding to the "traffic_processors" section of the apiaxle.json config file, and then be sure to run apiaxle-proxy without -q so that traffic gets passed to apiaxle-proxy-event-subscriber.

ple:

node apiaxle-proxy.js -f 1 -p 3000 node apiaxle-proxy-event-subscriber.js -f 1

inesis Firehose to Redshift

ple command for delivery stream to copy to redshift:

COPY api_hits FROM 's3:///' CREDENTIALS 'aws_access_key_id=; aws_secret_access_key=' MANIFEST DELIMITER ' ' TIMEFORMAT 'auto';