mozilla-services/hindsight

Name: hindsight

Owner: Mozilla Services

Description: Hindsight - light weight data processing skeleton

Created: 2015-03-09 23:05:07.0

Updated: 2018-05-23 17:01:21.0

Pushed: 2018-05-15 18:46:08.0

Homepage: null

Size: 1482

Language: C

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

Hindsight

Overview

Hindsight is a C based data processing infrastructure based on the lua sandbox project. I have received several inquiries about a lighter weight and faster data pipeline with delivery guarantees to replace Heka. Hindsight is that light weight skeleton around the same lua sandbox offering 'at least once' delivery semantics. The skeleton is supplemented by extension packages including hundreds of data structures, algorithms, plugins, parsers and grammars. The extensions repository is where most of the active development is happening now as the core infrastructure (Hindsight and the Lua Sandbox) is stable and changes infrequently. There is also a Hindsight Administration UI available for monitoring, debugging and plugin management (you can check out a running instance here: hsadmin)

• Full Documentation
• Support
  • IRC: #hindsight on irc.mozilla.org
  • Mailing list: https://mail.mozilla.org/listinfo/hindsight

Build

Prerequisites

• Clang 3.1 or GCC 4.7+
• CMake (3.6+) - http://cmake.org/cmake/resources/software.html
• lua_sandbox (1.2.3+) - https://github.com/mozilla-services/lua_sandbox

CMake Build Instructions

git clone https://github.com/mozilla-services/hindsight.git
cd hindsight
mkdir release
cd release

# Linux
cmake -DCMAKE_BUILD_TYPE=release ..
make
ctest
cpack -G TGZ # (DEB|RPM|ZIP)

# Cross platform support is planned but not supported yet

Releases

• The master branch is the current release and is considered stable at all times.
• New versions can be released as frequently as every two weeks (our sprint cycle). The only exception would be for a high priority patch.
• All active work is flagged with the sprint milestone and tracked in the project dashboard.
• New releases occur the day after the sprint finishes.
• The version in the dev branch is updated
• The changes are merged into master
• A new tag is created

Docker Images

Docker images are constructed from the master and dev branches and can be pulled, or built using the Dockerfile.

Note that the Docker image built here is only a bare bones image containing just lua_sandbox and hindsight. For a more full featured image that also contains all of the extensions, see the Docker image for the extensions repo.

Contributions

• All pull requests must be made against the dev branch, direct commits to master are not permitted.
• All non trivial contributions should start with an issue being filed (if it is a new feature please propose your design/approach before doing any work as not all feature requests are accepted).