Name: marathon-consul
Owner: Allegro Tech
Description: Integrates Marathon apps with Consul service discovery.
Created: 2015-12-04 11:26:26.0
Updated: 2018-01-03 19:56:30.0
Pushed: 2017-11-15 09:08:41.0
Size: 17505
Language: Go
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
Register Marathon Tasks as Consul Services for service discovery.
marathon-consul takes information provided by the Marathon event bus and
forwards it to Consul agents. It also re-syncs all the information from Marathon
to Consul on startup and repeats it with given interval.
Note: In the future release Event Bus (callbacks) will be considered deprecated and eventually removed in favor of Event Stream (SSE). Right now marathon-consul is supporting both solutions. SSE is provided as experimental feature, disabled by default (more).
This project is based on
To simply compile and run the source code:
un main.go [options]
To run the tests:
test
To build the binary:
build
Binary distribution of marathon-consul can be downloaded directly from the releases page.
Download the build dedicated to your OS. After unpacking the archive, run marathon-consul binary. You can also add some options, for example:
thon-consul --marathon-location=marathon.service.consul:8080 --sync-interval=5m --log-level=debug
If you are a Debian/Ubuntu user, you can easily install marathon-consul as a deb package using APT package manager. Both upstart and systemd service managers are supported.
All releases are published as deb packages to our repository at Bintray.
To install marathon-consul with apt-get, simply follow the instructions:
d our public key to apt
-s https://bintray.com/user/downloadSubjectPublicKey?username=allegro | sudo apt-key add -
d the repository url
"deb http://dl.bintray.com/v1/content/allegro/deb /" | sudo tee /etc/apt/sources.list.d/marathon-consul.list
date apt cache
apt-get -y update
stall latest release of marathon-consul
apt-get -qy install marathon-consul
Run it with service marathon-consul start. The configuration file is located at /etc/marathon-consul.d/config.json.
To build docker image run
docker
Then you can run it with
er run -d -P allegro/marathon-consul [options]
If you want to use SSL you will need to expose cert store to Docker. The Following line is only example, your cert store might be different depending on your system.
er run '/etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt' -P allegro/marathon-consul
consul will be registered in Consul.
If the consul label is left blank like "consul": "", the task will be registered with the app name
by default. A different name can be provided as the label's value, e.g.
"consul": "customName". As an exception to this rule, for backward compatibility with the 0.3.x branch, a value of true is resolved to the default name.: "my-new-app",
els": {
sul": ""
consul-tag property will be maintained. By default, "consul-tag": "marathon" is automatically added when the task is registered.tag are converted to consul-tags, and appear in Consul as ServiceTags.: "my-new-app",
els": {
sul": "",
nish": "tag",
rics": "tag"
-X GET http://localhost:8500/v1/catalog/service/my-new-app
viceName": "my-new-app",
viceTags": [
athon",
nish",
rics",
athon-task:my-new-app.6a95bb03-6ad3-11e6-beaf-080027a7aca0"
ery service registration contains an additional tag `marathon-task` specifying the Marathon task id related to this registration.
there are multiple ports in use for the same app, note that only the first one will be registered by marathon-consul in Consul.
ou need to register your task under multiple ports, refer to *Advanced usage* section below.
Task healthchecks
least one HTTP healthcheck should be defined for a task. The task is registered when Marathon marks it as alive.
e provided HTTP healthcheck will be transferred to Consul.
e [this](https://mesosphere.github.io/marathon/docs/health-checks.html)
more details.
Command healthchecks
thchecks commands are registered in Consul with a simple variable substitution.
OST is replaced by task hostname.
ORT0, $PORT1... are replaced by port number defined in task.
g a special shell syntax can break this variable substitution (\$HOST, ${PORT0} ...)
Sync
e scheduled Marathon-consul sync may run in two modes:
- Only on node that is the current [Marathon-leader](https://mesosphere.github.io/marathon/docs/rest-api.html#get-v2-leader),
`marathon-leader` parameter should be set to `hostname:port` the current node appears in the Marathon cluster.
This mode is **enabled by default** and the `marathon-leader` property is set to the hostname resolved by OS.
Note that there is a difference between `marathon-leader` and `marathon-location`: `marathon-leader` is used for
node leadership detection (should be set to cluster-wide node name), while `marathon-location` is used for
connection purpose (may be set to `localhost`)
- On every node, `sync-force` parameter should be set to `true`
marathon-consul fails on startup sync and you see following error
n't get Consul services: No Consul client available in agents cache"`
may be caused by empty consul agents cache. If this occurs try configuring
consul-local-agent-host` to Consul Master or Consul agent.
Options
ment | Default | Description
------------------------|-----------------|------------------------------------------------------
ig-file | | Path to a JSON file to read configuration from. Note: Will override options set earlier on the command line
ul-auth | `false` | Use Consul with authentication
ul-auth-password | | The basic authentication password
ul-auth-username | | The basic authentication username
ul-enable-tag-override | `false` | Disable the anti-entropy feature for all services
ul-ignored-healthchecks | | A comma separated blacklist of Marathon health check types that will not be migrated to Consul, e.g. command,tcp
ul-local-agent-host | | Consul Agent hostname or IP that should be used for startup sync and service listing operations
ul-name-separator | `.` | Separator used to create default service name for Consul
ul-get-services-retry | `3` | Number of retries on failure when performing requests to Consul. Each retry uses different cached agent
ul-max-agent-failures | `3` | Max number of consecutive request failures for agent before removal from cache
ul-port | `8500` | Consul port
ul-ssl | `false` | Use HTTPS when talking to Consul
ul-ssl-ca-cert | | Path to a CA certificate file, containing one or more CA certificates to use to validate the certificate sent by the Consul server to us
ul-ssl-cert | | Path to an SSL client certificate to use to authenticate to the Consul server
ul-ssl-verify | `true` | Verify certificates when connecting via SSL
ul-tag | `marathon` | Common tag name added to every service registered in Consul, should be unique for every Marathon-cluster connected to Consul
ul-timeout | `3s` | Time limit for requests made by the Consul HTTP client. A Timeout of zero means no timeout
ul-token | | The Consul ACL token
ts-queue-size | `1000` | Size of events queue
t-max-size | `4096` | Maximum size of event to process (bytes)
en | `:4000` | Accept connections at this address
file | | Save logs to file (e.g.: `/var/log/marathon-consul.log`). If empty logs are published to STDERR
format | `text` | Log format: JSON, text
level | `info` | Log level: panic, fatal, error, warn, info, or debug
thon-location | `localhost:8080`| Marathon URL
thon-password | | Marathon password for basic auth
thon-protocol | `http` | Marathon protocol (http or https)
thon-ssl-verify | `true` | Verify certificates when connecting via SSL
thon-timeout | `30s` | Time limit for requests made by the Marathon HTTP client. A Timeout of zero means no timeout
thon-username | | Marathon username for basic auth
thon-leader | | Marathon cluster-wide node name (defaults to <hostname>:8080), the some leader specific calls will be made only if the specified node is the current Marathon-leader. Set to `*` to always act like a Leader.
ics-interval | `30s` | Metrics reporting interval
ics-location | | Graphite URL (used when metrics-target is set to graphite)
ics-prefix | `default` | Metrics prefix (default is resolved to <hostname>.<app_name>
ics-target | `stdout` | Metrics destination stdout or graphite (empty string disables metrics)
ry-dsn | | Sentry DSN. If it's not set sentry will be disabled
ry-env | | Sentry environment
ry-level | `error` | Sentry alerting level (info|warning|error|fatal|panic)
ry-timeout | `1s` | Sentry hook initialization timeout
retries | `0` | Number of times to recover SSE stream.
retry-backoff | `0s` | Configuration of initial time between retries to recover SSE stream.
-enabled | `true` | Enable Marathon-consul scheduled sync
-force | `false` | Force leadership-independent Marathon-consul sync (run always)
-interval | `15m0s` | Marathon-consul sync interval
ers-pool-size | `10` | Number of concurrent workers processing events
Endpoints
oint | Description
------|------------------------------------------------------------------------------------
alth` | healthcheck - returns `OK`
dvanced usage
Register under multiple ports
ou need to map your Marathon task into multiple service registrations in Consul, you can configure marathon-consul
Marathon's `portDefinitions`:
“id”: “my-new-app”, “labels”: {
"consul": "",
"common-tag": "tag"
}, “portDefinitions”: [
{
"labels": {
"consul": "my-app-custom-name"
}
},
{
"labels": {
"consul": "my-app-other-name",
"specific-tag": "tag"
}
}
]
configuration would result in two service registrations:
curl -X GET http://localhost:8500/v1/catalog/service/my-app-custom-name … “ServiceName”: “my-app-custom-name”, “ServiceTags”: [ “marathon”, “common-tag”, “marathon-task:my-new-app.6a95bb03-6ad3-11e6-beaf-080027a7aca0” ], “ServicePort”: 31292, …
curl -X GET http://localhost:8500/v1/catalog/service/my-app-other-name … “ServiceName”: “my-app-other-name”, “ServiceTags”: [ “marathon”, “common-tag”, “specific-tag”, “marathon-task:my-new-app.6a95bb03-6ad3-11e6-beaf-080027a7aca0” ], “ServicePort”: 31293, …
ny port definition contains the `consul` label, then advanced configuration mode is enabled. As a result, only the ports
aining this label are registered, under the name specified as the label's value ? with empty value resolved to default name.
s don't have to be unique ? you can have multiple registrations under the same name, but on different ports,
aps with different tags. Note that the `consul` label still needs to be present in the top-level application labels, even
gh its value won't have any effect.
configured in the top-level application labels will be added to all registrations. Tags configured in the port definition
ls will be added only to corresponding registrations.
registrations share the same `marathon-task` tag.
igration to version 1.x.x
l 1.x.x marathon-consul would register services in Consul with registration id equal to related Marathon task id. Since 1.x.x registration ids are different and
dditional tag, `marathon-task`, is added to each registration.
ou update marathon-consul from version 0.x.x to 1.x.x, expect the synchronization phase during the first startup to
gister all healthy services managed by marathon-consul to the new format. Unhealthy services will get deregistered in the process.
SE Support
e using SSE please consider:
E is using Web module config for queues, event sizes, in the future will be moved to sse module,
E is using marathon-leader config for determining current leader, when this value match leader returned by marathon (/v2/leader endpoint)
SSE is started on this instance,
en enabled SSE is spawning its own own set of workers and separated dispatcher,
advised to disable marathon callback subscription when enabling SSE, otherwise it might result in doubling registers and deregisers.
TTP callbacks support
thon-Consul does not support HTTP callbacks.
thon [deprecated support for HTTP callbacks in 1.4](https://github.com/mesosphere/marathon/blob/master/changelog.md#deprecate-event-callback-subscriptions).
mechanism is no longer available starting from [Marathon 1.5](https://github.com/mesosphere/marathon/blob/master/changelog.md#event-subscribers-has-been-removed).
nown limitations
following section describes known limitations in `marathon-consul`.
Marathon when a deployment changing the application's service name (by changing its `labels`) is being stopped, it changes app's configuration anyway.
is means we loose the link between the app and the services registered with the old name in Consul.
ter on, if another deployment takes place, new services are registered with a new name, the old ones are not being deregistered though.
scheduled sync is required to wipe them out.
elease
elease new version of marathon-consul follow steps:
ommit all changes you need for release to master branch.
git checkout master`
git checkout -b release/<version>` e.g., `git checkout -b release/1.3.1`
make version v=<version>` e.g., `make version v=1.3.1`
git push`
reate pull request from branch `release/<version>` to master.
nce pull request gets merged put tag on this commit (remember to update your master with `git pull`)
git tag <version> -f` e.g., `git tag 1.3.1 -f`. Create annotated tag with release notes in it.
ravis will automatically prepare github release from tag on master. Go there and update release notes.
Copy github release to bintray.
icense
thon-consul is released under the Apache 2.0 license (see [LICENSE](LICENSE))