Name: routing-release
Owner: Cloud Foundry
Description: This is BOSH release for cloud foundry routers
Created: 2015-06-03 18:01:54.0
Updated: 2018-05-23 22:39:49.0
Pushed: 2018-05-23 22:39:47.0
Homepage: null
Size: 1840
Language: Shell
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
This repo is a BOSH release that delivers HTTP and TCP routing for Cloud Foundry.
For help or questions with this release or any of its submodules, you can reach the maintainers on Slack at cloudfoundry.slack.com in the #routing channel.
When working on individual components of TCP Router, work out of the submodules
under src/.
Run the individual component unit tests as you work on them using ginkgo.
Commits to this repo (including Pull Requests) should be made on the Develop branch.
Fetch release repo.
r -p ~/workspace
/workspace
clone https://github.com/cloudfoundry/routing-release.git
outing-release/
Automate $GOPATH and $PATH setup.
This BOSH release doubles as a $GOPATH. It will automatically be set up
for you if you have direnv installed.
nv allow
If you do not wish to use direnv, you can simply source the .envrc file
in the root of the release repo. You may manually need to update your
$GOPATH and $PATH variables as you switch in and out of the directory.
Initialize and sync submodules.
ripts/update
Install ginkgo
nstall github.com/onsi/ginkgo/ginkgo
Run unit tests
ripts/run-unit-tests
For high availability, configure a load balancer in front of the routers; for more information see High Availability. If high-availability is not required allocate a public IP to a single instance of each router. See Port Requirements for TCP Routing.
If you are using a load balancer see Configuring Load Balancer Healthcheck.
Choose domain names from which developers will configure HTTP and TCP routes for their applications. Separate domain names will be required for HTTP and TCP routing. Configure DNS to resolve these domain names to the load balancer in front of the routers. You may use the same or separate load balancers for the HTTP and TCP domains. If high-availability is not required configure DNS to resolve the domains directly to a single instance of the routers.
If your manifest is configured with self-signed certificates for UAA, configure routing components to skip validation of the TLS certificate; see Validation of TLS Certificates from Route Services and UAA.
Deploy Cloud Foundry using the instructions for cf-deployment.
Choose how many TCP routes you'd like to offer. For each TCP route, a port must be opened on your load balancer. Configure your load balancer to forward the range of ports you choose to the IPs of the TCP Router instances.
Routing API must be configured with the same range of ports. By default cf-deployment will enable 100 ports in the range 1024-1123. If you choose a range other than 1024-1123, you must configure this using the deployment manifest property
routing-api.router_groups.reservable_ports. This is a seeded value only;
after deploy, changes to this property will be ignored. To modify the
reservable port range after deployment, use the Routing API;
(see “To update a Router Group's reservable_ports field with a new port
range”).
me: routing_api
operties:
routing_api:
router_groups:
- name: default-tcp
reservable_ports: 1024-1123
type: tcp
- name: default-tcp
reservable_ports: [1066,1266]
type: tcp
- name: default-tcp
reservable_ports: 1111-2222,4444
type: tcp
The following components communicate with UAA via TLS:
Additionally, gorouter communicates with Route Services via TLS.
In all cases, these components will validate that certs are signed by a known CA and that the cert is for the requested domain. To disable this validation, as when deploying the routing subsystem to an environment with self-signed certs, configure the following property for Gorouter, routing-api and tcp-router.
me: routing-api
operties:
skip_ssl_validation: true
me: tcp_router
operties:
skip_ssl_validation: true
me: gorouter
operties:
router:
ssl_skip_validation: true
After deploying this release you must add the domain you chose (see Domain Names) to CF as a Shared Domain (admin only), associating it with the Router Group.
The CLI commands below require version 6.17+ of the cf CLI, and must be run as admin.
List available router-groups
cf router-groups
tting router groups as admin ...
me type
fault-tcp tcp
Note: If you receive this error: FAILED This command requires the
Routing API. Your targeted endpoint reports it is not enabled. This is due
to the CF CLI's ~/.cf/config.json having an old cached RoutingEndpoint
value. To fix this, just do a cf login again and this error should go away.
Create a shared-domain for the TCP router group
cf create-shared-domain tcp-apps-domain.com --router-group default-tcp
eating shared domain tcp-apps-domain.com as admin...
See Router Groups for details on that concept.
As ports can be a limited resource in some environments, the default quotas in Cloud Foundry for IaaS other than BOSH Lite do not allow reservation of route ports; required for creation of TCP routes. The final step to enabling TCP routing is to modify quotas to set the maximum number of TCP routes that may be created by each organization or space.
Determine whether your default org quota allows TCP Routes:
cf quota default
tting quota default info as admin...
tal Memory 10G
stance Memory unlimited
utes -1
rvices 100
id service plans allowed
p instance limit unlimited
served Route Ports 0
If Reserved Route Ports is greater than zero, you can skip the following step, as this attribute determines how many TCP routes can be created within each organization assigned this quota.
If Reserved Route Ports is zero, you can update the quota with the following command:
cf update-quota default --reserved-route-ports 2
dating quota default as admin...
$ cf quota default
tting quota default info as admin...
tal Memory 10G
stance Memory unlimited
utes -1
rvices 100
id service plans allowed
p instance limit unlimited
served Route Ports 2
Configuring Reserved Route Ports to -1 sets the quota attribute to unlimited. For more information on configuring quotas for TCP Routing, see Enabling TCP Routing.
Before running the acceptance tests errand, make sure to have the following setup.
properties.acceptance_tests.include_http_routes in
your manifest for the errand job.Run the following commands to execute the acceptance tests as an errand on bosh-lite
run errand routing_acceptance_tests
See the README for Routing Acceptance Tests
The simplest way to test TCP Routing is by pushing your app. By specifying
the TCP domain and including the --random-route option, a TCP route will
be created with a reserved port and the route mapped to your app.
$ cf p myapp -d tcp-apps-domain.com --random-route
Send a request to your app using the TCP shared domain and the port reserved for your route.
rl tcp-apps-domain.com:60073
On a BOSH deployed environment, you would configure DNS to resolve the TCP shared domain to a load balancer in front of the TCP Routers. However BOSH Lite does not expose ports used by TCP Router. To test TCP Routing against a local BOSH Lite, you can use its curl its IP directly using the route port (assumes route port is 60073):
rl 10.244.14.2:60073
Or you could add an entry to /etc/hosts and curl the domain and port:
44.14.2 tcp-apps-domain.com
To test this against a remote BOSH Lite, you must ssh into a job (like the TCP Router itself) and curl the IP of the router and the route port.
For step by step instructions on TCP router demo done at Cloud Foundry Summit 2016, refer to TCP Router demo
If you use the canonical manifest provided with cf-deployment the following prerequisites will be configured for you automatically.
UAA must be configured to terminate TLS for internal requests. Set the
following properties in your environment stub for cf-release when using the
manifest generation scripts, or set it directly in your manifest. The
routing-release's manifest generation scripts will set uaa.tls_port to the
value of uaa.ssl.port from the cf-release manifest.
me: uaa
operties:
uaa:
ssl:
port: <choose a port for UAA to listen to SSL on; e.g. 8443>
sslCertificate: |
<insert certificate>
sslPrivateKey: |
<insert private key>
You must add the routing.router_groups.read and
routing.router_groups.write scopes to your admin user.
me: uaa
operties:
uaa:
scim:
users:
- name: admin
password: PASSWORD
groups:
- scim.write
- scim.read
- openid
- cloud_controller.admin
- clients.read
- clients.write
- doppler.firehose
- routing.router_groups.read
- routing.router_groups.write
The following OAuth clients must be configured for UAA. All but the cf
client are new; the important change to the cf client is adding the
routing.router_groups.read and routing.router_groups.write scopes. If
you're using the manifest generation scripts for cf-release, you can skip
this step as the necessary clients are in the Spiff templates. If you're
handrolling your manifest for cf-release, you'll need to add them.
me: uaa
operties:
uaa:
clients:
cc_routing:
authorities: routing.router_groups.read
authorized-grant-types: client_credentials
secret: <your-secret>
cf:
override: true
authorized-grant-types: password,refresh_token
scope: cloud_controller.read,cloud_controller.write,openid,password.write,cloud_controller.admin,cloud_controller.admin_read_only,scim.read,scim.write,doppler.firehose,uaa.user,routing.router_groups.read,routing.router_groups.write
authorities: uaa.none
access-token-validity: 600
refresh-token-validity: 2592000
gorouter:
authorities: routing.routes.read
authorized-grant-types: client_credentials,refresh_token
secret: <your-secret>
tcp_emitter:
authorities: routing.routes.write,routing.routes.read
authorized-grant-types: client_credentials,refresh_token
secret: <your-secret>
tcp_router:
authorities: routing.routes.read
authorized-grant-types: client_credentials,refresh_token
secret: <your-secret>
UAA must be configured to accept requests using an internal hostname. The
manifest generation scripts for cf-release will do this for you (both BOSH
Lite and non). However, if you override the uaa.zones.internal.hostnames
property yourself, be sure to include uaa.service.cf.internal in your
stub.
name: uaa properties: uaa:
internal_url: https://uaa.service.cf.internal:8443
ca_cert: "((uaa_ca.certificate))"
Routing API requires a relational database as a data store; MySQL and PostgreSQL are supported. Routing API does not create the database; one must exist on startup. cf-deployment uses CF MySQL Release by default. For any deployment you can use this or provide your own MySQL or PostgreSQL database. Configure the credentials for the database with the following properties.
name: routing-api properties: routing_api: sqldb:
type: <mysql || postgres>
host: <IP of SQL Host>
port: <Port for SQL Host>
schema: <Schema name>
username: <Username for SQL DB>
password: <Password for SQL DB>
ou are using [cf-mysql-release](https://github.com/cloudfoundry/cf-mysql-release), then the values for these properties can be obtained from the following properties in the manifest for that release.
ype` should be `mysql`
ost` corresponds to the IP address of the `proxy_z1` job
ort` is `3306`
chema` corresponds to `cf_mysql.mysql.seeded_databases[].name`
sername` corresponds to `cf_mysql.mysql.seeded_databases[].username`
assword` corresponds to `cf_mysql.mysql.seeded_databases[].password`
If you use the CF MySQL Release, you can seed the required database on deploy using the manifest property cf_mysql.mysql.seeded_databases. Do not use the same deployment of cf-mysql-release that is exposed as a CF marketplace service. Instead, use a deployment intended for internal platform use; for these deployments you should set broker instances to zero. Update the following manifest properties before deploying.
name: database properties: cf_mysql:
mysql:
seeded_databases:
- name: routing-api
username: <your-username>
password: <your-password>
The Diego Route Emitter component must be configured to emit TCP routes in order to support TCP routing.
name: route_emitter
properties:
tcp:
enabled: true
A Router Group represents a horizontally scalable cluster of identically configured routers. Only one router group is currently supported. Shared domains in Cloud Foundry are associated with one router group; see Post Deploy Configuration. To create a TCP route for their application, a developer creates it from a TCP domain; see Create a TCP Route. For each TCP route, Cloud Foundry reserves a port on the CF router. Each port is dedicated to that route; route ports may not be shared by multiple routes. The number of ports available for reservation dictates how many TCP routes can be created.
A router group is limited to maximum port range 1024-65535; defaulting to
1024-1123. As one router group is supported, the maximum number of TCP routes
than can be created in CF is 64512 (65535-1024). Though multiple Shared Domains
can be assigned to the router group, they share a common pool of ports. E.g.
given Shared Domains tcp-1.example.com and tcp-2.example.com are assigned to
thedefault-tcprouter group, and a route for port 1024 is created from
domaintcp-1.example.com, the same port could not be reserved for a route
from domaintcp-2.example.com`. Eventually we may support multiple router
groups and/or TCP routes that share a port.
The same reservable port range must be configured both on the load balancer,
and in this release using the manifest property
routing-api.router_groups.reservable_ports. The port range can be modified
after deploying using the Routing API.
Note: when modifying the port range using the Routing API, consider that the new port range must include those ports that have already been reserved.
The TCP Router and Routing API are stateless and horizontally scalable. The TCP Routers must be fronted by a load balancer for high-availability. The Routing API depends on a database, that can be clustered for high-availability. For high availability, deploy multiple instances of each job, distributed across regions of your infrastructure.
For details refer to Routing API.
For documentation on metrics available for streaming from Routing components through the Loggregator Firehose, visit the CloudFoundry Documentation. You can use the NOAA Firehose sample app to quickly consume metrics from the Firehose.
Steps for enabling PROXY Protocol on the GoRouter can be found here.