Name: dcos-bot-branches
Owner: DC/OS
Description: Mesosphere-MergeBot branches for testing out changes to components
Created: 2016-10-27 22:33:05.0
Updated: 2017-05-12 07:28:53.0
Pushed: 2017-10-05 22:01:24.0
Homepage: https://github.com/mesosphere-mergebot
Size: 4310
Language: Python
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
The easiest way to run microservices, big data, and containers in production.
Like traditional operating systems, DC/OS is system software that manages computer hardware and software resources and provides common services for computer programs.
Unlike traditional operating systems, DC/OS spans multiple machines within a network, aggregating their resources to maximize utilization by distributed applications.
To learn more, see the DC/OS Overview.
DC/OS releases are publicly available on http://dcos.io/releases/
Release artifacts are managed by Mesosphere on Amazon S3, using a CloudFront cache.
To find the git SHA of any given release, check the latest commit in the versioned branches on GitHub: https://github.com/dcos/dcos/branches/
| Release Type | URL Pattern |
|————–|——————–|
| Latest Stable| https://downloads.dcos.io/dcos/stable/dcos_generate_config.sh |
| Specific Stable Release | https://downloads.dcos.io/dcos/stable/commit/<git-sha>/dcos_generate_config.sh |
| Latest Early Access | https://downloads.dcos.io/dcos/EarlyAccess/dcos_generate_config.sh |
| Specific Early Access Release | https://downloads.dcos.io/dcos/EarlyAccess/commit/<git-sha>/dcos_generate_config.sh |
| Latest Master | https://downloads.dcos.io/dcos/testing/master/dcos_generate_config.sh |
| Specific PR, Latest Build | https://downloads.dcos.io/dcos/testing/pull/<github-pr-number>/dcos_generate_config.sh |
Linux is required for building and testing DC/OS.
tar needs to be GNU tar for the set of flags useddocker run alpine /bin/echo 'Hello, World!' when run at a new terminal as your user should just print "Hello, World!". If it says something like “Unable to find image 'alpine:latest' locally” then re-run and the message should go away.sudo pacman -S pythonsudo apt-get install make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev xz-utils liblzma-devpyenv install 3.5.2pyenv virtualenv 3.5.2 dcospyenv activate dcossudo dnf install pxzUnit tests can be run locally but require the development environment specified above.
Tox is used to run the codebase unit tests, as well as coding standard checks. The config is in tox.ini.
Integration tests can be run on any deployed DC/OS cluster. For installation instructions, see https://dcos.io/install/.
Integration tests are installed via the dcos-integration-test Pkgpanda package.
SSH into a master node The tests can be run via Pytest while SSH'd as root into a master node of the cluster to be tested.
Switch to root
su -
Add the test user
ce /opt/mesosphere/environment.export
on /opt/mesosphere/active/dcos-oauth/bin/dcos_add_user.py albert@bekstil.net
This test user has a known login token with far future expiration. DO NOT USE IN PRODUCTION!
After the test, remember to delete the test user. For more information, see User Management.
Configure the tests
ce /opt/mesosphere/active/dcos-integration-test/test_env.export
rt SLAVE_HOSTS=<PRIVATE-AGENT-IP-1>,<PRIVATE-AGENT-IP-2>
rt PUBLIC_SLAVE_HOSTS=<PUBLIC-AGENT-IP-1>,<PUBLIC-AGENT-IP-2>
The test_env.export script tries to look up cluster metadata, but can't distinguish between public and private nodes yet. So those have to be manually specified.
Run the tests with Pytest
opt/mesosphere/active/dcos-integration-test
est
DC/OS can be built locally but requires the development environment specified above.
DC/OS builds are packaged as a self-extracting Docker image wrapped in a bash script called dcos_generate_config.sh.
WARNING: Building a release from scratch the first time on a modern dev machine (4 cores / 8 hyper threads, SSD, reasonable internet bandwidth) takes about 1 hour.
ild_local.sh
That will run a simple local build, and output the resulting DC/OS installers to $HOME/dcos-artifacts. You can run the created `dcos_generate_config.sh like so:
OME/dcos-artifacts/testing/`whoami`/dcos_generate_config.sh
If you look inside of the bash script build_local.sh there are the commands with descriptions of each.
The general flow is to:
release tool configuration if one doesn't existrelease toolThese steps can all be done by hand and customized / tweaked like standard python projects. You can hand create a virtualenvironment, and then do an editable pip install (pip install -e) to have a “live” working environment (as you change code you can run the tool and see the results).
This release tool always loads the config in dcos-release.config.yaml in the current directory.
The config is YAML. Inside it has two main sections. storage which contains a dictionary of different storage providers which the built artifacts should be sent to, and options which sets general DC/OS build configuration options.
Config values can either be specified directly, or you may use $ prefixed environment variables (the env variable must set the whole value).
All the available storage providers are in release/storage. The configuration is a dictionary of a reference name for the storage provider (local, aws, my_azure), to the configuration.
Each storage provider (ex: aws.py) is an available kind prefix. The dictionary factories defines the suffix for a particular kind. For instance kind: aws_s3 would map to the S3StorageProvider.
The configuration options for a storage provider are the storage provider's constructor parameters.
Sample config storage that will save to my home directory (/home/cmaloney):
age:
cal:
kind: local_path
path: /home/cmaloney/dcos-artifacts
Sample config that will store to a local archive path as wll as AWS S3. Environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY would need to be set to use the config (And something like a CI system could provide them so they don't have to be committed to a code repository).
age:
s:
kind: aws_s3
bucket: downloads.dcos.io
object_prefix: dcos
download_url: https://downloads.dcos.io/dcos/
access_key_id: $AWS_ACCESS_KEY_ID
secret_access_key: $AWS_SECRET_ACCESS_KEY
region_name: us-west-2
cal:
kind: local_path
path: /mnt/big_artifact_store/dcos/
Pull requests automatically trigger a new build and several tests.
Most of the triggered tests are required for merge, but some are optional (usually ones on flaky infrastructure).
tox in the top-level dir to run all syntax checks as well as pytest (unit-tests). See tox.ini for more details--web option and then controlled entirely by the HTTP APIDC/OS itself is composed of many individual components precisely configured to work together in concert.
This repo contains the release and package building tools necessary to produce installers for various on-premises and cloud platforms.
| Directory | Contents | | ——— | ——– | | cloud_images | Base OS image building tools | config | Release configuration | docs | Documentation | flake8_dcos_lint | Flake8 plugin for testing code quality | dcos_installer | Backend for Web, SSH, and some bits of the Advanced installer. Code is being cleaned up | gen | Python library for rendering yaml config files for various platforms into packages, with utilities to do things like make “late binding” config set by CloudFormation | packages | Packages which make up DC/OS (Mesos, Marathon, AdminRouter, etc). These packages are built by pkgpanda, and combined into a “bootstrap” tarball for deployment. | pkgpanda | DC/OS baseline/host package management system. Tools for building, deploying, upgrading, and bundling packages together which live on the root filesystem of a machine / underneath Mesos. | release | Release tools for DC/OS. (Building releases, building installers for releases, promoting between channels) | ssh | AsyncIO based parallel ssh library used by the installer | test_util | various scripts, utilities to help with integration testing