cyverse/magnetosphere

Name: magnetosphere

Owner: CyVerse

Description: A configuration for producing (via Ansible) an Atmosphere development environment

Forked from: lenards/magnetosphere

Created: 2016-02-26 20:17:23.0

Updated: 2016-10-31 21:16:47.0

Pushed: 2017-08-22 15:30:17.0

Homepage: null

Size: 27

Language: Python

GitHub Committers

UserMost Recent Commit# Commits

Other Committers

UserEmailMost Recent Commit# Commits

README

Magnetosphere

Magnetosphere provides a vagrant provisioning approach to creating an Atmosphere development environment.

Table of Contents

(Assuming) Mac

 tap caskroom/cask
 install brew-cask
 cask install virtualbox

 cask install vagrant
using vagrant

Note: depending on your connection's bandwidth this could take 2 to 5 minutes…

grant box add ubuntu/trusty64

box: Loading metadata for box 'ubuntu/trusty64'
box: URL: https://atlas.hashicorp.com/ubuntu/trusty64
box: Adding box 'ubuntu/trusty64' (v20160127.0.0) for provider: virtualbox
box: Downloading: https://vagrantcloud.com/ubuntu/boxes/trusty64/versions/20160127.0.0/providers/virtualbox.box
box: Progress: 1% (Rate: 320k/s, Estimated time remaining: 0:21:01)
... 2 to 10 minutes elapse ...
box: Successfully added box 'ubuntu/trusty64' (v20160127.0.0) for 'virtualbox'!
Get Magnetosphere
clone https://github.com/lenards/magnetosphere.git
Inside MAGNETOSPHERE_HOME

We will stage the missing “bits” that are necessary for installation in the folder where our Vagrantfile is. I will refer to this as MAGNETOSPHERE_HOME when walking through the steps for getting the vagrant VM working.

For the most part, the missing “bits” are all available for the Atmosphere team in the secrets repo. But Clank discusses these files as well.

Our goal is for MAGNETOSPHERE_HOME to looking like this:

ee -L 1 . 

LICENSE
README.md
Vagrantfile
pre-flight-check.sh
kickstart.sh
scratch
??? {{secrets-repo}}
??? clank

We need to:

NOTE: you may want to change MOCK_USERNAME in variables.yml@vagrant

Get Clank
Secrets

… you'll have to ask @lenards or @steve-gregory about that (for now)

Inside Your Vagrant Box

To begin …

grant up 
ging machine 'atmosphere-dev' up with 'virtualbox' provider...
atmosphere-dev: Importing base box 'ubuntu/trusty64'...
atmosphere-dev: Matching MAC address for NAT networking...
atmosphere-dev: Checking if box 'ubuntu/trusty64' is up to date...
atmosphere-dev: A newer version of the box 'ubuntu/trusty64' is available! You currently
atmosphere-dev: have version '20150907.0.0'. The latest is version '20160120.0.1'. Run
atmosphere-dev: `vagrant box update` to update.
atmosphere-dev: Setting the name of the VM: magnetosphere_atmosphere-dev_1453751731307_24339
atmosphere-dev: Clearing any previously set forwarded ports...
atmosphere-dev: Clearing any previously set network interfaces...
atmosphere-dev: Preparing network interfaces based on configuration...
atmosphere-dev: Adapter 1: nat
atmosphere-dev: Adapter 2: hostonly
atmosphere-dev: Forwarding ports...
atmosphere-dev: 22 => 2222 (adapter 1)
atmosphere-dev: Running 'pre-boot' VM customizations...
atmosphere-dev: Booting VM...
atmosphere-dev: Waiting for machine to boot. This may take a few minutes...
atmosphere-dev: SSH address: 127.0.0.1:2222
atmosphere-dev: SSH username: vagrant
atmosphere-dev: SSH auth method: private key
atmosphere-dev: Warning: Connection timeout. Retrying...
atmosphere-dev: 
atmosphere-dev: Vagrant insecure key detected. Vagrant will automatically replace
atmosphere-dev: this with a newly generated keypair for better security.
atmosphere-dev: 
atmosphere-dev: Inserting generated public key within guest...
atmosphere-dev: Removing insecure key from the guest if it's present...
atmosphere-dev: Key inserted! Disconnecting and reconnecting using new SSH key...
atmosphere-dev: Machine booted and ready!
atmosphere-dev: Checking for guest additions in VM...
atmosphere-dev: The guest additions on this VM do not match the installed version of
atmosphere-dev: VirtualBox! In most cases this is fine, but in rare cases it can
atmosphere-dev: prevent things such as shared folders from working properly. If you see
atmosphere-dev: shared folder errors, please make sure the guest additions within the
atmosphere-dev: virtual machine match the version of VirtualBox you have installed on
atmosphere-dev: your host and reload your VM.
atmosphere-dev: 
atmosphere-dev: Guest Additions Version: 4.3.10
atmosphere-dev: VirtualBox Version: 5.0
atmosphere-dev: Configuring and enabling network interfaces...
atmosphere-dev: Mounting shared folders...
atmosphere-dev: /vagrant => /Users/lenards/devel/magnetosphere
atmosphere-dev: Running provisioner: shell...
atmosphere-dev: Running: inline script
....
....
....
atmosphere-dev: Setting up python2.7-dev (2.7.6-8ubuntu0.2) ...
atmosphere-dev: Setting up python-dev (2.7.5-5ubuntu3) ...
atmosphere-dev: Setting up python-distlib (0.1.8-1ubuntu1) ...
atmosphere-dev: Setting up python-distlib-whl (0.1.8-1ubuntu1) ...
atmosphere-dev: Setting up python-html5lib (0.999-3~ubuntu1) ...
atmosphere-dev: Setting up python-html5lib-whl (0.999-3~ubuntu1) ...
atmosphere-dev: Setting up python-six-whl (1.5.2-1ubuntu1) ...
atmosphere-dev: Setting up python-urllib3-whl (1.7.1-1ubuntu4) ...
atmosphere-dev: Setting up python-requests-whl (2.2.1-1ubuntu0.3) ...
atmosphere-dev: Setting up python-setuptools-whl (3.3-1ubuntu2) ...
atmosphere-dev: Setting up python-pip-whl (1.5.4-1ubuntu3) ...
atmosphere-dev: Setting up python-setuptools (3.3-1ubuntu2) ...
atmosphere-dev: Setting up python-pip (1.5.4-1ubuntu3) ...
atmosphere-dev: Setting up python-wheel (0.24.0-1~ubuntu1) ...
atmosphere-dev: Processing triggers for libc-bin (2.19-0ubuntu6.6) ...
atmosphere-dev: Python 2.7.6
atmosphere-dev: pip 1.5.4 from /usr/lib/python2.7/dist-packages (python 2.7)
atmosphere-dev: Downloading/unpacking pip from https://pypi.python.org/packages/py2.py3/p/pip/pip-8.0.3-py2.py3-none-any.whl#md5=b234250205337ff67967dff300001e3d
atmosphere-dev: Downloading/unpacking virtualenv
atmosphere-dev: Installing collected packages: pip, virtualenv
atmosphere-dev:   Found existing installation: pip 1.5.4
atmosphere-dev:     Not uninstalling pip at /usr/lib/python2.7/dist-packages, owned by OS
atmosphere-dev: Successfully installed pip virtualenv
atmosphere-dev: Cleaning up...

You can see a live capture of the vagrant up process if you are curious.

Once that's done, you've got a running virtual machine to log into.

Snapshot - it's quite progressive

Since we are using VirtualBox (VBox) as a provider, we'll use a VBox-specific vagrant plugin to handling saving snapshots.

grant plugin install vagrant-vbox-snapshot 
alling the 'vagrant-vbox-snapshot' plugin. This can take a few minutes...
alled the plugin 'vagrant-vbox-snapshot (0.0.10)'!

The commands are slightly different than what you'll see in the Vagrant docs for vagrant snapshot:

grant snapshot help
e: vagrant snapshot <command> [<args>]

lable subcommands:
 back
 delete
 go
 list
 take

help on any individual command run `vagrant snapshot <command> -h`

Before we log into the box, we should create an initial snapshot that we can restore to later (if needed). The command within the plugin we want is take, we need to provide a name for the snapshot and vm-name (this is defined within the Vagrantfile):

grant snapshot take --help 
 snapshot

e: vagrant snapshot take [vm-name] <SNAPSHOT_NAME>

By default, the Vagrantfile file specifies atmosphere-dev as the vm-name.

grant snapshot take atmosphere-dev INIT
ng snapshot INIT
.10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
shot taken. UUID: 2d60fb56-49ab-4a11-ad15-876c0129cdb1

For more information, see the README for vagrant-vbox-snapshot repository.

Step inside …

Let's log into the vagrant box we've created:

ant ssh

Everything on your local machine (which is the “host”) is accessible within the vagrant (guest) box via /vagrant. We can use this to our advantage when providing data or trying to get files out of the guest to the host.

Run kickstart.sh

(ensure that the pre-flight-check.sh passes prior to this step)

This is going to run the combination of ratchet.py & playbooks in clank.

We're going to use a slight abstraction to run ratchet & clank that mimics how all build & deploys happen (we do this via Jenkins as a shell-script build step, but we're calling that step kickstart.sh).

{{not complete}}

grant ssh 

do su - 
 /vagrant
pre-flight-check.sh

kickstart.sh

«such output; much fact; lvl=ansible»
rvice atmosphere status

rvice nginx restart
rvice atmosphere start

asciicast

Getting the Databases Loaded…

NOTE: this step is something that should be (very soon) done by Clank

You will need to know what the database names are in use. Lucky for us, the information was present in the BUILD_ENV file.

So we can grep for DBNAME and find the names of the databases that Clank created during the kickstart:

@vagrant-ubuntu-trusty-64:/vagrant# grep DBNAME $BUILD_ENV_PATH/variables.yml@vagrant 
_DBNAME: aeolian
O_DBNAME: thermogram

We need to do the next actions as postgres user:

 - postgres
 /vagrant

`ls` or check on your sql-dump paths since we're being all manual

ql -d aeolian < /vagrant/prod-sql-dump-for-atmosphere-release-date.sql
such output ...
ql -d thermogram < /vagrant/prod-sql-dump-for-troposphere-release-date.sql
such output ... 
it 
rvice postgresql restart 
rvice atmosphere restart

You can verify that the API is working as expected by going to the Django REST Framework page for the API (in your host, aka your Mac):

 https://192.168.72.19/api/sizes
After …
Notes
VirtualBox-specific vagrant snapshot

If you hit a failure in ./kickstart.sh, or you are developing extensions to the existing Magnetosphere, you might find these examples useful.

List available snapshots 
agrant snapshot list 
ing snapshots for 'atmosphere-dev':
ame: INIT (UUID: 2d60fb56-49ab-4a11-ad15-876c0129cdb1)
Going back to a previous state

(You will need to include the vm-name to do this)

grant snapshot back -h 
 to current snapshot

e: vagrant snapshot back [vm-name]

If we wanted to move atmosphere-dev back to the previous state:

agrant snapshot back atmosphere-dev
.10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
oring snapshot 28c20f48-58cf-448e-9ab2-6a4083faaee5
ting restored VM
atmosphere-dev: Checking if box 'ubuntu/trusty64' is up to date...
atmosphere-dev: A newer version of the box 'ubuntu/trusty64' is available! You currently
atmosphere-dev: have version '20160127.0.0'. The latest is version '20160222.0.0'. Run
atmosphere-dev: `vagrant box update` to update.
atmosphere-dev: Resuming suspended VM...
atmosphere-dev: Booting VM...
atmosphere-dev: Waiting for machine to boot. This may take a few minutes...
atmosphere-dev: SSH address: 127.0.0.1:2200
atmosphere-dev: SSH username: vagrant
atmosphere-dev: SSH auth method: private key
atmosphere-dev: Machine booted and ready!
Restoring a specific snapshot

We can just go to the previous snapshot with vagrant snapshot back [vm-name].

But we might want to go back to a state prior to just the previous one and we can do that with the following:

ant snapshot go [vm-name] <SNAPSHOT_NAME>
VirtualBox Provider Info …

You might want install Mac specifics for hosting VirtualBox

Note: if you have a Virtualbox VMs running, they must be suspended/halted…

It appears that the current virtualbox cask is including the extension pack. But if for some reason you need to install it, there is a brew cask:

 cask install virtualbox-extension-pack
This work is supported by the National Institutes of Health's National Center for Advancing Translational Sciences, Grant Number U24TR002306. This work is solely the responsibility of the creators and does not necessarily represent the official views of the National Institutes of Health.