Name: congressional-data-django
Owner: SFBrigade
Description: null
Created: 2018-04-14 23:49:38.0
Updated: 2018-05-09 19:04:04.0
Pushed: 2018-05-09 19:04:02.0
Homepage: null
Size: 193
Language: Python
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
Welcome to the Congressional Data Django README file. This is a living breathing document so don't be shy and feel free to contribute documentation!
Who are you?
This guide will walk you through getting your local environment stood up.
In order to stand up a local postgres instance first you have to install several dependencies.
First log in as the super user so you don't have to sudo everything
su
Next install the following packages
get update
get install python-pip python-dev libpq-dev postgresql postgresql-contrib
Optional: configure a start-up on boot
te-rc.d postgresql enable
update
install python postgresql
coming soon…
if you are using Ubuntu on Windows, please see Ubuntu Users
Start up the service!
ice postgresql start
Switch from super user to the postgres user
postgres
Last, make sure your pg_hba.conf is set up correctly
/etc/postgresql/<postgres version>/main/pg_hba.conf
Make sure that the following line appears
PE DATABASE USER ADDRESS METHOD
l all all md5
The default values may be the postgres user for the database and user as well as “peer” for the method. Make sure that the database is set to all, the users are set to all, and the method is set to md5 and not peer.
Restart to apply configuration changes
ice postgresql restart
tl -D /usr/local/var/postgres start && brew services start postgresql
Then check the running version.
gres -V
postgres # You may need to run this with sudo
This should start the postgres cli tool. (You can tell if your console prepend says, “postgres=“. Let's first use it to check out our users.
You should see a Superuser role created automatically. Let's create a service account user as well as change the settings to what Django expects. Remember: end every sql statement with a semicolon in the CLI tool.
TE USER <username> WITH LOGIN PASSWORD '<password>';
R ROLE <username> SET client_encoding TO 'utf8';
R ROLE <username> SET default_transaction_isolation TO 'read committed';
R ROLE <username> SET timezone TO 'UTC';
Replace the angle brackets with your desired username/password combination, eg foo/bar. This is the account you will use for your local Django settings.
Optionally, at this point you can give your service account user permissions to create databases.
R ROLE <username> CREATEDB;
Lets create a database! If you want to create it with your service account user then exit the CLI tool (type \q) and re-enter using
postgres -U <username>
Now that we are logged in, lets create our local database and grant all privileges to our service account user.
TE DATABASE congressionaldata;
T ALL PRIVILEGES ON DATABASE congressionaldata TO <username>;
At this point you should have your local postgres set up.
Note: To copy data from a CSV file to a postgres database or copy data from a postgres database to a CSV file, you need to be logged into postgres through an account that has a Superuser role.
The examples below will demonstrate how to create a schema and table for the average donations per candidate per election cycle. Any text within angle brackets (<>) may be replaced with whichever names necessary for your data.
For most, just entering the following command will start your postgres cli tool in Superuser mode.
postgres
Connect to the Database To add a schema and tables to a database, you must first be connected to that database.
nect <congressionaldata>
Next, we need to add a schema to the database. Create Schema
TE SCHEMA <donations>;
You will see CREATE SCHEMA printed if the schema is successfully created.
Create a Table in the Schema Just Created This table will be different for different data sets.
TE TABLE donations.avg_per_candidate(
EXT NOT NULL,
tacc TEXT NOT NULL,
y_name TEXT NOT NULL,
tion_year TEXT NOT NULL,
est_name TEXT NOT NULL,
l_votes REAL NOT NULL,
mbent_flag CHAR(1) NOT NULL,
inner BOOLEAN NOT NULL,
age_amount_per_contrib REAL NOT NULL,
trans INT NOT NULL,
l_contrib REAL NOT NULL);
You will see CREATE TABLE printed if the table is successfully created.
Copy Data from a CSV File
<donations.avg_per_candidate> from
ur/file/path>/congressional-data-django/avg_donation_received_by_the_candidate.csv'
HEADER;
You will see COPY [number of records] printed if the file is successfully copied.
Test New Database To test that the data from the CSV file was saved to the database, you can perform any sql statement on the newly added table. An example is below.
CT * FROM <donations.avg_per_candidate>;
Note: If you are unable to access the schema created from a different postgres user, you will need to grant the user permission from your Superuser account.
T USAGE ON SCHEMA <donations> TO <username>;
congresionaldata/settings/local.py
This file contains the settings specific to your local environment. Notice the DATABASES settings are extracting data from a local settings file (which should not exist if you have just cloned the repo).
BASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': os.environ['LOCAL_POSTGRES_DATABASE_NAME'],
'USER': os.environ['LOCAL_POSTGRES_USER'],
'PASSWORD': os.environ['LOCAL_POSTGRES_PASSWORD'],
'HOST': 'localhost',
'PORT': '',
}
Make sure to create the directory and file described below.
secrets/local_settings.py
This file is ignored by git and contains the actual credentials which will be used to authenticate your local postgres.
rt os
nviron['LOCAL_POSTGRES_DATABASE_NAME'] = 'congressionaldata'
nviron['LOCAL_POSTGRES_USER'] = '<username>'
nviron['LOCAL_POSTGRES_PASSWORD'] = '<password>'
t('Local Secrets Loaded!')
It is recommended to contain the print statement at the bottom of your local secrets file for debugging purposes.
If you are using Djanga manage to run your local server you can use the settings flag to choose which environment to run.
nage.py runserver --settings=congressionaldata.settings.local
If you are using your IDE to run the server then make sure you have the following environment variable set in your run configuration.
GO_SETTINGS_MODULE=congressionaldata.settings.local
If this is the first time you have ran the Django application against your local database you are going to have to perform a Django migration.
First run the makemigrations command to generate the necessary migration files.
nage.py makemigrations
Next run the migrate command (and make sure the settings is pointed at your local environment) to sync the existing Django model with your database tables.
nage.py migrate --settings=congressionaldata.settings.local
Once the migration has been completed you will have to create a super user to authenticate in the Django admin page.
nage.py createsuperuser --settings=congressionaldata.settings.local
Follow the steps output by this command.
Production secrets such as special keys, tokens, and credentials cannot be stored in the project repository. Travis allows us to encrypt all of our secrets using public key unique to each secret. That secret will only be able to be decrypted using the private key stored on travis.
Before you install the travis cli tool, first make sure that you have updated your Ruby and Gem versions. There are some known issues with the travis cli tool with ruby versions < 2.5
Once you have a recent version of Ruby and Gem, install travis
install travis -v 1.8.8 --no-rdoc --no-ri
First thing you need to do is authenticate the travis cli tool so you can have access to your travis account.
is login
Now that you have installed travis, and authenticated, you can begin storing secrets.
Suppose you want to store the key value pair SOMEVAR=“secretvalue”
Travis will generate an RSA keypair exposing the public key at https://api.travis-ci.org/repos/somevar/key. The private key will remain secret known only to travis. The cli tool will use this new public key to encrypt the key value pair. The –add flag will immediately add this encrypted keyvalue pair to the travis.yml file.
is encrypt SOMEVAR="secretvalue" --add -r <organization>/<repository>
This will store the encrypted secret on travis
rt SOMEVAR="secretvalue"