newsdev/campfin-loader

Name: campfin-loader

Owner: NYT Newsroom Developers

Description: A new data loader for campaign finance that is totally separate from the front end

Created: 2017-06-26 15:11:20.0

Updated: 2018-04-06 23:16:08.0

Pushed: 2017-07-17 20:35:00.0

Homepage: null

Size: 22

Language: Python

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

campfin-loader

A new data loader for campaign finance that is totally separate from the front end

This module uses nyt-pyfec to parse .fec files and dumps the results in a postgres database. It doesn't stray far from the FEC's raw data, but it does mark filings as amended or covered by a periodic filing (and marks newest=False in those cases).

To query valid transactions, include WHERE newest=true in any queries. If you want totals that are close to itemized totals reported by the FEC, also include where memo_entry=false

quickstart

1. Use python3
2. Install postgres
3. Create a postgres database
4. If using virtualenvwrapper (which is STRONGLY RECOMMENDED): mkvirutalenv campfin-loader --python $(which python3)
5. pip install -r requirements.txt
6. Add your database credentials as environment varibales DB_NAME, DB_HOST, DB_USER and DB_PASSWORD (the latter defaults to None). Recommended: do this in $VIRTUAL_ENV/bin/postactivate if you're using virtualenvwrapper
7. Get an FEC API key. (Or submit a pull request to scrape the page!)
8. Use the methods in fec_scrape.py. The easiest way to scrape local filings right now is to run python -c 'from fec_scrape import load_recent_filings; load_recent_filings()'. Commandline interface hopefully coming soon.