Name: scrapers-us-municipal
Owner: datamade
Description: Scrapers for US municipal governments.
Forked from: opencivicdata/scrapers-us-municipal
Created: 2017-06-30 15:11:55.0
Updated: 2018-05-24 19:25:12.0
Pushed: 2018-05-24 19:27:38.0
Size: 1923
Language: Python
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
DataMade's source for municipal scrapers.
To find out more about the ins-and-outs of these scrapers, as well as how to create your own, head on over to docs.opencivicdata.org's scraping page.
Issues with the data coming from these scrapers should be filed at the OCD Data issue tracker
All Open Civic Data issues can be browsed and filed at the Open Civic Data JIRA instance.
Requires python3, postgresql
Assuming that you want to have your database be called opencivicdata on your local machine
install -r requirements.txt
tedb opencivicdata
rt DATABASE_URL=postgresql:///opencivicdata
dbinit us
Initializing the database may take some time - but if it above goes as expected, then you should see something like this:
ations to perform:
ply all migrations: contenttypes, core, legislative, pupa
ing migrations:
plying contenttypes.0001_initial... OK
plying contenttypes.0002_remove_content_type_name... OK
plying core.0001_initial... OK
plying legislative.0001_initial... OK
plying legislative.0002_more_extras... OK
plying legislative.0003_time_changes... OK
plying pupa.0001_initial... OK
plying pupa.0002_auto_20150906_1458... OK
plying pupa.0003_auto_20151118_0408... OK
plying pupa.0004_identifier... OK
plying pupa.0005_auto_20170522_1935... OK
plying pupa.0006_identifier_jurisdiction... OK
84 divisions found in the CSV, and 0 already in the DB
Finally, initialize your new scraper (if you so desire):
init YOUR_CITY_SCRAPER
Your database expects the postgis extension. Do you have this? If not, running pupa dbinit us may throw an error:
go.db.utils.ProgrammingError: permission denied to create extension "postgis"
: Must be superuser to create this extension.
Create this extension:
-d opencivicdata
TE EXTENSION postgis
At times, the release of ocd-django on PyPI differs from that of Github. This may cause problems if you need to create and run migrations. Specifically, you might encounter an ImproperlyConfigured error that instructs you to do the following:
must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.
Fix the problem by running:
rt DJANGO_SETTINGS_MODULE=pupa.settings
Then, you should be able to successfully run:
go-admin makemigrations
go-admin migrate
Before submitting a PR, please run pupa update YOUR_CITY_SCRAPER
rt DATABASE_URL=postgresql:///opencivicdata
update YOUR_CITY_SCRAPER
We want changes in this repo to all be rebased off opencivicdata/scrapers-us-municipal.
To achieve this, first pull from origin of the fork.
pull origin master
Then, make your changes and commit them locally. Next, rebase your changes onto upstream master.
pull upstream master
ash
push origin master