Name: stock-scraper
Owner: The Public Knowledge Workshop
Description: Python scraping commands for stock market data
Created: 2016-02-22 17:44:01.0
Updated: 2017-08-03 09:30:29.0
Pushed: 2016-02-22 19:56:31.0
Homepage: null
Size: 11
Language: Python
GitHub Committers
User | Most Recent Commit | # Commits |
---|
Other Committers
User | Most Recent Commit | # Commits |
---|
A collection of commands allowing extraction of Israeli stock market data. The data is scraped from the http://maya.tase.co.il/bursa website.
In order to run the commands you need to setup the db and tool.
Data scraped:
$ pip install -r requirements.txt
.The database will keep the scraped data.
Create a db and update bursa/settings.py
accordingly. For example:
BASE = {
ass': 'MySQLDatabase',
ame': 'bursa',
ost': '',
ser': 'root',
assword': ''
Create the tables by running $ python manage.py
Since scrapers use web browsers, a browser should be made available on the machine running the scrape commands. Currently, the scraper is hardcoded to use Firefox so make sure it's installed either as the standard version or headless.
Basically, each type of company data has it's own scraper. Some data is static (e.g. company name) and some is time based (e.g market cap).
Here are the available commands:
python scrape.py info
- extracts all companies and they're static information. Companies already in the db are ignored.python scrape.py market_cap
- extracts market cap data for each company in the db.python scrape.py stakeholder
- extracts stakeholder data for each company in the db.python scrape.py management
- extracts management data for each company in the db.python scrape.py financial_report
- extracts financial report data for each company in the db.