Name: diffengine
Owner: Documenting the Now
Description: track changes to the news, where news is anything with an RSS feed
Created: 2017-01-03 02:45:48.0
Updated: 2017-12-20 22:21:16.0
Pushed: 2017-11-08 15:35:55.0
Size: 166
Language: Python
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
diffengine is a utility for watching RSS feeds to see when story content changes. When new content is found a snapshot is saved at the Internet Archive, and a diff is generated for sending to social media. The hope is that it can help draw attention to the way news is being shaped on the web. It also creates a database of changes over time that can be useful for research purposes.
diffengine draws heavily on the inspiration of NYTDiff and NewsDiffs which almost did what we wanted. NYTdiff is able to create presentable diff images and tweet them, but was designed to work specifically with the NYTimes API. NewsDiffs provides a comprehensive framework for watching changes on multiple sites (Washington Post, New York Times, CNN, BBC, etc) but you need to be a programmer to add a parser module for a website that you want to monitor. It is also a full-on website which involves some commitment to install and run.
With the help of feedparser, diffengine takes a different approach by working with any site that publishes an RSS feed of changes. This covers many news organizations, but also personal blogs and organizational websites that put out regular updates. And with the readability module, diffengine is able to automatically extract the primary content of pages, without requiring special parsing to remove boilerplate material. And like NYTDiff, instead of creating another website for people to watch, diffengine pushes updates out to social media where people are already, while also building a local database of diffs that can be used for research purposes.
In order to run diffengine you need to pick a directory location where you can store the diffengine configuration, database and diffs. For example I have a directory in my home directory, but you can use whatever location you want, you just need to be able to write to it.
The first time you run diffengine it will prompt you to enter an RSS or Atom feed URL to monitor and will authenticate with Twitter.
% diffengine /home/ed/.diffengine
What RSS/Atom feed would you like to monitor? https://inkdroid.org/feed.xml
Would you like to set up tweeting edits? [Y/n] Y
Go to https://apps.twitter.com and create an application.
What is the consumer key? <TWITTER_APP_KEY>
What is the consumer secret? <TWITTER_APP_SECRET>
Log in to https://twitter.com as the user you want to tweet as and hit enter.
Visit https://api.twitter.com/oauth/authorize?oauth_token=NRW9BQAAAAAAyqBnAAXXYYlCL8g
What is your PIN: 8675309
Saved your configuration in /home/ed/.diffengine/config.yaml
Fetching initial set of entries.
Done!
After that you just need to put diffengine in your crontab to have it run regularly, or you can run it manually at your own intervals if you want. Here's my crontab to run every 30 minutes to look for new content.
0,30 * * * * /usr/local/bin/diffengine /home/ed/.diffengine
You can examine your config file at any time and add/remove feeds as needed. It
is the config.yaml file that is stored relative to the storage directory you
chose, so in my case /home/ed/.diffengine/config.yaml.
Here is a list of known diffengine Twitter accounts. If you start one please send us a pull request or create an issue so it can be added here. For convenience Ryan Baumann's “diffengine” Twitter list aggregates many of these accounts.
If you are setting multiple accounts, and multiple feeds if may be helpful to setup a directory for each account. For example:
/home/nruest/.torontosun/home/nruest/.torontostar/home/nruest/.globemail/home/nruest/.canadaland/home/nruest/.cbcThen you will configure a cron entry for each account:
,30,45 * * * * /usr/bin/flock -xn /tmp/globemail.lock -c "/usr/local/bin/diffengine /home/nruest/.globemail"
,30,45 * * * * /usr/bin/flock -xn /tmp/torontosun.lock -c "/usr/local/bin/diffengine /home/nruest/.torontosun"
,30,45 * * * * /usr/bin/flock -xn /tmp/cbc.lock -c "/usr/local/bin/diffengine /home/nruest/.cbc"
,30,45 * * * * /usr/bin/flock -xn /tmp/lapresse.lock -c "/usr/local/bin/diffengine /home/nruest/.lapresse"
,30,45 * * * * /usr/bin/flock -xn /tmp/calgaryherald.lock -c "/usr/local/bin/diffengine /home/nruest/.calgaryherald"
If there are multiple feeds for an account, you can setup the config.yml like so:
me: The Globe and Mail - Report on Business
itter:
access_token: ACCESS_TOKEN
access_token_secret: ACCESS_TOKEN_SECRET
l: http://www.theglobeandmail.com/report-on-business/?service=rss
me: The Globe and Mail - Opinion
itter:
access_token: ACCESS_TOKEN
access_token_secret: ACCESS_TOKEN_SECRET
l: http://www.theglobeandmail.com/opinion/?service=rss
me: The Globe and Mail - News
itter:
access_token: ACCESS_TOKEN
access_token_secret: ACCESS_TOKEN_SECRET
l: http://www.theglobeandmail.com/news/?service=rss
tomjs: phantomjs
ter:
nsumer_key: CONSUMER_KEY
nsumer_secret: CONSUMER_SECRET
Here's how to get started hacking on diffengine with pyenv:
env virtualenv 3.6.0 diffengine
env activate diffengine
t clone https://github.com/docnow/diffengine
diffengine
p install -r requirements
test
========================= test session starts ==============================
form linux -- Python 3.5.2, pytest-3.0.5, py-1.4.32, pluggy-0.4.0
dir: /home/ed/Projects/diffengine, inifile:
ected 5 items
_diffengine.py .....
======================= 5 passed in 8.09 seconds ===========================