ropensci/mocker

Name: mocker

Owner: rOpenSci

Description: Cache API calls to the web

Created: 2013-10-27 17:18:58.0

Updated: 2017-05-28 05:08:41.0

Pushed: 2015-05-11 16:18:09.0

Homepage: null

Size: 159

Language: R

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

mocker

Install mocker

all.packages("devtools")
ary(devtools)
all_github("ropensci/mocker")
ary(mocker)

Install redis

using brew (OSX)

or their site for other options.

 install redis

Remember to start Redis on your cli

s-server 

A redis admin panel in browser is available from the Node.js library redis-commander

 npm install -g redis-commander
s-commander

Install CouchDB

using brew (OSX)

 install couchdb

Remember to start CouchDB on your cli

hdb 

A CouchDB admin panel in your browser, Futon, comes with CouchDB. You can access it in your browser at localhost:5984/_utils. A new dashboard replacement called Fauxton, from Cloudant, can be installed from NPM, and is found at localhost:5984/_utils/fauxton/.

install fauxton

Examples

Define query

 "theory"

Local storage via saveRDS

This is the default, 1st run with cache=TRUE same as cache=FALSE, then 2nd time faster. Same for any backend option.

em.time( cachefxn(q=q, cache=TRUE, path="~/scottscache/") )

ser  system elapsed 
096   0.004   0.453 

offee
em.time( cachefxn(q=q, cache=TRUE, path="~/scottscache/") )

offee
er  system elapsed 
004   0.000   0.005

Local storage via R.cache

em.time( cachefxn(q=q, cache=TRUE, backend="rcache") )

offee
ser  system elapsed 
033   0.002   0.279 

offee
em.time( cachefxn(q=q, cache=TRUE, backend="rcache") )

offee
ser  system elapsed 
006   0.000   0.006 

Redis

NOTE: startup redis in your shell first

em.time( cachefxn(q=q, cache=TRUE, backend="redis") )

offee
ser  system elapsed 
036   0.004   0.384 

offee
em.time( cachefxn(q=q, cache=TRUE, backend="redis") )

offee
ser  system elapsed 
007   0.001   0.007 

SQLite

ary(filehashSQLite)
eate("foodb", type = "SQLite") # or skip if database already created
b <- dbInit("foodb", type = "SQLite")
em.time( cachefxn(q=q, cache=TRUE, backend="sqlite", db=sqldb) )

offee
ser  system elapsed 
038   0.002   0.293 

offee
em.time( cachefxn(q=q, cache=TRUE, backend="sqlite", db=sqldb) )

offee
ser  system elapsed 
014   0.000   0.014

CouchDB

NOTE: startup couchdb in your shell first

_createdb("cachecall")
<- "cachecall"
em.time( cachefxn(q=q, cache=TRUE, backend="couchdb", db=cdb) )

offee
ser  system elapsed 
032   0.001   0.315 

offee
em.time( cachefxn(q=q, cache=TRUE, backend="couchdb", db=cdb) )

offee
er  system elapsed 
025   0.001   0.028 

All methods

With microbenchmark

ary(microbenchmark)
obenchmark(
al=cachefxn(q=q, cache=TRUE, path="~/scottscache/"),
ache=cachefxn(q=q, cache=TRUE, backend="rcache"),
is=cachefxn(q=q, cache=TRUE, backend="redis"),
ite=cachefxn(q=q, cache=TRUE, backend="sqlite", db=sqldb),
chdb=cachefxn(q=q, cache=TRUE, backend="couchdb", db=cdb),
es=50

offee
: milliseconds
expr       min        lq    median        uq       max neval
ocal  4.007978  4.278275  4.362870  4.816612  6.675667    50
cache  4.461892  4.824427  5.038775  5.801503  8.543470    50
edis  5.624845  6.146504  6.401435  7.075442  9.408585    50
lite 10.074079 10.652784 11.210765 12.425844 18.450480    50
chdb 25.964903 27.661443 29.219574 32.668773 36.355845    50

Indeed, local caching is fastest, either via the simplest of writing files via saveRDS (“local”), or via R.cache. However, Redis isn't far behind, and could offer more flexibility over simple file storage if you are interested. SQLite and CouchDB probably aren't worth it unless you really need them.

Explanation of the cachefxn function

i.e., how you could incorporate this into a package or a script

Here's the function inside this package that is like one we would use to make a web API call, with explanation. The two additional arguments needed beyond whatever is already in a fxn are cache and backend.

efxn <- function(q="*:*", db=NULL, cache=FALSE, backend='local', path)

get api query ready
l = "http://api.plos.org/search"
gs <- list(q=q, fl='id,author,abstract', fq='doc_type:full', wt='json', limit=50)

create a key
chekey <- make_key(url, args)

if cache=TRUE, check for data in backend using key, if cache=FALSE, returns NULL
t <- suppressWarnings(cache_get(cache, cachekey, backend, path, db=db))

if out=NULL, proceed to make call to web
(!is.null(out)){ out } else
 
tt <- GET(url, query=args)
stop_for_status(tt)
temp <- content(tt, as="text")
# If cache=TRUE, cache key and value in chosen backend
cache_save(cache, cachekey, temp, backend, path, db=db)
return( temp )