Name: muse
Owner: funcool
Description: Clojure(Script) library that makes remote data access code elegant and efficient
Created: 2015-11-03 11:13:23.0
Updated: 2017-06-03 22:44:55.0
Pushed: 2015-12-04 10:53:35.0
Size: 74
Language: Clojure
GitHub Committers
User | Most Recent Commit | # Commits |
---|
Other Committers
User | Most Recent Commit | # Commits |
---|
Muse is a Clojure library that works hard to make your relationship with remote data simple & enjoyable. We believe that concurrent code can be elegant and efficient at the same time.
Oftentimes, your business logic relies on remote data that you need to fetch from different sources: databases, caches, web services or 3rd party APIs, and you can't mess things up. Muse helps you to keep your business logic clear of low-level details while performing efficiently:
Having all this gives you the ability to access remote data sources in a concise and consistent way, while the library handles batching and overlapping requests to multiple data sources behind the scenes.
Heavily inspired by:
Talks:
A core problem of many systems is balancing expressiveness against performance.
uire '[clojure.set :refer [union intersection]])
n num-common-friends
y]
ount (intersection (friends-of x) (friends-of y))))
Here, (friends-of x)
and (friends-of y)
are independent, and you want it to be fetched concurrently in a single batch. Furthermore, if x
and y
refer to the same person, you don't want to redundantly re-fetch their friend list.
Muse allows your data fetches to be implicitly concurrent:
uire '[muse.core :as muse])
n num-common-friends [x y]
use/fmap (comp count intersection) (friends-of x) (friends-of y)))
e/run! (num-common-friends 1 2))
Attention! API is subject to change
Include the following to your lein project.clj
dependencies:
e "0.4.0"]
All functions are located in muse.core
:
uire '[muse.core :as muse])
Simple helper to emulate async request to the remote source with unpredictable response latency:
In ClojureScript:
uire '[promesa.core :as prom])
n remote-req [id result]
rom/promise
(fn [resolve reject]
(let [wait (rand 1000)]
(println "-->" id ".." wait)
(js/setTimeout #(do (println "<--" id)
(resolve result))
wait)))))
In Clojure:
uire '[promesa.core :as prom])
n remote-req [id result]
rom/promise
(fn [resolve reject]
(let [wait (rand 1000)]
(println "-->" id ".." wait)
(Thread/sleep wait)
(println "<--" id)
(resolve result)))))
Define data source (list of friends by given user id):
uire '[muse.core :as muse])
record FriendsOf [id]
se/DataSource
etch [_] (remote-req id (set (range id)))))
n friends-of [id]
riendsOf. id))
Run simplest scenario:
ends-of 10)
> #core.FriendsOf{:id 10}
e/run! (friends-of 10))
-> 10 .. 877.3953983155727
-- 10
> #<Promise {:status :pending}>
ef (muse/run! (friends-of 10)))
-> 10 .. 412.97080768100585
-- 10
> #{0 7 1 4 6 3 2 9 5 8}
e/run!! (friends-of 10)) ;; blocks until done
-> 10 .. 834.4564727277141
-- 10
> #{0 7 1 4 6 3 2 9 5 8}
There is nothing special about it (yet), let's do something more interesting:
e/fmap count (friends-of 10))
> #<MuseMap (clojure.core$count@1b932280 core.FriendsOf[10])>
e/run!! (muse/fmap count (friends-of 10)))
-> 10 .. 844.5086574753595
-- 10
> 10
e/fmap inc (muse/fmap count (friends-of 3)))
> #<MuseMap (clojure.core$comp$fn__4192@4275ef0b core.FriendsOf[3])>
e/run!! (muse/fmap inc (muse/fmap count (friends-of 3))))
-> 3 .. 334.5374146247876
-- 3
> 4
Let's imagine we have another data source: users' activity score by given user id.
record ActivityScore [id]
se/DataSource
etch [_] (remote-req id (inc id))))
Nested data fetches (you can see 2 levels of execution):
n first-friend-activity []
>> (friends-of 10)
(muse/fmap sort)
(muse/fmap first)
(muse/flat-map #(ActivityScore. %))))
e/run!! (first-friend-activity))
-> 10 .. 576.5833162596521
-- 10
-> 0 .. 275.28637368204966
-- 0
> 1
And now a few amazing facts.
uire '[clojure.set :refer [union intersection]])
n num-common-friends [x y]
use/fmap (comp count intersection) (friends-of x) (friends-of y)))
1) muse
automatically runs fetches concurrently:
e/run!! (num-common-friends 3 4))
-> 3 .. 50.56579162982433
-> 4 .. 247.60281831534402
-- 3
-- 4
> 3
2) muse
detects duplicated requests and caches results to avoid redundant work:
e/run!! (num-common-friends 5 5))
-> 5 .. 781.2024344113081
-- 5
> 5
3) seq operations will also run concurrently:
n friends-of-friends [id]
>> (friends-of id)
(muse/traverse #(friends-of %))
(muse/fmap (partial apply union))))
e/run!! (friends-of-friends 5))
-> 5 .. 972.1322804759812
-- 5
-> 0 .. 498.6426390505534
-> 1 .. 136.49940971567355
-> 4 .. 874.777296180928
-- 1
-> 3 .. 910.0740298270428
-- 0
-> 2 .. 995.5441177163739
-- 4
-- 3
-- 2
> #{0 1 3 2}
4) you can implement BatchedSource
protocol to tell muse
how to batch requests:
record FriendsOf [id]
se/DataSource
etch [_] (remote-req id (set (range id))))
se/BatchedSource
etch-multi [_ users]
(let [ids (cons id (map :id users))]
(->> ids
(map #(vector %1 (set (range %1))))
(into {})
(remote-req ids)))))
e/run!! (friends-of-friends 5))
-> 5 .. 783.7984574012655
-- 5
-> (0 1 4 3 2) .. 420.575997272024
-- (0 1 4 3 2)
> #{0 1 3 2}
If you come from Haskell you will probably like shortcuts:
e/<$> inc (muse/<$> count (friends-of 3)))
> #<MuseMap (clojure.core$comp$fn__4192@6f2c4a58 core.FriendsOf[3])>
e/run!! (muse/<$> inc (muse/<$> count (friends-of 3))))
> 4
Custom response cache id:
record Timeline [username]
se/DataSource
etch [_] (remote-req username (str username "'s timeline ")))
se/LabeledSource
esource-id [_] username))
e/fmap count (Timeline. "@kachayev"))
> #<MuseMap (clojure.core$count@1b932280 core.Timeline[@kachayev])>
e/run!! (muse/fmap count (Timeline. "@kachayev")))
-> @kachayev .. 929.3864355882571
-- @kachayev
> 21
e/run!! (muse/fmap str (Timeline. "@kachayev") (Timeline. "@kachayev")))
@kachayev .. 809.035607308747
@kachayev
chayev's timeline @kachayev's timeline "
Thorough documentation coming soon.
Muse
can be used from ClojureScript code with few minor differences:
run!!
macro isn't provided (as we don't have blocking experience)LabeledSource
protocol (return pair [resource-name id]
)MuseAST
monad is compatible with cats
library, so you can use mlet/return
interface as well as fmap
& bind
functions provided by cats.core
:
uire '[cats.core :as m])
record Post [id]
se/DataSource
etch [_] (remote-req id {:id id :author-id (inc id) :title "Muse"})))
record User [id]
se/DataSource
etch [_] (remote-req id {:id id :name "Alexey"})))
n get-post [id]
/mlet [post (Post. id)
user (User. (:author-id post))]
(m/return (assoc post :author user))))
e/run!! (get-post 10))
-> 10 .. 813.5857785197163
-- 10
-> 11 .. 449.5124897284112
-- 11
> {:author {:id 11, :name "Alexey"}, :id 10, :author-id 11, :title "Muse"}
HTTP calls:
uire '[muse.core :as muse])
uire '[promesa.core :as prom])
n async-get [url]
rom/future (slurp url)))
record Gist [id]
se/DataSource
etch [_] (async-get (str "https://gist.github.com/" id))))
e/run!! (muse/fmap count (Gist. "21e7fe149bc5ae0bd878")))
> 86085
n gist [id]
use/fmap count (Gist. id)))
ill fetch 2 gists concurrently
e/run!! (muse/fmap compare (gist "21e7fe149bc5ae0bd878") (gist "b5887f66e2985a21a466")))
> 1
For an example of the use with database queries, see a detailed example here: “Solving the N+1 Selects Problem with Muse”).
You define data sources that you want to work with using DataSource
protocol (describe how fetch
should be executed).
You declare what do you want to do with the result of each data source fetch. Yeah, right, your data source is a functor now.
You build an AST of all operations placing data source fetching points as leaves using muse
low-level building blocks (value
/fmap
/flat-map
) and higher-level API (collect
/traverse
/etc). Read more about free monads approach.
muse
implicitly rebuilds AST to work with tree levels instead of separate leaves that gives ability to batch requests and run independent fetches concurrently.
muse/run!
is an interpreter that reduces AST level by level until the whole computation is finished (it returns a promise that you can read from).
java.util.concurrent.CompletableFuture
promesa
library only (if you use other async mechanism, like future
s you can easily turn your code to be compatible with promises)run!
call (in case it's impossible you should probably look into other ways to solve your problem, i.e. data stream libraries)Release under the MIT license. See LICENSE for the full license.
feature-*
branch to start making your changes.or simply…
Thanks go to Simon Marlow for creating/leading Haxl project (and talking about it). And to Facebook for open-sourcing it.