Name: dask-tutorial
Owner: dask
Description: Dask tutorial
Created: 2015-07-16 13:56:54.0
Updated: 2018-05-24 21:28:50.0
Pushed: 2018-05-22 19:55:20.0
Homepage: https://dask.pydata.org
Size: 48983
Language: Jupyter Notebook
GitHub Committers
User | Most Recent Commit | # Commits |
---|
Other Committers
User | Most Recent Commit | # Commits |
---|
This tutorial was last given at SciPy 2017 in Austin Texas. A video is available online.
Dask provides multi-core execution on larger-than-memory datasets.
We can think of dask at a high and a low level
threading
or
multiprocessing
libraries in complex cases or other task scheduling
systems like Luigi
or IPython parallel
.Different users operate at different levels but it is useful to understand
both. This tutorial will interleave between high-level use of dask.array
and
dask.dataframe
(even sections) and low-level use of dask graphs and
schedulers (odd sections.)
You should clone this repository
git clone http://github.com/dask/dask-tutorial
and then install necessary packages.
You will need the following core libraries
conda install numpy pandas h5py Pillow matplotlib scipy toolz pytables snakeviz dask distributed
You may find the following libraries helpful for some exercises
pip install graphviz cachey
In the repo directory
conda env create -f environment.yml
and then on osx/linux
source activate dask-tutorial
on windows
activate dask-tutorial
You can build a docker image out of the provided Dockerfile.
Windows users can install graphviz as follows
Alternatively one can use the following conda commands (one installs graphviz and one installs python-bindings for graphviz):
From the repo directory
python prep.py
From the repo directory
jupyter notebook
dask
tag on Stack OverflowOverview - dask's place in the universe
Foundations - low-level Dask and how it does what it does
Bag - the first high-level collection: a generalized iterator for use with a functional programming style and o clean messy data.
Distributed - Dask's scheduler for clusters, with details of how to view the UI.
Array - blocked numpy-like functionality with a collection of numpy arrays spread across your cluster.
Advanced Distributed - further details on distributed computing, including how to debug.
Dataframe - parallelized operations on many pandas dataframes spread across your cluster.
Dataframe Storage - efficient ways to read and write dataframes to disc.