Python as the Zen of Data Science
Travis E. Oliphant, Ph.D.
Peter Wang
Continuum Analytics

Data Science
2

禪 Zen
3

道
4

Zen Approach
• Right Practice
• Right Attitude
• Right Understanding
“The Zen way of calligraphy is to write in the most straightforward,
simple way as if you were a beginner not trying to make something
skillful or beautiful, but simple writing with the full attention as if
you were discovering what you were writing for the first time.”
— Zen Mind, Beginner’s Mind
5

“Right Understanding”
The purpose of studying Buddhism is not to study Buddhism
Zen: but to study ourselves. You are not your body.
Python: The purpose of writing Python code is not to just
produce software, but to study ourselves. You are
not your technology stack!
6

Pythonic Approach
• Compose language primitives, built-ins, classes whenever
possible
• Much more powerful and accessible than trying to memorize a
huge list of proprietary functions
• Reject artificial distinctions between where data “should live” and
where it gets computed
• Empower each individual to use their own knowledge, instead of
taking design power out of their hands with pre-ordained
architectures and “stacks”.
7

Why Python?
Analyst
Analyst / Data
Programmer
Developer
• Uses graphical tools
• Builds simple apps & workflows
• Creates frameworks
• Can call functions,
• Used to be "just an analyst"
& compilers
cut & paste code
• Likes coding to solve problems
• Uses IDEs
• Can change some
• Doesn't want to be a "full-time
• Degree in CompSci
variables
programmer"
• Knows multiple
languages
Gets paid for:
Gets paid (like a rock star) for:
Gets paid for:
Insight
Code that produces insight
Code
Excel, VB, Tableau,
SAS, R, Matlab,
C, C++, Java, JS,
Python
Python
Python
8

Pythonic Data Analysis
• Make an immediate connection with the data (using Pandas+ / NumPy+ /
scikit-learn / Bokeh and matplotlib)
• PyData stack means agility which al ows you to rapidly build
understanding with powerful modeling and easy manipulation.
• Let the Data drive the analysis not the technology stack.
• Empower the data-scientist, quant, geophysicist, biochemist, directly with
simple language constructs they can use that “fits their brain.”
• Scale-out is a later step. Python can work with you there too. There are
great tools no matter where your data is located or how it is stored or how
your cluster is managed. Not tied to a particular distributed story.
9

Zen of Python
>>> import this
The Zen of Python, by Tim Peters
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
10

Zen of NumPy (and Pandas)
• Strided is better than scattered
• Contiguous is better than strided
• Descriptive is better than imperative (use data-types)
• Array-oriented and data-oriented is often better than object-oriented
• Broadcasting is a great idea – use where possible
• Split-apply-combine is a great idea – use where possible
• Vectorized is better than an explicit loop
• Write more ufuncs and generalized ufuncs (numba can help)
• Unless it’s complicated — then use numba
• Think in higher dimensions
11

“Zen of Data Science”
• Get More and better data.
• Better data is determined by better models.
• How you compute matters.
• Put the data in the hands and minds of people with knowledge.
• Fail quickly and often — but not in the same way.
• Where and how the data is stored is secondary to analysis and
understanding.
• Premature horizontal scaling is the root of al evil.
• When you must scale — data locality and paral el algorithms are the key.
• Learn to think in building blocks that can be paral elized.
12

PyData Stack -- about 3,000,000 users
... many many more ...
OpenCV
astropy
PySAL
BioPython
GDAL
PyTables/Numexpr
Dask / Blaze
SymPy
Numba
scikit-image
statsmodels
scikit-learn
Cython
SciPy
pandas
Matplotlib
Bokeh
NumPy
13

How do we get 100s of dependencies delivered?
We started Continuum with 3 primary goals:
1. Scale the NumPy/PyData stack horizontal y
2. Make it easy for Python users to produce data-science applications in the browser/
notebook
3. Get more adoption of the PyData stack
So, we wrote a package manager, conda.
and a distribution of Python + R, Anaconda.
14

Game-Changing
Enterprise Ready
Python Distribution
• 2 mil ion downloads in last 2 years
• 200k / month and growing
• conda package manager serves up 5 mil ion
packages per month
• Recommended instal er for IPython/Jupyter,
Pandas, SciPy, Scikit-learn, etc.
15

Some Users
16

Anaconda — Portable Environments
PYTHON & R OPEN SOURCE ANALYTICS
NumPy
SciPy
Pandas
Scikit-learn
Jupyter/ IPython
• Easy to instal
Numba
Matplotlib
Spyder
Numexpr
Cython
Theano
• Quick & agile data exploration
• Powerful data analysis
Scikit-image
NLTK
NetworkX
IRKernel
dplyr
shiny
• Simple to col aborate
ggplot2
tidyr
caret
nnet
And 330+ packages
• Accessible to al
conda
17

Traditional Analytics
Data
Ad hoc usage
and/or BI
Data
report
Data Mining
ETL
and/or
Model ing
Production
Data
deployment
Data

Traditional Analytics
Data
ANACONDA
Ad hoc usage
and/or BI
Data
report
Data Mining
ETL
and/or
Model ing
Production
Data
deployment
Data

Package Managers
Linux
OSX
Windows
yum (rpm)
macports
chocolatey
apt-get (dpkg)
homebrew
npackd
fink
Cross-platform
conda
Sophisticated light-weight
environments included!
http://conda.pydata.org
20

Conda features
• Excel ent support for “system-level” environments — like having mini VMs but much
lighter weight than docker (micro containers)
• Minimizes code-copies (uses hard/soft links if possible)
• Simple format: binary tar-bal + metadata
• Metadata al ows static analysis of dependencies
• Easy to create multiple “channels” which are repositories for packages
• User instal able (no root privileges needed)
• Integrates very wel with pip and other language-specific package managers.
• Cross Platform
21

Basic Conda Usage
Instal a package
conda install sympy
List al instal ed packages
conda list
Search for packages
conda search llvm
Create a new environment
conda create -n py3k python=3
Remove a package
conda remove nose
Get help
conda install --help
22

Advanced Conda Usage
Instal a package in an environment
conda install -n py3k sympy
Update al packages
conda update --all
Export list of packages
conda list --export packages.txt
Instal packages from an export
conda install --file packages.txt
See package history
conda list --revisions
Revert to a revision
conda install --revision 23
Remove unused packages and cached tarbal s
conda clean -pt
23

Environments
• Environments are simple: just link the package to a different directory
• Hard-links are very cheap, and very fast — even on Windows.
• Conda environments are completely independent instal ations of
everything
conda create -n py3k python=3.5
• No fiddling with PYTHONPATH or sym-linking site-packages
• “Activating” an environment just means changing your PATH so
that its bin/ or Scripts/ comes first.
• Unix:
source activate py3k
• Windows:
activate py3k
24

25

Anaconda Platform Analytics Repository
• Commercial long-term support
• Private, on-premise package mirror
• Proprietary tools for building custom
distribution, like Anaconda
• Enterprise tools for managing custom
packages and environments
• Available on the cloud at
http://anaconda.org
26

Anaconda Cluster: Anaconda + Hadoop + Spark
For Data Scientists:
• Rapidly, easily create clusters on EC2, DigitalOcean, on-prem cloud/provisioner
• Manage Python, R, Java, JS packages across the cluster
For Operations & IT:
•
Robustly manage runtime state across the cluster
• Outside the scope of rpm, chef, puppet, etc.
•
Isolate/sandbox packages & libraries for different jobs or groups of users
• Without introducing complexity of Docker / virtualization
•
Cross platform - same tooling for laptops, workstations, servers, clusters
27

Cluster Creation
$ acluster create mycluster —profile=spark_profile
$ acluster submit mycluster mycode.py
spark_profile:
aws_east:
provider: aws_east
secret_id: <aws_access_key_id>
secret_key: <aws_secret_access_key>
num_nodes: 4
keyname: id_rsa.pub
node_id: ami-3c994355
location: us-east-1
node_type: m1.large
private_key: ~/.ssh/id_rsa
cloud_provider: ec2
security_group: all-open
$ acluster destroy mycluster
http://continuumio.github.io/conda-cluster/quickstart.html
28

Cluster Management
Package & environment management:
$ acluster manage mycluster list
... info -e
... install python=3 pandas flask
... set_env
... push_env <local> <remote>
Easy SSH & remote commands:
$ acluster ssh mycluster
$ acluster run.cmd mycluster "cat /etc/hosts"
29
http://continuumio.github.io/conda-cluster/manage.html

Anaconda Cluster & Spark
# example.py
conf = SparkConf()
conf.setMaster("yarn-client")
conf.setAppName("MY APP")
sc = SparkContext(conf=conf)
# analysis
sc.parallelize(range(1000)).map(lambda x: (x, x % 2)).take(10)
$ acluster submit MY_CLUSTER /path/to/example.py
Remember that Blaze has a higher-level interface to Spark and Dask provides
a more Pythonic approach.
30

ANACONDA OPEN SOURCE
TECHNOLOGY
31

Blaze
Blaze is an extensible high-level interface for data
analytics. It feels like NumPy/Pandas. It drives other
data systems. Blaze expressions enable high-level
reasoning. It’s an ecosystem of tools.
• Infrastructure for meta-data, meta-compute, and expression graphs/dataflow
• Data glue for scale-up or scale-out
• Generic remote computation & query system
• (NumPy+Pandas+LINQ+OLAP+PADL).mashup()
http://blaze.pydata.org
32

Glue 2.0
Python’s legacy as a powerful
glue language
• manipulate files
• call fast libraries
Next-gen Glue:
• Link data silos
• Link disjoint memory &
compute
• Unify disparate runtime
models
• Transcend legacy models of
computers
33

34

Data
35

Data
“Math”
36

Big Data
Math
37

Big Data
Math
38

Big Data
Math
39

Big Data
Programs
Math
40

“General Purpose Programming”
41

Domain-Specific
Query Language
Analytics System
42

html,css,js,…
py,r,sql,…
java,c,cpp,cs
43

?
44

Expressions
Me
tadata
untime
R
45

+ - / * ^ []
join, groupby, filter
map, sort, take
where, topk
d
,

a
R
h
ta
d
sh sh
ark, , ...
p
f
ro 5
a
a
,
t

p
p
ndas,
o
jso e
e
L, Sp ndra
b
, ,
u
n
st d
, Pa
f
,
,

ri
t
SQ

csv
yp
assa
a
d
vro
e
e
,
,

umPy
xl
,
N
lia, K,
. s
.
ngo, C
.
Ju Mo
46

APIs, syntax, language
Expressions
blaze
datashape
metadata
compute
Data
Runtime
storage/containers
paral elize
optimize, JIT
dask
odo
47

Blaze
Interface to query data on different storage systems http://blaze.pydata.org/en/latest/
from blaze import Data
iris = Data('iris.csv')
CSV
SQL
iris = Data('sqlite:///flowers.db::iris')
iris = Data('mongodb://localhost/mydb::iris')
MongoDB
iris = Data('iris.json')
JSON
S3
iris = Data('s3://blaze-data/iris.csv')
…
Current focus is the “dark data” and pydata stack for run-time (dask, dynd,
numpy, pandas, x-ray, etc.) + customer needs (i.e. kdb, mongo).
48

Blaze
Select columns
iris[['sepal_length', 'species']]
Filter iris[(iris.species == 'Iris-setosa') & (iris.sepal_length > 5.0)]
Operate log(iris.sepal_length * 10)
Reduce iris.sepal_length.mean()
Split-apply by(iris.species, shortest=iris.petal_length.min(),
longest=iris.petal_length.max(),
-combine average=iris.petal_length.mean())
transform(iris, sepal_ratio = iris.sepal_length /
Add new iris.sepal_width, petal_ratio = iris.petal_length /
columns
iris.petal_width)
iris.relabel(petal_length='PETAL-LENGTH',
Relabel columns
petal_width='PETAL-WIDTH')
Text matching
iris.like(species='*versicolor')
49

blaze
datashape
Blaze uses datashape as its type system (like DyND)
>>> iris = Data('iris.json')
>>> iris.dshape
dshape("""var * {
petal_length: float64,
petal_width: float64,
sepal_length: float64,
sepal_width: float64,
species: string
}""")
50

Datashape
A structured data description language
http://datashape.pydata.org/
unit types
ordered struct dtype
datashape
dimension
dtype
*
record
col ection of types keyed by labels
var
int32
*
{ x : int32, y : string, z : float64 }
3
string
*
tabular datashape
4
4
float64
var * { x : int32, y : string, z : float64 }
*
51

{
Datashape
flowersdb: {
iris: var * {
petal_length: float64,
petal_width: float64,
sepal_length: float64,
sepal_width: float64,
# Arrays
species: string
3 * 4 * int32
}
3 * 4 * int32
},
# Arrays of Structures
iriscsv: var * {
10 * var * float64
100 * {
sepal_length: ?float64,
3 * complex[float64]
name: string,
sepal_width: ?float64,
birthday: date,
petal_length: ?float64,
# Structure of Arrays
address: {
petal_width: ?float64,
{
street: string,
species: ?string
x: 100 * 100 * float32,
city: string,
},
y: 100 * 100 * float32,
postalcode: string,
irisjson: var * {
u: 100 * 100 * float32,
country: string
petal_length: float64,
v: 100 * 100 * float32,
}
petal_width: float64,
}
}
sepal_length: float64,
sepal_width: float64,
species: string
},
# Function prototype
irismongo: 150 * {
(3 * int32, float64) -> 3 * float64
petal_length: float64,
petal_width: float64,
# Function prototype with broadcasting dimensions
sepal_length: float64,
(A... * int32, A... * int32) -> A... * int32
sepal_width: float64,
species: string
}
52
}

Blaze Server — Lights up your Dark Data
Builds off of Blaze uniform interface
server.yaml
to host data remotely through a JSON
iriscsv:
source: iris.csv
web API.
irisdb:
source: sqlite:///flowers.db::iris
$ blaze-server server.yaml -e
irisjson:
localhost:6363/compute.json
source: iris.json
dshape: "var * {name: string, amount: float64}"
irismongo:
source: mongodb://localhost/mydb::iris
53

Blaze Server
Blaze Client
>>> from blaze import Data
>>> s = Data('blaze://localhost:6363')
>>> t.fields
[u'iriscsv', u'irisdb', u'irisjson', u’irismongo']
>>> t.iriscsv
sepal_length sepal_width petal_length petal_width species
0 5.1 3.5 1.4 0.2 Iris-setosa
1 4.9 3.0 1.4 0.2 Iris-setosa
2 4.7 3.2 1.3 0.2 Iris-setosa
>>> t.irisdb
petal_length petal_width sepal_length sepal_width species
0 1.4 0.2 5.1 3.5 Iris-setosa
1 1.4 0.2 4.9 3.0 Iris-setosa
54
2 1.3 0.2 4.7 3.2 Iris-setosa

Compute recipes work with existing libraries and have multiple
backends
• python list
• numpy arrays
• dynd
• pandas DataFrame
• Spark, Impala
• Mongo
• dask
55

• Ideal y, you can layer expressions over any data
• Write once, deploy anywhere
• Practical y, expressions wil work better on specific data
structures, formats, and engines
• Use odo to copy from one format and/or engine to another
56

Dask: Out-of-Core PyData
• A paral el computing framework
• That leverages the excel ent Python ecosystem
• Using blocked algorithms and task scheduling
• Written in pure Python
Core Ideas
• Dynamic task scheduling yields sane paral elism
• Simple library to enable paral elism
• Dask.array/dataframe to encapsulate the functionality
• Distributed scheduler
57

Example: Ocean Temp Data
• http://www.esrl.noaa.gov/psd/data/gridded/
data.noaa.oisst.v2.highres.html
• Every 1/4 degree, 720x1440 array each day
58

Bigger data...
36 years: 720 x 1440 x 12341 x 4 = 51 GB uncompressed
If you don't have this much RAM. .
. . better start chunking.
59

DAG of Computation
60

•
Col ections build task graphs
•
Schedulers execute task graphs
•
Graph specification = uniting interface
•
A generalization of RDDs
61

Simple Architecture for Scaling
Dask col ections
• dask.array
• dask.dataframe
• dask.bag
• dask.imperative*
Dask Graph Specification
Python Ecosystem
Dask Schedulers
62

dask.array: OOC, paral el, ND array
Arithmetic: +, *, ...
Reductions: mean, max, ...
Slicing: x[10:, 100:50:-2]
Fancy indexing: x[:, [3, 1, 2]]
Some linear algebra: tensordot, qr, svd
Parallel algorithms (approximate quantiles, topk, ...)
Slightly overlapping arrays
Integration with HDF5
63

Dask Array
dask
numpy
>>> import dask.array as da
>>> import numpy as np
>>> da_ones = da.ones((5000000, 1000000),
>>> np_ones = np.ones((5000, 1000))
chunks=(1000, 1000))
>>> da_ones.compute()
>>> np_ones
array([[ 1., 1., 1., ..., 1., 1., 1.],
array([[ 1., 1., 1., ..., 1., 1., 1.],
[ 1., 1., 1., ..., 1., 1., 1.],
[ 1., 1., 1., ..., 1., 1., 1.],
[ 1., 1., 1., ..., 1., 1., 1.],
[ 1., 1., 1., ..., 1., 1., 1.],
...,
...,
[ 1., 1., 1., ..., 1., 1., 1.],
[ 1., 1., 1., ..., 1., 1., 1.],
[ 1., 1., 1., ..., 1., 1., 1.],
[ 1., 1., 1., ..., 1., 1., 1.],
[ 1., 1., 1., ..., 1., 1., 1.]])
[ 1., 1., 1., ..., 1., 1., 1.]])
>>> da_y = da.log(da_ones + 1)[:5].sum(axis=1)
>>> np_y = np.log(np_ones + 1)[:5].sum(axis=1)
>>> np_da_y = np.array(da_y) #fits in memory
>>> np_y
array([ 693.14718056, 693.14718056, 693.14718056,
array([ 693.14718056, 693.14718056, 693.14718056,
693.14718056, …, 693.14718056])
693.14718056, 693.14718056])
# If result doesn’t fit in memory
64
>>> da_y.to_hdf5('myfile.hdf5', 'result')

dask.dataframe: OOC, paral el, ND array
Elementwise operations: df.x + df.y
Row-wise selections: df[df.x > 0]
Aggregations: df.x.max()
groupby-aggregate: df.groupby(df.x).y.max()
Value counts: df.x.value_counts()
Drop duplicates: df.x.drop_duplicates()
Join on index: dd.merge(df1, df2, left_index=True,
right_index=True)
65

Dask Dataframe
pandas
dask
>>> import dask.dataframe as dd
>>> import pandas as pd
>>> ddf = dd.read_csv('*.csv')
>>> df = pd.read_csv('iris.csv')
>>> ddf.head()
>>> df.head()
sepal_length sepal_width petal_length petal_width species
0 5.1 3.5 1.4 0.2 Iris-setosa
sepal_length sepal_width petal_length petal_width species
1 4.9 3.0 1.4 0.2 Iris-setosa
0 5.1 3.5 1.4 0.2 Iris-setosa
2 4.7 3.2 1.3 0.2 Iris-setosa
1 4.9 3.0 1.4 0.2 Iris-setosa
3 4.6 3.1 1.5 0.2 Iris-setosa
2 4.7 3.2 1.3 0.2 Iris-setosa
4 5.0 3.6 1.4 0.2 Iris-setosa
3 4.6 3.1 1.5 0.2 Iris-setosa
…
4 5.0 3.6 1.4 0.2 Iris-setosa
>>> max_sepal_length_setosa = df[df.species ==
>>> d_max_sepal_length_setosa = ddf[ddf.species ==
'setosa'].sepal_length.max()
'setosa'].sepal_length.max()
5.7999999999999998
>>> d_max_sepal_length_setosa.compute()
5.7999999999999998
66

More Complex Graphs
cross validation
67

http://continuum.io/blog/xray-dask
68

from dask import dataframe as dd
columns = ["name", "amenity", "Longitude", "Latitude"]
data = dd.read_csv('POIWorld.csv', usecols=columns)
with_name = data[data.name.notnull()]
with_amenity = data[data.amenity.notnull()]
is_starbucks = with_name.name.str.contains('[Ss]tarbucks')
is_dunkin = with_name.name.str.contains('[Dd]unkin')
starbucks = with_name[is_starbucks]
dunkin = with_name[is_dunkin]
locs = dd.compute(starbucks.Longitude,
starbucks.Latitude,
dunkin.Longitude,
dunkin.Latitude)
# extract arrays of values fro the series:
lon_s, lat_s, lon_d, lat_d = [loc.values for loc in locs]
%matplotlib inline
import matplotlib.pyplot as plt
from mpl_toolkits.basemap import Basemap
def draw_USA():
"""initialize a basemap centered on the continental USA"""
plt.figure(figsize=(14, 10))
return Basemap(projection='lcc', resolution='l',
llcrnrlon=-119, urcrnrlon=-64,
llcrnrlat=22, urcrnrlat=49,
lat_1=33, lat_2=45, lon_0=-95,
area_thresh=10000)
m = draw_USA()

# Draw map background
m.fillcontinents(color='white', lake_color='#eeeeee')
m.drawstates(color='lightgray')
m.drawcoastlines(color='lightgray')
m.drawcountries(color='lightgray')
m.drawmapboundary(fill_color='#eeeeee')
# Plot the values in Starbucks Green and Dunkin Donuts Orange
style = dict(s=5, marker='o', alpha=0.5, zorder=2)
m.scatter(lon_s, lat_s, latlon=True,
label="Starbucks", color='#00592D', **style)
m.scatter(lon_d, lat_d, latlon=True,
label="Dunkin' Donuts", color='#FC772A', **style)
plt.legend(loc='lower left', frameon=False);
69

Distributed
Pythonic Multiple-machine Parallelism that understands Dask graphs
1) Defines Center (dcenter) and Worker (dworker)
2) Simplified setup with dcluster for example —
dcluster 192.168.0.{1,2,3,4}
or
Center
dcluster —hostfile hostfile.txt
3) Create Executor objects like
concurrent.futures (Python 3) or
futures (Python 2.7 back-port)
4) Data locality supported with ad-hoc task graphs
by returning futures wherever possible
New Libray but stabilizing quickly — communicate with blaze-dev@continuum.io
70

Python and Hadoop
(without the JVM)
Chat: http://gitter.im/blaze/dask
Join the conversation!
Email: blaze- ‐dev@continuum.io
71

HDFS without Java
1. HDFS splits large files into many small blocks replicated on many
datanodes
2. For efficient computation we must use data directly on datanodes
3. distributed.hdfs queries the locations of the individual blocks
4. distributed executes functions directly on those blocks on the
datanodes
5. distributed+pandas enables distributed CSV processing on HDFS
in pure Python
6. Coming soon — dask on hdfs
72

$ hdfs dfs -cp yellow_tripdata_2014-01.csv /data/nyctaxi/
>>> from distributed import hdfs
>>> blocks = hdfs.get_locations('/data/nyctaxi/', '192.168.50.100', 9000)
>>> columns = ['vendor_id', 'pickup_datetime', 'dropoff_datetime',
'passenger_count', 'trip_distance', 'pickup_longitude', 'pickup_latitude',
'rate_code', 'store_and_fwd_flag', 'dropoff_longitude', 'dropoff_latitude',
'payment_type', 'fare_amount', 'surcharge', 'mta_tax', 'tip_amount',
'tolls_amount', 'total_amount']
>>> from distributed import Executor
>>> executor = Executor('192.168.1.100:8787')
>>> dfs = [executor.submit(pd.read_csv, block['path'], workers=block['hosts'],
... columns=columns, skiprows=1)
... for block in blocks]
These operations produce Future objects that point to remote results on the worker computers. This does not
pull results back to local memory. We can use these futures in later computations with the executor.
73

def sum_series(seq):
result = seq[0]
for s in seq[1:]:
result = result.add(s, fill_value=0)
return result
>>> counts = executor.map(lambda df: df.passenger_count.value_counts(), dfs)
>>> total = executor.submit(sum_series, counts)
>>> total.result()
0 259
1 9727301
2 1891581
3 566248
4 267540
5 789070
6 540444
7 7
8 5
9 16
208 19
74

Bokeh
• Interactive visualization
• Novel graphics
• Streaming, dynamic, large data
• For the browser, with or without a server
• No need to write Javascript
http://bokeh.pydata.org
75

Versatile Plots
76

Novel Graphics
77

Previous: Javascript code generation
HTML
App Model
server.py
Browser
js_str = """ <d3.js>
D3
<highchart.js>
highcharts
<etc.js>
flot
"""
crossfilter
plot.js.template
etc. ...
One-shot; no MVC interaction; no data streaming
78

bokeh.py & bokeh.js
JSON
App Model
server.py
Browser
bokeh-server
bokeh.py
BokehJS
object graph
object graph
79

80

4GB Interactive Web Viz
81

rBokeh
http://hafen.github.io/rbokeh
82

83

84

85

hGp://nbviewer.ipython.org/github/bokeh/bokeh- ‐notebooks/blob/master/tutorial/00 - ‐ intro.ipynb#InteracLon
86

Additional Demos & Topics
• Airline flights
• Pandas table
• Streaming / Animation
• Large data rendering
87

Numba
• Dynamic, just-in-time compiler for Python & NumPy
• Uses LLVM
• Outputs x86 and GPU (CUDA, HSA)
• (Premium version is in Accelerate part of
Anaconda Workgroup and Anaconda Enterprise subscriptions)
http://numba.pydata.org
88

Python Compilation Space
Ahead Of Time
Just In Time
Cython
Relies on
Numba
Shedskin
CPython /
HOPE
Nuitka (today)
libpython
Theano
Pythran
Replaces
CPython /
Nuitka (future)
Pyston
libpython
PyPy
89

Example
Numba
90

~1500x speed-up
@jit('void(f8[:,:],f8[:,:],f8[:,:])')
def filter(image, filt, output):
M, N = image.shape
m, n = filt.shape
for i in range(m//2, M-m//2):
for j in range(n//2, N-n//2):
result = 0.0
for k in range(m):
for l in range(n):
result += image[i+k-m//2,j+l-n//2]*filt[k, l]
output[i,j] = result
91

Numba Features
• Numba supports:
–
Windows, OS X, and Linux
– 32 and 64-bit x86 CPUs and NVIDIA GPUs
– Python 2 and 3
– NumPy versions 1.6 through 1.9
•
Does not require a C/C++ compiler on the user’s system.
•
< 70 MB to instal .
•
Does not replace the standard Python interpreter
(al of your existing Python libraries are stil available)
92

Numba Modes
• object mode: Compiled code operates on Python objects. Only
significant performance improvement is compilation of loops that can be
compiled in nopython mode (see below).
• nopython mode: Compiled code operates on “machine native” data.
Usual y within 25% of the performance of equivalent C or FORTRAN.
93

The Basics
94

The Basics
Numba decorator
(nopython=True not required)
Array Allocation
Looping over ndarray x as an iterator
Using numpy math functions
Returning a slice of the array
95
2.7x speedup!

CUDA Python (in open-source Numba!)
CUDA Development
using Python syntax for
optimal performance!
You have to understand
CUDA at least a little —
writing kernels that launch
in paral el on the GPU
96

Example: Black-Scholes
97

Black-Scholes: Results
core i7
GeForce GTX 560 Ti
About 9x faster
on this GPU
~ same speed as
CUDA-C
98

Other interesting things
• CUDA Simulator to debug your code in Python interpreter
• Generalized ufuncs (@guvectorize)
• Cal ctypes and cffi functions directly and pass them as arguments
• Preliminary support for types that understand the buffer protocol
• Pickle Numba functions to run on remote execution engines
• “numba annotate” to dump HTML annotated version of compiled code
• See: http://numba.pydata.org/numba-doc/0.20.0/
99

What Doesn’t Work?
(A non-comprehensive list)
• Sets, lists, dictionaries, user defined classes (tuples do work!)
• List, set and dictionary comprehensions
• Recursion
• Exceptions with non-constant parameters
• Most string operations (buffer support is very preliminary!)
• yield from
• closures inside a JIT function (compiling JIT functions inside a closure works…)
• Modifying globals
• Passing an axis argument to numpy array reduction functions
• Easy debugging (you have to debug in Python mode).
100

How Numba Works
@jit
def do_math(a,b):
…
>>> do_math(x, y)
Python Function
Function Arguments
(bytecode)
Type Inference
Rewrite IR
Bytecode
Numba IR
Analysis
Lowering
Cache
Execute!
Machine Code
LLVM JIT
LLVM IR
101

Recently Added Numba Features
• A new GPU target: the Heterogenous System Architecture, supported by AMD APUs
• Support for named tuples in nopython mode
• Limited support for lists in nopython mode
• On-disk caching of compiled functions (opt-in) — both LLVM and pre-compiled
• A simulator for debugging GPU functions with the Python debugger on the CPU
• Can choose to release the GIL in nopython functions
• Ahead of time compilation
• vectorize and guvectorize on GPU and paral el targets now in open-source Numba!
• JIT Classes — coming soon!
102