Python Avanzado

Member of the Helmholtz Association
Scientific Visualization
May 22, 2014

PGI-1 / IAS-1 | Scientific Visualization Workshop | Josef Heinen
Outline
Motivation

Scientific visualization software

Visualization with Python

Python performance optimizations

Development tools

Conclusion

Future plans

Discussion
May 22, 2014
Josef Heinen, Forschungszentrum Jlich, Peter Grnberg Institute, Scientific IT Systems
Motivation
We need easy-to-use methods for:

visualizing and analyzing two- and threedimensional data sets, perhaps with a dynamic
component

creating publication-quality graphics

making glossy figures for high impact journals or
press releases
May 22, 2014
Scientific plotting methods

line / bar graphs, curve plots

scatter plots

surface plots, mesh rendering with iso-surface
generation

contour plots

vector / streamline plots

volume graphics

molecule plots
May 22, 2014
Scientific visualization tools

Gnuplot

Xmgrace

OpenDX

ParaView

Mayavi2

MATLAB

Mathematica

Octave, Scilab, Freemat
May 22, 2014
drawbacks
Gnuplot limited functionality

Xmgrace too old, requires OSF/Motif (X11)

OpenDX no longer maintained (2007)

ParaView not very intuitive

Mayavi2 not very responsive

MATLAB 5 floating, 3 user licenses (16K /year)

Mathematica expensive (~2.500 /user)

Octave, Scilab, Freemat no syntax compatibility
May 22, 2014
Scientific visualization APIs

Matplotlib

mlab,VTK

OpenGL

pgplot

PyQtGraph

PyQwt / PyQwt3D
May 22, 2014
Scientific visualization APIs

Matplotlib de-facto standard (workhorse)

mlab,VTK versatile, but difficult to learn; slow

OpenGL large and complex

pgplot too old

PyQtGraph no yet mature

PyQwt / PyQwt3D currently unmaintained
May 22, 2014
Remaining solutions
GUI + API:
ParaView

Mayavi2

API:
both based on VTK
matplotlib

n.n.

Lets talk about this later
May 22, 2014
ParaView
May 22, 2014
10
Mayavi2
May 22, 2014
11
matplotlib
May 22, 2014
12
Problems so far
separated 2D and (hardware accelerated) 3D
world

some graphics backends "only" produce pictures
no presentation of continuous data streams

bare minimum level of interoperability
limited user interaction

poor performance on large data sets

APIs are partly device- and platform-dependent
May 22, 2014
13
Isnt there an all-in-one solution?
All these components provide powerful APIs

for Python !

!
There must be a reason for that

May 22, 2014
14
so lets push for Python

free and open

dynamic typed language, easy to understand

powerful modules for science, technology, engineering
and mathematics (STEM): NumPy, SciPy, Pandas, SymPy

great visualization libraries: Matplotlib, MayaVi,VTK,
PyOpenGL

techniques to boost execution speed: PyPy, Cython,
PyOpenCL, PyCUDA, Numba

wrapper for GUI toolkits: PyQt4, PyGTK, wxWidgets
May 22, 2014
15
get it up and running
IPython + NumPy + SciPy + Matplotlib

What else do we need?
May 22, 2014
16
achieve more Python performance

Numba: compiles annotated Python and NumPy code
to LLVM (through decorators)

just-in-time compilation

vectorization

parallelization

NumbaPro: adds support for multicore and GPU
architectures
(* Numba (Pro) is part of Anaconda (Accelerate), a (commercial) Python distribution from Continuum Analytics
May 22, 2014
17
achieve more graphics performance and interop

GR framework: a universal framework for crossplatform visualization (*

procedural graphics backend
presentation of continuous data streams

coexistent 2D and 3D world

interoperability with GUI toolkits
good user interaction
(* The GR framework is an in-house project initiated by the group Scientific IT Systems
May 22, 2014
18
Our Scientific Python distribution
IPython + NumPy + SciPy + Numba + GR framework +
PyOpenGL + PyOpenCL + PyCUDA + PyQt4/PyGTK/wxWidgets
more performance and interoperability

May 22, 2014
19
How can we use it?

GR framework (and other mentioned packages)
available on all Linux and OS X machines
(Python and IPython) at PGI / JCNS:
% gr
% igr"
GR framework can also be used with Anaconda:
% anaconda

Windows version(s) on request
May 22, 2014
20
Batteries included
NumPy package for numerical computation

SciPy collection of numerical algorithms and specific toolboxes

Matplotlib popular plotting package

Pandas provides high-performance, easy to use data structures

SymPy symbolic mathematics and computer algebra

IPython rich interactive interface (including IPython notebook)

Mayavi2 3D visualization framework, based on VTK

scikit-image algorithms for image processing

h5py, PyTables managing hierarchical datasets (HDF5)
May 22, 2014
21
Visualization with Python

IPython / PyPy/ Anaconda
Qt / wx
event loop
glgr / iGR App
C / C++
GR
...
GR3
off-screen
rendering
socket
communication
direct
rendering
JavaScript
generation
POV-Ray
generation
C / ObjC
OpenGL
ES
Browser
WebGL
GKS
OpenGL (WGL / CGL / GLX)
POV-Ray
GKS logical device drivers

Qt
gksqt
May 22, 2014
wx
X11
GLUT

wxGLCanvas

QGLWidget
Quartz
GKSTerm
Win32
Java
gksweb
PDF MOV
PS
0MQ SVG
OpenGL ...
HTML5
More logical device drivers / plugins:

CGM, GKSM, GIF, RF, UIL

WMF, Xfig

GS (BMP, JPEG, PNG, TIFF)
Highlights:

simultaneous output to multiple output devices

direct generation of MPEG4 image sequences

flicker-free display ("double buffering")
22
Presentation of continuous data streams in 2D ...

from numpy import sin, cos, sqrt, pi, array
import gr
def rk4(x,
k1 = h
k2 = h
k3 = h
k4 = h
return
h, y, f):
* f(x, y)
* f(x + 0.5 * h, y + 0.5 * k1)
* f(x + 0.5 * h, y + 0.5 * k2)
* f(x + h, y + k3)
x + h, y + (k1 + 2 * (k2 + k3) + k4) / 6.0
def damped_pendulum_deriv(t, state):

theta, omega = state
return array([omega, -gamma * omega - 9.81 / L * sin(theta)])
def pendulum(t, theta, omega)

gr.clearws()
...
# draw pendulum (pivot point, rod, bob, ...)
gr.updatews()
theta = 70.0
# initial angle
gamma = 0.1
# damping coefficient
L = 1
# pendulum length
t = 0
dt = 0.04
state = array([theta * pi / 180, 0])
while t < 30:

t, state = rk4(t, dt, state, damped_pendulum_deriv)
theta, omega = state
pendulum(t, theta, omega)
May 22, 2014
23
... with full 3D functionality

from numpy import sin, cos, array
import gr, gr3
def rk4(x,
k1 = h
k2 = h
k3 = h
k4 = h
return
h, y, f):
* f(x, y)
* f(x + 0.5 * h, y + 0.5 * k1)
* f(x + 0.5 * h, y + 0.5 * k2)
* f(x + h, y + k3)
x + h, y + (k1 + 2 * (k2 + k3) + k4) / 6.0
def pendulum_derivs(t, state):

t1, w1, t2, w2 = state
a = (m1 + m2) * l1
b = m2 * l2 * cos(t1 - t2)
c = m2 * l1 * cos(t1 - t2)
d = m2 * l2
e = -m2 * l2 * w2**2 * sin(t1 - t2) - 9.81 * (m1 + m2) * sin(t1)
f = m2 * l1 * w1**2 * sin(t1 - t2) - m2 * 9.81 * sin(t2)
return array([w1, (e*d-b*f) / (a*d-c*b), w2, (a*f-c*e) / (a*d-c*b)])
def double_pendulum(theta, length, mass):

gr.clearws()
gr3.clear()
!
!
May 22, 2014
...
# draw pivot point, rods, bobs (using 3D meshes)
gr3.drawimage(0, 1, 0, 1, 500, 500, gr3.GR3_Drawable.GR3_DRAWABLE_GKS)

gr.updatews()
24
... in real-time
import wave, pyaudio
import numpy
import gr
SAMPLES=1024
FS=44100
!
f
!
# Sampling frequency
= [FS/float(SAMPLES)*t for t in range(1, SAMPLES/2+1)]
wf = wave.open('Monty_Python.wav', 'rb')
pa = pyaudio.PyAudio()
stream = pa.open(format=pa.get_format_from_width(wf.getsampwidth()),
channels=wf.getnchannels(), rate=wf.getframerate(),
output=True)
!
...
!
data = wf.readframes(SAMPLES)
while data != '' and len(data) == SAMPLES * wf.getsampwidth():
stream.write(data)
amplitudes = numpy.fromstring(data, dtype=numpy.short)
power = abs(numpy.fft.fft(amplitudes / 65536.0))[:SAMPLES/2]
gr.clearws()
...
gr.polyline(SAMPLES/2, f, power)
gr.updatews()
data = wf.readframes(SAMPLES)
May 22, 2014
25
... with user interaction

import gr3
from OpenGL.GLUT import *
# ... Read MRI data
width = height = 1000
isolevel = 100
angle = 0
def display():
vertices, normals = gr3.triangulate(data, (1.0/160, 1.0/160, 1.0/200), (-0.5, -0.5, -0.5), isolevel)
mesh = gr3.createmesh(len(vertices)*3, vertices, normals, np.ones(vertices.shape))
gr3.drawmesh(mesh, 1, (0,0,0), (0,0,1), (0,1,0), (1,1,1), (1,1,1))
gr3.cameralookat(-2*math.cos(angle), -2*math.sin(angle), -0.25, 0, 0, -0.25, 0, 0, -1)
gr3.drawimage(0, width, 0, height, width, height, gr3.GR3_Drawable.GR3_DRAWABLE_OPENGL)
glutSwapBuffers()
gr3.clear()
gr3.deletemesh(ctypes.c_int(mesh.value))
def motion(x, y):
isolevel = 256*y/height
angle = -math.pi + 2*math.pi*x/width
glutPostRedisplay()
glutInit()
glutInitWindowSize(width, height)
glutCreateWindow("Marching Cubes Demo")
glutDisplayFunc(display)
glutMotionFunc(motion)
glutMainLoop()
May 22, 2014
26
... with Qt
May 22, 2014
27
... and wxWidgets
May 22, 2014
28
Scalable graphics in Web browsers
May 22, 2014
29
Import PDF
import gr
(w, h, data) = gr.readimage("fs.pdf")
if w < h:
r = float(w)/h
gr.setviewport(0.5*(1-r), 0.5*(1+r), 0, 1);
else:
r = float(h)/w
gr.setviewport(0, 1, 0.5*(1-r), 0.5*(1+r));
gr.drawimage(0, 1, 0, 1, w, h, data)
gr.updatews()
May 22, 2014
30
Success stories (I)

Worlds most powerful laboratory
small-angle X-ray scattering facility at
Forschungszentrum Jlich
May 22, 2014
31
Success stories (II)

Nframes = 100
radius = 1
height = 4
distance = 5
def RunSimulation():
# defining materials
mAir = HomogeneousMaterial("Air", 0.0, 0.0)
mSubstrate = HomogeneousMaterial("Substrate", 6e-6, 2e-8)
mParticle = HomogeneousMaterial("Particle", 6e-4, 2e-8)
# collection of particles
cylinder_ff = FormFactorCylinder(radius, height)
cylinder = Particle(mParticle, cylinder_ff)
particle_layout = ParticleLayout()
particle_layout.addParticle(cylinder)
# interference function
interference = InterferenceFunction1DParaCrystal(distance, 3 * nanometer)
particle_layout.addInterferenceFunction(interference)
# air layer with particles and substrate form multi layer
air_layer = Layer(mAir)
air_layer.setLayout(particle_layout)
substrate_layer = Layer(mSubstrate)
multi_layer = MultiLayer()
multi_layer.addLayer(air_layer)
multi_layer.addLayer(substrate_layer)
# build and run experiment
simulation = Simulation()
simulation.setDetectorParameters(250, -4*degree, 4*degree, 250, 0*degree, 8*degree)
simulation.setBeamParameters(1.0 * angstrom, 0.2 * degree, 0.0 * degree)
simulation.setSample(multi_layer)
simulation.runSimulation()
return simulation.getIntensityData().getArray()
BornAgain

A software to simulate and fit neutron
and x-ray scattering at grazing incidence
(GISANS and GISAXS), using distortedwave Born approximation (DWBA)
def SetParameters(i):
radius = (1. + (3.0/Nframes)*i) * nanometer
height = (1. + (4.0/Nframes)*i) * nanometer
distance = (10. - (1.0/Nframes)*i) * nanometer
for i in range(100):
SetParameters(i)
result = RunSimulation()
gr.pygr.imshow(numpy.log10(numpy.rot90(result, 1)), cmap=gr.COLORMAP_PILATUS)
May 22, 2014
32
Success stories (III)

NICOS

a network-based control
system written for
neutron scattering
instruments at the FRM II
May 22, 2014
33
Coming soon:
Python moldyn package
May 22, 2014
34
with video output
May 22, 2014
35
and POV-Ray output
May 22, 2014
36
in highest resolution
May 22, 2014
37
Performance optimizations
May 22, 2014
NumPy
module for handling multi-dimensional arrays (ndarray)

Numba (Anaconda)
just-in-time compilation driven by @autojit- or @jitdecorators (LLVM)

vectorization of ndarray based functions (ufuncs)

Numba Pro (Anaconda Accelerate)
parallel loops and ufuncs

execution of ufunfs on GPUs

Python GPU kernels

GPU optimized libraries (cuBLAS, cuFFT, cuRAND)
38
Realization
May 22, 2014
NumPy
vector operations on ndarrays instead of loops
works in any NumPy Python environment

Numba (Anaconda)
add @jit and @autojit decorators
useful for many function calls with big arrays

Numba Pro (Anaconda Accelerate)
add @vectorize decorators
implementation of multi-core / GPU kernels in "Python"
switch to GPU-optimized features
useful only for "large" arrays
performance
39
Particle simulation
import
numpy as np
!
! numba.decorators import autojit
from
!
N
# number of particles
! = 300
M
# masses
! = 0.05 * np.ones(N)
size
# particle size
! = 0.04
!
!
@autojit
def
! step(dt, size, a):
# update positions
! a[0] += dt * a[1]
!
! n = a.shape[1]
! D = np.empty((n, n), dtype=np.float)
! for i in range(n):
for j in range(n):
!
dx = a[0, i, 0] - a[0, j, 0]
!
dy = a[0, i, 1] - a[0, j, 1]
!
D[i, j] = np.sqrt(dx*dx + dy*dy)
!
!
! ... # find pairs of particles undergoing a collision
! ... # check for crossing boundary
! return a
...
!
!
!a[0, :] = -0.5 + np.random.random((N, 2)) # positions
a[1,
! :] = -0.5 + np.random.random((N, 2)) # velocities
a[0,
! :] *= (4 - 2*size)
dt
! = 1. / 30
!
while
True:
!
! a = step(dt, size, a)
....
May 22, 2014
40
Diffusion
!
import
numpy
! numba.decorators import jit
from
!
! = 0.005
dx
! = 0.005
dy
! = 0.5
a
! = dx*dx*dy*dy/(2*a*(dx*dx+dy*dy))
dt
!
timesteps
= 300
!
! = int(1/dx)
nx
! = int(1/dy)
ny
! = numpy.zeros([nx,ny])
ui
! = numpy.zeros([nx,ny])
u
!
!
! diff_step(u, ui):
def
! for i in range(1,nx-1):
!
for j in range(1,ny-1):
!
uxx = ( ui[i+1,j] - 2*ui[i,j] + ui[i-1, j] )/(dx*dx)
!
uyy = ( ui[i,j+1] - 2*ui[i,j] + ui[i, j-1] )/(dy*dy)
!
u[i,j] = ui[i,j]+dt*a*(uxx+uyy)
!
!
diff_step_numba
= jit('void(f8[:,:], f8[:,:])')(diff_step)
!
! m in range(timesteps):
for
diff_step_numba(u,
diff_step
(u, ui)
! ui = numpy.copy(u)
...
May 22, 2014
41
Mandelbrot set
from numbapro import vectorize
import numpy as np
@vectorize(['uint8(uint32, f8, f8, f8, f8, uint32, uint32, uint32)'], target='gpu')

def mandel(tid, min_x, max_x, min_y, max_y, width, height, iters):
pixel_size_x = (max_x - min_x) / width
pixel_size_y = (max_y - min_y) / height
!
!
!
!
!
!
!
x = tid % width
y = tid / width
real = min_x + x * pixel_size_x
imag = min_y + y * pixel_size_y
c = complex(real, imag)
z = 0.0j
for i in range(iters):
z = z * z + c
if (z.real * z.real + z.imag * z.imag) >= 4:
return i
return 255
def create_fractal(min_x, max_x, min_y, max_y, width, height, iters):

tids = np.arange(width * height, dtype=np.uint32)
return mandel(tids, np.float64(min_x), np.float64(max_x), np.float64(min_y),
np.float64(max_y), np.uint32(height), np.uint32(width),
np.uint32(iters))
May 22, 2014
42
Performance comparison
Calculation of Mandelbrot set
May 22, 2014
43
Numba (Pro) review
functions with numerical code can be compiled with little effort

and lead to impressive results

numerical code should be separated from logic statements (and
processing of lists, dictionaries)

advanced Technologie due to LLVM intermediate language
(LLVM IR)

easy installation and maintenance
Download link (Continuum Analytics): http://www.continuum.io/downloads
% bash Anaconda-1.x.x-[Linux|MacOSX]-x86[_64].sh
% conda update conda
% conda update anaconda
May 22, 2014
44
Development tools
You can use your favorite editor and start Python in a shell. But
the impatient user should chose a development environment:
May 22, 2014
45
IPython console
May 22, 2014
46
IPython notebook
May 22, 2014
47
Spyder
May 22, 2014
48
PyCharm
May 22, 2014
49
Bokeh
import numpy as np
from scipy.integrate import odeint
from bokeh.plotting import *
sigma = 10
rho = 28
beta = 8.0/3
theta = 3 * np.pi / 4
def lorenz(xyz, t):

x, y, z = xyz
x_dot = sigma * (y - x)
y_dot = x * rho - x * z - y
z_dot = x * y - beta* z
return [x_dot, y_dot, z_dot]
initial = (-10, -7, 35)

t = np.arange(0, 100, 0.006)
!
solution = odeint(lorenz, initial, t)
!
x = solution[:, 0]
y = solution[:, 1]
z = solution[:, 2]
xprime = np.cos(theta) * x - np.sin(theta) * y
colors = ["#C6DBEF", "#9ECAE1", "#6BAED6", #4292C6",

"#2171B5", "#08519C", "#08306B",]
!
output_file("lorenz.html", title="lorenz.py example")
!
multi_line(np.array_split(xprime, 7), np.array_split(z, 7),

line_color=colors, line_alpha=0.8, line_width=1.5,
tools=pan,wheel_zoom,box_zoom,reset,previewsave",
title="lorenz example", name="lorenz_example")
show()
May 22, 2014
# open a browser
50
Resources
May 22, 2014
Website: http://gr-framework.org

PyPI: https://pypi.python.org/pypi/gr

Git Repository: http://github.com/jheinen/gr

Binstar: https://binstar.org/jheinen/gr

Talk: Scientific Visualization Workshop (PDF, HTML)
51
Website
May 22, 2014
52
Git-Repo
May 22, 2014
53
PyPI
May 22, 2014
54
Binstar
May 22, 2014
55
Conclusion
May 22, 2014
The use of Python with the GR framework and Numba (Pro)

extensions allows the realization of high-performance
visualization applications in scientific and technical environments

The GR framework can seamlessly be integrated into "foreign"
Python environments, e.g. Anaconda, by using the ctypes
mechanism

Anaconda (Accelerate) is an easy to manage (commercial)
Python distribution that can be enhanced by the use of the GR
framework with its functions for real-time or 3D visualization
applications
56
Future plans
May 22, 2014
implement your(!) feature requests

moldyn package for Python

more

tutorials

convenience functions

documentation

examples (gallery)

IPython notebook integration

Bokeh integration
57
Thank you for your attention

References:
Numba, A Dynamic Python compiler for Science: http://lanyrd.com/2013/pycon/scdyzh
Continuum Analytics: http://www.continuum.io

!
Contact:
j.heinen@fz-juelich.de
@josef_heinen"
!
Thanks to:
Florian Rhiem, Ingo Heimbach, Christian Felder, David Knodt, Jrg Winkler, Fabian Beule,
Marcel Dck, Marvin Goblet, et al.
May 22, 2014
58

Python Avanzado

Încărcat de

Informații document

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Python Avanzado

Încărcat de

Drepturi de autor:

Formate disponibile

Member of the Helmholtz Association

May 22, 2014

PGI-1 / IAS-1 | Scientific Visualization Workshop | Josef Heinen

Josef Heinen, Forschungszentrum Jlich, Peter Grnberg Institute, Scientific IT Systems

May 22, 2014

Josef Heinen, Forschungszentrum Jlich, Peter Grnberg Institute, Scientific IT Systems

Scientific plotting methods

Josef Heinen, Forschungszentrum Jlich, Peter Grnberg Institute, Scientific IT Systems

Scientific visualization tools

Josef Heinen, Forschungszentrum Jlich, Peter Grnberg Institute, Scientific IT Systems

Josef Heinen, Forschungszentrum Jlich, Peter Grnberg Institute, Scientific IT Systems

Scientific visualization APIs

May 22, 2014

Josef Heinen, Forschungszentrum Jlich, Peter Grnberg Institute, Scientific IT Systems

Scientific visualization APIs

May 22, 2014

Josef Heinen, Forschungszentrum Jlich, Peter Grnberg Institute, Scientific IT Systems

both based on VTK

May 22, 2014

Josef Heinen, Forschungszentrum Jlich, Peter Grnberg Institute, Scientific IT Systems

May 22, 2014

Josef Heinen, Forschungszentrum Jlich, Peter Grnberg Institute, Scientific IT Systems

May 22, 2014

Josef Heinen, Forschungszentrum Jlich, Peter Grnberg Institute, Scientific IT Systems

May 22, 2014

Josef Heinen, Forschungszentrum Jlich, Peter Grnberg Institute, Scientific IT Systems

Josef Heinen, Forschungszentrum Jlich, Peter Grnberg Institute, Scientific IT Systems

Isnt there an all-in-one solution?

All these components provide powerful APIs

There must be a reason for that

Josef Heinen, Forschungszentrum Jlich, Peter Grnberg Institute, Scientific IT Systems

so lets push for Python

Josef Heinen, Forschungszentrum Jlich, Peter Grnberg Institute, Scientific IT Systems

get it up and running

IPython + NumPy + SciPy + Matplotlib

May 22, 2014

Josef Heinen, Forschungszentrum Jlich, Peter Grnberg Institute, Scientific IT Systems

achieve more Python performance

May 22, 2014

Josef Heinen, Forschungszentrum Jlich, Peter Grnberg Institute, Scientific IT Systems

achieve more graphics performance and interop

May 22, 2014

Josef Heinen, Forschungszentrum Jlich, Peter Grnberg Institute, Scientific IT Systems

Our Scientific Python distribution

IPython + NumPy + SciPy + Numba + GR framework +

PyOpenGL + PyOpenCL + PyCUDA + PyQt4/PyGTK/wxWidgets

more performance and interoperability

Josef Heinen, Forschungszentrum Jlich, Peter Grnberg Institute, Scientific IT Systems

How can we use it?

May 22, 2014

Josef Heinen, Forschungszentrum Jlich, Peter Grnberg Institute, Scientific IT Systems

May 22, 2014

Josef Heinen, Forschungszentrum Jlich, Peter Grnberg Institute, Scientific IT Systems

Visualization with Python

glgr / iGR App

OpenGL (WGL / CGL / GLX)

GKS logical device drivers

May 22, 2014

Josef Heinen, Forschungszentrum Jlich, Peter Grnberg Institute, Scientific IT Systems

More logical device drivers / plugins:

Presentation of continuous data streams in 2D ...

def damped_pendulum_deriv(t, state):

def pendulum(t, theta, omega)

while t < 30: