Data

Analysis
With Python

A. Beck

Introduction
Data Analysis With Python
Using
Python

Basic
Python
Arnaud Beck
Scipy

Data I/O

Visualization
Laboratoire Leprince-Ringuet, cole Polytechnique, CNRS/IN2P3

Space Science Training Week

 Outline

Data
Analysis
With Python

A. Beck

Introduction
1 Introduction
Using
Python

Basic 2 Using Python

Python

Scipy

Data I/O 3 Basic Python

Visualization

4 Scipy

5 Data I/O

6 Visualization
 Why come to Python ?

Data
Analysis
Should I use low-level,compiled language or an interpreted language ?
With Python Commercial or open source ?
A. Beck

Introduction
C/C++ Matlab Python
Using Easy and flexible X X
Python

Basic Performances X
Python

Scipy
Free and available on any system X X
Data I/O

Visualization
 Why stick to Python ?

Data
Analysis
Python is distinguished by its large and active scientific computing community.
With Python There are people developing libraries for virtually anything.
A. Beck
Glue to other languages
Introduction

Using Libraries to interface other languages (C/C++/Fortran)...

Python
...with the same performances ! !
Basic
Python Critical part of codes are written in a lower level language.
Scipy

Data I/O Parallelization

Visualization
MPI
OpenMP
GPU

Data management and visualization

IO data in any format (HDF5, VTK, ...)

Data management dedicated libraries (scipy, pandas)
Direct visualization or interfaces with other softwares (Paraview, Mayavi)
 Outline

Data
Analysis
With Python

A. Beck

Introduction
1 Introduction
Using
Python

Basic 2 Using Python

Python

Scipy

Data I/O 3 Basic Python

Visualization

4 Scipy

5 Data I/O

6 Visualization
 Getting Python for data analysis

Data
Analysis
With Python

A. Beck

Basic Python distribution

Introduction

Using Available on any Linux or Mac OS.

Python

Basic
Python
Critical for data analysis
Scipy

Data I/O
Modules : Scipy, Matplotlib
Visualization

Application specific
Modules : mpi4py, VTK, pytable, etc.

It is possible to install fully pre-built scientific Python environment : Enthought

Python Distribution or Python(x,y) for Windows.
 Running Python

Data
Analysis
Interactive mode in a Python shell
With Python

A. Beck

Introduction

Using
Python

Basic
Python

Scipy
Use of a script
Data I/O

Visualization

Turn your python script into a unix script

You can compile scripts into binary .pyc files. Mostly for developers.
 IPython : a convenient and comfortable Python shell

Data
Analysis
With Python

A. Beck

Introduction
Interesting features
Using
Python Command history
Basic
Python Any Xterm command accessible via !
Scipy Commands auto-completion
Data I/O
Quick help through the use of ?
Visualization
Inline and interactive graphics
Timing and profiling tools
Many many more ...

Best tool for exploring, debugging or work interactively. Have a look !

 IPython example

Data
Analysis
With Python

A. Beck

Introduction

Using
Python

Basic
Python

Scipy

Data I/O

Visualization
 Outline

Data
Analysis
With Python

A. Beck

Introduction
1 Introduction
Using
Python

Basic 2 Using Python

Python

Scipy

Data I/O 3 Basic Python

Visualization

4 Scipy

5 Data I/O

6 Visualization
 Python is an object oriented language

Data
Analysis
With Python

A. Beck

Introduction
In Python, we do things with stuff !
Using
Python
things = operations
Basic
stuff = objects
Python

Scipy

Data I/O Type Example

Visualization
Numbers 128, 3.14, 4+5j
Strings 'Rony', "Giovannis"
Lists [1,"string",2.45]
Tuples (1,"string",2.45)

Strings, Lists and Tuples are sequences.

Strings and Tuples are immutable.
 Numbers

Data
Analysis
With Python

A. Beck

Introduction

Using
Python

Basic
Python

Scipy

Data I/O

Visualization
 Strings
Ordered collection (or sequence) of characters

Data
Analysis
With Python

A. Beck

Introduction

Using
Python

Basic
Python

Scipy

Data I/O

Visualization
 String Methods

Data
Analysis
With Python

A. Beck

Introduction

Using
Python

Basic
Python

Scipy

Data I/O

Visualization
 Lists
Sequence of any objects

Data
Analysis
With Python

A. Beck

Introduction

Using
Python

Basic
Python

Scipy

Data I/O

Visualization
 Slices
Manipulating sequences

Data
Analysis
With Python

A. Beck

Introduction

Using
Python

Basic
Python

Scipy

Data I/O

Visualization
 Importing modules

Data
Analysis
With Python

A. Beck
Modules define new object types and operations.
Introduction

Using
Python

Basic
Python

Scipy

Data I/O

Visualization

The large and growing Python users community provides an increasing number
of modules that already do what you need.
 Outline

Data
Analysis
With Python

A. Beck

Introduction
1 Introduction
Using
Python

Basic 2 Using Python

Python

Scipy

Data I/O 3 Basic Python

Visualization

4 Scipy

5 Data I/O

6 Visualization
 The Scipy module

Data
Analysis
With Python

A. Beck Scipy is a collection of powerful , high level functions for mathematics and data
Introduction
management. It is based on the numpy.ndarray object type and vectorized
Using
operations. The operations are optimized and coded in C to deliver high
Python performances.
Basic
Python

Scipy

Data I/O

Visualization

If you are using a for loop, you are probably doing something wrong !
 Creating an ndarray

Data
Analysis
With Python

A. Beck

Introduction

Using
Python

Basic
Python

Scipy

Data I/O

Visualization
 Manipulating ndarrays

Data
Analysis
With Python

A. Beck

Introduction
Slicing is still the basis of array manipulation.
Using
Python Reshape > Change number and size of dimensions of the array.
Basic
Python
Sort > Quite self explanatory.
Scipy Delete, insert, append > Remove or add parts of the array.
Data I/O
Squeeze, flatten, ravel > More ways to control dimensionality of the array.
Visualization
Transpose,swapaxes, rollaxis > More ways to arange the dimensions as
you want

These functions are important because a well aranged data is a quickly

processed data.
 Extracting information from your data

Data
Analysis
With Python

A. Beck

Introduction

Using
Python

Basic
Python

Scipy

Data I/O

Visualization

Intersection (convenient for filtering)

Histograms (perfect for distribution functions)
Convolution
Integration
Interpolation
Name it ...
 Outline

Data
Analysis
With Python

A. Beck

Introduction
1 Introduction
Using
Python

Basic 2 Using Python

Python

Scipy

Data I/O 3 Basic Python

Visualization

4 Scipy

5 Data I/O

6 Visualization
 Reading data

Data
Analysis
With Python

A. Beck

Introduction
The whole game is to fit your data in a ndarray.
Using
Python

Basic
Python

Scipy
data = scipy.fromfile("file",dtype=float32,count=-1,sep=" ")
Data I/O

Visualization Works with raw binary files and ASCII files but not very flexible.

data = scipy.loadtxt("file",skiprows=0,delimiter=",")

More flexible but works only with text files.

 The file object

Data
Analysis
With Python

A. Beck
The file object is a basic python type. It is created by
Introduction

Using
Python

Basic
Python
fid = open("filename","r")
Scipy
"r" for read, "w" for write.
Data I/O

Visualization

fid.readline() > reads a line in a string

fid.readlines() > reads all line in a list of strings
fid.tell() > returns the files current position (in byte)
fid.seek(n) > goes to position n
fid.read() > reads all file in a string
fid.close()
 Manipulating a file

Data
Analysis
With Python

A. Beck

Introduction

Using
Python

Basic
Python

Scipy

Data I/O

Visualization
 Quick words about reading HDF5 files

Data
Analysis
Reading HDF5 files is module dependant. You can use either tables or h5py
With Python for instance.
A. Beck

Introduction These modules coexist well with Scipy and load data directly into ndarray.
Using
Python

Basic tables example

Python

Scipy

Data I/O

Visualization
 Writing data

Data
Analysis scipy.save("file",ndarray) and scipy.load("file") in order to use
With Python
the binary scipy format to store arrays.
A. Beck
ndarray.tofile() in order to store an array in a text file or raw binary.
Introduction
fileobject.write("any_string") to write a string in a text file.
Using
Python The h5py and tables modules are used to write HDF5 files.
Basic
Python VTK script
Scipy

Data I/O

Visualization
 Outline

Data
Analysis
With Python

A. Beck

Introduction
1 Introduction
Using
Python

Basic 2 Using Python

Python

Scipy

Data I/O 3 Basic Python

Visualization

4 Scipy

5 Data I/O

6 Visualization
 Visualization workflow

Data
Analysis
With Python

A. Beck

Introduction
Python Python
Using Raw Data Postprocessed data Formated data file
Python

Basic
Python

Scipy

Data I/O
Python, matplotlib
Visualization
Visualization software
Paraview, Visit, Mayavi ...

Plot
Visualization
 Matplotlib : the figure object

Data
Analysis
fig = figure([options])
With Python

A. Beck Options include :

Introduction Size in inches
Using
Python Dpi 1.0

Basic Face and edge colors

Python

Scipy Frame layout 0.8

Data I/O

Normalized to maximum
Visualization
Operations include :
Title and axis labels 0.6

fig.xlabel("string")
Axis ticks and extent 0.4

fig.ticks(ndarray)
Injected Charge
Display a colorbar
0.2 Bubble Size
fig.colorbar()
Laser Amplitude
Display a legend
fig.legend() 0.0
0 5 10 15

Save figure (png or eps) Propagation length [mm]

fig.savefig()
 Matplotlib : Simple plots

Data
Analysis
plot(x,y,[options])
With Python

A. Beck

Introduction If x is omitted, default is x=range(len(y)).

Using
Python

Basic
All typical options are here : lines (style, color, width ...), markers (size, shape,
Python colors ...), labels for legend, antialiasing, transparency, many more ...
Scipy

Data I/O

Visualization
 Matplotlib 2D plots : imshow and pcolor

Data
Analysis
With Python

A. Beck 2Dar = rand((100,100)) 2Dar = rand((100,100))

imshow(2Dar,[options]) pcolor(2Dar,[options])
Introduction

Using
Python

Basic
Python
1.0 100 1.0

Scipy
0.9 0.9
0
Data I/O
0.8 80 0.8
Visualization
20
0.7 0.7

0.6 60 0.6
40

0.5 0.5

60
0.4 40 0.4

0.3 0.3
80

0.2 20 0.2

0.1 0.1
0 20 40 60 80

0.0 0 0.0
0 20 40 60 80 100
 2D plots with a little bit of tuning

Data
Analysis
With Python

A. Beck 3.0
60

Introduction

Using
40
Python

Basic
Python
20 2.0
Scipy
y [m]

Data I/O
0
Visualization

20 1.0

40

60
0.0
60 40 20 0

x ct [m]
 Other features of matplotlib

Data
Analysis
With Python

A. Beck

Introduction

Using
Python

Basic
Python

Scipy

Data I/O

Visualization
 Matplotlib has native LATEX rendering

Data
Analysis
label = r"$Math \LaTex code$"
With Python

A. Beck

Introduction

Using
Python

Basic
Python

Scipy

Data I/O

Visualization
 The futur of visualization in Python

Data
Analysis
With Python

A. Beck

Introduction

Using
Python

Basic It is an extremely vast, active and changing domain.

Python

Scipy
New modules are emerging : Chaco, MayaVi, Bokeh, stressing interactivity and
Data I/O

Visualization
dynamic data visualizations in web browsers and in 3D.

What you saw today is extremely basic and is only a tiny part of what Python is
capable of.