GPU Computer Vision

The next generation of machines that see

Joe Joe Stam Stam jstam@nvidia.com jstam@nvidia.com

NVIDIA Confidential

Agenda
Why GPUs for Computer Vision? Examples Demos Discussion, future research topics

NVIDIA Confidential

Why use GPUs for Computer Vision

Obvious Answer: Many vision algorithms are highly parallel, and GPUs have lots of cores!

NVIDIA Confidential

Data-Parallel architecture
Same math to many pixels leverage redundancy

NVIDIA Confidential

Massive Memory Bandwidth

Nearly 100 GB/s of on-card memory bandwidth Dedicated Memory not shared with other tasks

NVIDIA Confidential

Latency Hiding
Even with massive bandwidth, memory is very slow relative to compute Caches are not big enough for many vision applications (especially with video) GPUs hide memory latency through fast thread context switch

NVIDIA Confidential

Shared Memory
Acts like a local cache, allowing pixels to access their neighbors with extremely high speed Intermediate results can also be shared between pixels

NVIDIA Confidential

Custom Hardware - Textures

Fast, Cached access to image data Hardware for:

Bilinear Interpolation Type conversion Boundary handling

Excellent for geometrical distortions

NVIDIA Confidential

Video Decoder / Encoder

NVIDIA Confidential

Examples: Panaroma Stitching

Panorama generated from three 3840x2880 images completed in 0.577s on a GTX280 GPU

Work by James Fung at NVIDIA

NVIDIA Confidential

Panaroma Algorithm
Left Image Right Image

Keypoint Detection & Extraction (SIFT)

Keypoint Matching

Recover Homography (RANSAC)

Create Laplacian Pyramid

NVIDIA Confidential

Projective Transform

Multi-Band Blend

Panaroma Performance
Comparison of GPU vs. Photoshop CS3 Photomerge Tool
Time to Stitch 3 Images (11 Megapixels / image)

CPU Xeon E5440 2.83 GHz Quad-Core 4 GB RAM 3.3 s

GPU Tesla C1060 4 GB Framebuffer 0.668 s

NVIDIA Confidential

Stereo Vision
Object

B = baseline distance

FIGURE 1
NVIDIA Confidential

Computationally Insane
Need to find correspondence between left and right image for every pixel For 640 x 480 pixels, 11x11 block match, this is 2 Billion computations for 50-pixel search (with no optimization)

NVIDIA Confidential

GPU Strategy
T H R E A D 0 T H R E A D 1 T H R E A D 2 T H R E A D 3 T H R E A D 4 T H R E A D 5 T H R E A D 6 T H R E A D 7 T H R E A D 8 T H R E A D 9 T H R E A D 10 T H R E A D 11 T H R E A D 12 T H R E A D 13 T H R E A D 14 T H R E A D 15 T H R E A D 0 T H R E A D 1 T H R E A D 2 T H R E A D 3

0 1 2 ...

2
L R

THREAD 0

THREAD 2

col(0..4)
THREAD 1

col(2..6) col(1..5)

NVIDIA Confidential

Heavily leverages SMEM for intermediate computations to share work between threads

Whats Next: GPUs enable practical 3D vision!

Ushers in a whole new class of Human-Machine Interface & Gaming Concepts

NVIDIA Confidential

And the future..

Biologically inspired vision Talk to Nicolas

NVIDIA Confidential