CS267 Assignment 0

Ekaterina Gonina
Bio:
I am a first year PhD student in CS at UC Berkeleys ParLab. I am interested in parallel
graph algorithms and optimizing them for different parallel hardware platforms. I am also
interested in developing parallel programming frameworks to make programming newly
emerging parallel hardware easier for regular programmers. As and undergrad at UIUC, I
worked with Professor L.V. Kale on parallelizing and optimizing minimum spanning tree
algorithms using MPI and Charm++ (www.charm.cs.uiuc.edu).
What Im hoping to gain from this course is to learn in a nice systematic way the current
state of the art of developing parallel applications and its implications with hands-on
examples. Im hoping to gain a solid foundation for my parallel programming Prelim
exam.
Application:

Data-Parallel Large Vocabulary Continuous Speech

Recognition on GPUs
Speech Recognition is an important application that the Parallel Computing Lab at UC
Berkeley is exploring at to mine for parallel parallel patterns and eventually create
frameworks and techniques to implement inference engines efficiently on manycore
platforms. Speech recognition is a key technology that enables human-computer
interaction in many emerging applications, for example, keeping a diary of a conference
meeting or recording a one-on-one research meeting presents very useful applications
that both the industrial and research community could benefit from. However, the size of
language vocabulary models are extremely large and are proportional to the accuracy of
recognition we want, thus parallel processing is extremely in need here we want to be
able to recognize human speech in small amounts of time, ideally, in real-time, and that is
only possible if we take full advantage of the parallelism of current manycore platforms.
One effective approach for solving the large vocabulary continuous speech recognition
(LVCSR) problem is to use the Hidden Markov Model (HMM) with beam search
approximate inference algorithm. This system uses a recognition network that is
compiled offline from a variety of knowledge sources using powerful statistical learning
techniques. Spectral-based speech features are extracted by signal-processing, the audio
input and the inference engine then computes the most likely word sequence based on the
extracted speech features and the recognition network [1]. Since it is not feasible to
explore the whole recognition network to find the most likely word sequence given the
set of signals from speech, they used the beam search heuristic the reduce the problem
space to a feasible size.

Parallel Platform:
The application targets NVIDIA G8x series GPUs as its parallel platform with SIMD
architecture. It has an array of Shared Multiprocessors (SM) each of which has 8 scalar
processors. They used CUDA programming environment to implement the parallel
version of the algorithm. The application is organized into a sequential host program and
one or more parallel kernels that get invoked from the sequential program to run on the
GPU. The basis of parallel execution is a CUDA thread, the kernel executes a scalar
sequential program across the set of threads. The programmer organizes the threads into
thread blocks that get scheduled to run on the SIMD lanes of one SM multithreaded
processor. Each SM has 16KB of on-chip memory that has high bandwidth and low
latency, this memory is shared among the threads in a block. For more information about
CUDA and the NVIDIA GPU programming see [2].
Inference Engine Implementation:
The inference engine implements the beam search algorithm and iterates over set of
active states to infer the set of next active states in the recognition network graph. Each
iteration begins with a set of active states that represent the set of most likely sequence of
words up to the current observation. The first step computes the observation probabilities
of all potential next states, the second step computes the next state likelihoods, and the
third step selects the most likely next state to retain as the set of new active states for the
next iteration [1]. Figure below illustrates the HMM graph and the high level overview of
the iteration process and the figure below is the detailed breakdown of the inference
engines iterations:

The major bottleneck in the algorithm is the transfer of data from CPU to GPU, thus the
key challenge in effectively using the GPU is to keep all computations and intermediate
results in the GPU memory.
The most significant parallelization potential in this algorithm is in the data level
parallelism. In computing next state likelihoods for example, they parallelize over the set
of active end states and do computations for each state in parallel, and in computing
observation probabilities they take advantage of the embarrassingly parallel structure of
the problem they compute probability for each state independently in parallel.
Results:
The application performed really well on the GPU, it got overall of 9x speedup over its
sequential equivalent, 19x in the computation of observation probabilities kernel and
11.8x in updating the next state likelihoods. The key result is that the performance of the
parallel version is about 6.25x better than the performance of sequential version [1]. The
two versions of the algorithm retain the same accuracy. Below is some performance data
they gathered, broken down by each kernel:

Conclusion:
This project presents a significant step in parallelizing automatic speech recognition on
manycore platforms, specifically on NVIDIA GPUs using CUDA. It uses the data
parallel model to achieve 9x speedup over the sequential version of the algorithm
illustrating that we have a great space to explore in improving performance of speech
recognition applications on such architectures. The next step that we are currently
working on is implementing the inference engine using a different recognition network
Weighted Finite State Transducer (WFST) that is optimized for state space. We hope to
see great speedup and performance improvement from parallelizing speech recognition
using this model.
References:
1. Jike Chong, Youngmin Yi, Nadathur Rajagopalan Satish, Kurt Keutzer. "Data Parallel
Large Vocabulary Continuous Speech Recognition on Graphics Processing Unit". Poster,
GSRC Annual Symposium 2008, 29, September, 2008, 1.1.1.15.
2. NVIDIA CUDA Manual,
http://developer.download.nvidia.com/compute/cuda/2_0/docs/CudaReferenceManual_2.
0.pdf