Indoor Navigation using AR in Warehouses
CS 4605 -- Project 20: Report
Sarthak Srinivas
ssrinivas32@gatech.edu
Sarah Marland
smarland3@gatech.edu
ABSTRACT
Order picking is the process of finding and
picking items inside a warehouse. This process
accounts for up to 55% of the operational cost
of a warehouse1. Travel between different
packages further accounts for 55% of order
picking costs2. Despite variance in how densely
packed orders are between different
warehouses (referred to as pick density in
industry), most warehouses still use old
systems such as static maps and paper to pick
orders. A more performant order-picking
process could optimize this large warehouse
cost, saving employee time and company
money. The goal of this project was to
determine whether heads-up displays on
augmented reality headsets could decrease
order picking time or increase accuracy in
low-density picking environments, thus
reducing travel time. We developed AR
software on Google Glass to create a visual
and intuitive order-picking experience, and
tested it with books at the Georgia Tech
library. We found that using AR headsets like
Google Glass reduced the time taken to pick
books by 38.7%, on average. They are also
easier to use and more comfortable than paper
pick lists, and our participants said that they
would prefer to use AR headsets if they ever
had to pick items in the future.
INTRODUCTION
1
​Bartholdi, John J., and Steven T. Hackman. ​Warehouse &
Distribution Science: Release 0.98.​ Supply Chain and
Logistics Institute, 2017.
2
​Bartholdi, John J., and Steven T. Hackman. ​Warehouse &
Distribution Science: Release 0.98​. Supply Chain and
Logistics Institute, 2017.
​Zack Littke-Smith
zls3@gatech.edu
Daniel Marcos
danielmarcos@gatech.edu
Order picking in warehouses is currently
dominated by paper-based systems, which are
clunky, unwieldy, and largely ignorant of
technological advancements. Simply put, it can
be improved. A more technologically-advanced
system could improve picking times in a
low-density environment (an environment
which has picks spread relatively thinly)
greatly, consequently reducing a great deal of
the running costs to warehouses by making
every order-picking employee much more
efficient. Currently, entirely-robotic systems
don’t have enough dexterity or adaptability to
accommodate warehouse order picking, and
likely won’t for the next few decades. This
leaves a large gap of time in which the
obsoleteness of current paper-based systems
will becoming increasingly glaring. One
available solution is the use of augmented
reality (AR) to make order picking more
portable and dynamic. The aim of this study
was to develop a reasonable AR system and
study its effectiveness against a paper-based
system in order picking. More specifically, the
goal was to determine if the use of an AR head
display could increase the speed of order
picking or increase the accuracy of order
picking in low-density picking environments. In
order to determine a solution to this problem,
our process included the following:
1. Create a test environment to
accurately simulate a warehouse
order-picking scenario.
2. Create a minimalistic user interface to
visualize order picking with the goal of
minimizing the amount of time spent
on the task.
1
 3. Randomly assign participants to a
paper-based or AR order picking
condition and measure the average
amount of time each group takes to
complete a series of picks.
4. Analyze the user data and determine
the effectiveness of an AR picking
system against a paper-based one.
BACKGROUND
Baumann [1] demonstrated it is possible to
improve order picking speed by up to 37% by
using a head mounted display to display visual
cues for order picking]. By utilizing Google
Glass, our application aimed to improve the
speed of order picking in warehouses with a
head-worn display. ​Our user interface took
small inspiration from the ones described by
Schwerdtfeger & Klinker [2] and Guo et al. [6],
who described two possible interfaces to
display visual cues for low density picking
environments using Augmented Reality.
Bartholdi & Hackman [3] provided details
about costs associated with order picking in a
warehouse, which helped us understand the
value of a better solution that we hoped to
create and test. Haynes [4] described the
optimal placement of head-mounted displays
to ensure optimal comfort for wearers,
especially for industrial tasks.
low density picking environments). We
implemented an HUD on Google Glass, and
tested it against a paper-based pick list in the
Georgia Tech library.
Our first step was defining a warehouse-like
environment we could use to test our product
and run studies. We chose the science-fiction
section of the Georgia Tech library as it was
large enough for picking objects that were far
away from each other, yet feasible for
implementation and research. We wrote a
web-scraper to collect book data from the
library’s website, and manually mapped each
book to its physical location. However, we ran
into the problem of inconsistencies between
scraped data from the online library catalog
and real books inside the library. To address
this, we ended up manually scanning 1 book
per shelf in the science fiction section of the
library, or about 288 books total. This became
our master database of books in the library.
With our test environment and data prepared,
we moved on to designing Pick by Paper and
Pick by AR. For Pick by Paper, we created a
pick list by exporting 33 random books from
our master database into a CSV file resembling
Figure 1 below.

Our interface also took small inspiration from

Weaver et al. [5], who described and analyzed
various interfaces for picking in high density
picking environments, as well as sources 7 to
11 which provided additional background
information on the same topic. All of these
studies together helped form our
understanding of the problem at hand, how an
interface for an effective solution might look,
how a significant reduction in order-picking
time could be achieved, and finally how to test
that solution in a realistic environment.

IMPLEMENTATION
Our goal was to determine if the presence of
visual information for routing on a head-up AR
display decreases the amount of time it takes a
human to pick orders in a warehouse or
warehouse-lime environment (specifically in
Figure 1: Example paper pick list

2

For Pick by AR, we first investigated what the
right approach would be for picking books
inside the library. We gathered feedback and
iterated over several possible user interfaces
(UI) to find a design that showed useful
information without being too busy. We
evaluated the user experience (UX) to make
the interaction intuitive: our final design
required a single tap to find items in their
locations, and another to move to the next
pick.
After finalizing our UI and UX, we implemented
an order-picking Android app for Google Glass.
We first created a real-time database on
Firebase to host the book catalogue and
experimental data. Then, we developed the
Glass app using Java and the Android and
Glass APIs. Like Pick by Paper, we ensured that
the pick list for Google Glass also consisted of
33 books.
Figure 2: The two views of our final UI for Pick
by AR
After preparing Pick by Paper and
Pick-by-HUD, we eliminated glitches in our
experiment design by picking a few sample
books by ourselves. We did this to avoid using
up valuable time when our participants came
to test. We discovered that randomly picking
out books in both our paper and AR pick list
was not a good approach, since some pick lists
clustered books closely together while other
pick lists scattered books far apart by pure
chance. To solve this, we modified the
algorithm to assign 33 books out of the total
288 lexicographically instead of randomly. We
also realized that our original pick lists, which
consisted of between 10 and 14 books, did not
match the length of pick-lists in industry,
which ranged from 30 to 50 items per picker.
Therefore, the length of our pick lists was
adjusted to pick 33 items per pick to be
ecologically valid. Finally, we found that we
had trouble reading the titles of books for
smaller volumes, and that we had to pull out
many books in a shelf to actually read the book
title and author. Therefore, we repicked most
of our books to be large hard-bound books
whose titles and authors could be read on the
spine of the book itself.
Once we worked out most of the kinks in our
application, we deployed the app to Glass
devices and printed CSV files to form paper
pick lists. We then conducted a user study to
compare our solution to traditional paper pick
lists.
RESULTS
To compare Pick by Paper with Pick by AR, the
team conducted a lab-based user study with 6
participants. ​Before the user study the
participant was familiarized with the layout of
the science fiction section of the library. The
participant was also told to simply slide the
book in and out of the library shelf, since we
saw no benefit from actually picking out then
replacing the book. Finally, we explained how
instructions would be delivered to the
participant from the AR headset or the paper
pick list. During the user study the participant
would first pick using the paper pick list, get a
five minute break, then pick from the AR
headset. Besides timing Pick by AR through
in-built logging capabilities inside the Google
Glass application, each study was also filmed
and timed. Participants were also asked about
their experience at the end of the test.
During the user study, two parameters were
measured or observed. The first was the time
taken to pick all books in the pick list. The
second was the average task workload. Ideally,
we should have also been able to measure the
accuracy of using both methods. However,
since the books being picked were guaranteed
to be in place, we had an accuracy of 100% for
both methods.
Pick by Paper
In Pick by Paper, participants reported that it
was easy to lose track of which book was being
3
picked since rows in a pick list look largely that
same. Some participants doubled back after
they realized they accidentally skipped a row.
One participant reported that concentrating on
keeping track of picks prevented her from
focusing on finding the book. All participants
said they did not like using paper to pick
books.
Figure 3: One of the authors using Pick by AR
Timing Data
On average, it took 18.0 seconds to pick a
book using Pick by Paper. By contrast, it took
10.8 seconds to pick a book using Pick by AR.
Specific results of the user studies are
tabulated and visualized below:

Subject Paper HUD

(mm:ss) (mm:ss)

1 11:34 6:13

2 9:27 6:48

3 8:57 4.24

4 9:49 6:38

5 9:25 5:55

6 10:01 6:16
Figure 4: One of the authors using Pick by
Paper
Figure 3: Time taken to pick 33 books
Pick by AR
In Pick by AR, participants reported an easier Subject Difference % change
experience understanding where books were (mm:ss) (%)
located. One participant said that once she
understood the visualization of which shelf a 1 5:21 46.3
book was placed on, she stopped looking at the
2 2:39 28.0
aisle and shelf numbers on the shelf entirely
and relied completely on our interface. By 3 4:33 50.8
contrast, all participants said they required a
longer time to get used to the glass 4 3:11 32.4
visualization. This invisible training cost must
be taken into account in the cost of using the 5 3:30 37.2
system. 6 3:45 37.4

Figure 4: Time difference in approaches

The average percent change in time between

Pick by Paper and Pick by HUD is 38.7%.

4

Figure 5: Comparison of Pick by Paper and Pick
by AR
Evaluation
By examining the quantitative data and
qualitative feedback from participants, we
hypothesize that traditional paper pick lists
present too much information to the
participant at the start of the experiment,
which means a lot of mental effort needs to be
exerted to keep track of the current pick,
which prevents the participant from focus on
where the next book actually is. By forcing the
participant to only focus on one book at a time,
we think that we greatly reduced the cognitive
load on the participant.
Since pick lists between Pick by Paper and Pick
by AR were similar, we believe there were
negligible differences between travel times
between different order picking methods as
hypothesized in the beginning of this study.
Rather, we think the large differences in
between trials comes from the ease of use and
the contextual design of our AR interface. We
think the visual mapping of shelves is easier to
understand than a linear alphanumeric listing
of locations, and that by forcing the participant
to pick one book at a time, the participant is
not given the opportunity to be distracted and
can focus effectively at the task at hand.
CONCLUSION AND REFLECTION
From the qualitative and quantitative data we
have collected from the study, we have shown
that Pick by AR is faster than Pick by Paper in
low pick density environments. This is novel
because previous order picking research were
only conducted in high pick density
environments. We believe that although travel
time was mostly identical, the speedups we
observed could be attributed to reduced
context switching and enhanced user
understanding of the task at hand.
We believe there are many other parameters
we can measure in a future study. First, we
could expand the methods being examined to
include Pick by Voice and Pick by RFID, two
other methods commonly used in industry. We
believe it would have been really interesting to
quantify how much mental effort was exerted
by each participant so we could further
investigate how cognitively demanding each
method is – this could have been accomplished
by using NASA-TLX. In future studies, we also
want to increase the number of participants we
work with to 12 so that we could contribute to
HCI workshops or conferences. We realized
upon conversing with the other order picking
group in the class that we did not account for
the time taken to learn both interfaces at the
start of the study. We believe we can
normalize all effects associated with learning to
use a system by making participants pick
sample pick paths on both paper and AR
headsets so they grow used to the system. We
could then plot these different trials to show
how quickly participants learn an order picking
system and therefore what the cost of learning
a system is.
REFERENCES
[1] Baumann, H. (2013). Order picking
supported by mobile computing. ​Doctoral
Dissertation (on pp. 39-43). University of
Bremen​.
(​https://elib.suub.uni-bremen.de/edocs/00102
979-1.pdf​).
[2] Schwerdtfeger, B., & Klinker, G. (2008,
September). Supporting order picking with
augmented reality. In ​Mixed and Augmented
Reality, 2008. ISMAR 2008. 7th IEEE/ACM
International Symposium on (pp. 91-94). IEEE.
(​https://mediatum.ub.tum.de/doc/1094499/fil
e.pdf​).
[3] Bartholdi, J. J., & Hackman, S. T. (2017).
Warehouse & Distribution Science: Release
0.98 (p. 13). Supply Chain and Logistics
5
Institute.
(​http://www2.isye.gatech.edu/~jjb/wh/book/e
ditions/wh-sci-0.98.pdf​)
[4] Haynes, M. (2017). Lateral Positioning of a
Monocular Head-Worn Display. ​Doctoral
Dissertation. Georgia Institute of Technology.
(​https://drive.google.com/file/d/0ByziGNAKVP
ZZRWgwMktIWFJGcHc/view​).
[5] K. Weaver, H. Baumann, T. Starner, H.
Iben, and M. Lawo. “An empirical task analysis
of warehouse order picking using
head-mounted displays.” ACM CHI, pp.
1695-1704, 2010.
(​https://vimeo.com/224844004​).
[6] A. Guo, S. Raghu, X. Xie, S. Ismail, X. Luo,
J. Simoneau, S. Gilliland, H. Baumann, C.
Southern, T. Starner. “A Comparison of Order
Picking Assisted by Head-Up Display (HUD),
Cart-Mounted Display (CMD), Light, and Paper
Pick List.” ACM ISWC, Seattle, WA, September
2014.
[7] A. Guo, X. Wu, Z. Shen, T. Starner, S.
Gilliland, and H. Baumann. “Order Picking with
HeadUp
Displays.” IEEE Computer. 48(6): pp. 16-24,
June 2015.
[8] H. Baumann, T. Starner, H. Iben, A.
Lewandowski, and P. Zschaler. “Evaluation of
graphical user-interfaces for order picking
using head-mounted displays.” ACM ICMI, pp.
377-384,
[9] X. Wu, M. Haynes, Y. Zhang, Z. Jiang, Z.
Shen, A. Guo, T. Starner, S. Gilliland.
“Comparing Order Picking Assisted by Head-Up
Display versus Pick-by-Light with Explicit Pick
Confirmation.” ACM ISWC. Osaka, Japan,
September 2015.
[10] X. Wu, M. Haynes, A. Guo, and T.
Starner. “A comparison of order picking
methods augmented with weight checking
error detection.” ISWC. Heidelberg, Germany,
September 2016.
[11] H. Baumann, T. Starner, and P. Zschaler.
“Studying Order Picking in an Operating
Automobile Manufacturing Plant.” IEEE ISWC,
Newcastle, UK, June 2012.