Projected Interactive Display

for Public Spaces

--------------------

A Thesis Proposal
Presented to the Faculty of the
Department of Electronics and Communications Engineering
College of Engineering, De La Salle University

--------------------

In Partial Fulfillment of
The Requirements for the Degree of
Bachelor of Science in Electronics and Communications Engineering

--------------------

by

Arcellana, Anthony A.
Ching, Warren S.
Guevara, Ram Christopher M.
Santos, Marvin S.
So, Jonathan N.

July 2006
1. Introduction

1.1.Background of the Study

Human-computer interaction (HCI) is the study of the interaction between the

users and the computers. The basic goal of HCI is to improve the interaction between

users and computers by making the computers more user-friendly and accessible to

users. HCI in the large is an interdisciplinary area. It is emerging as a specialty

concern within several disciplines, each with different emphases: computer science,

psychology, sociology and industrial design (Hewett et. al., 1996). The ultimate goal

of HCI is to design systems that would minimize the barrier between the human’s

cognitive model of what they want to accomplish and the computer’s understanding

of the user’s task.

The thesis applies a new way to interact with sources of information using an

interactive projected display. For a long time the ubiquitous mouse and keyboard has

been used to control a graphical display. With the advent of increased processing

power and technology, there has been great interest from the academic and

commercial sector in developing new and innovative human computer interfaces in

the past decades. (Myers et. al., 1996). Recently advances and research in human

computer interaction (HCI) has paved the way for techniques such as vision, sound,

speech recognition, and context-aware devices that allow for a much richer,

multimodal interaction between man and machine. (Turk, 1998; Porta, 2002). This

type of recent research moves away from traditional input devices which are

essentially blind into the so called Perceptual User Interfaces (PUI). PUI are

interfaces that emulate the natural capabilities of humans to sense, perceive, and
reason. It models human-computer interaction after human-human interaction. Some

of the advantages of PUIs are as follows: (1) it reduces the dependence on being in

proximity that is required by keyboards and mouse systems, (2) it makes use of

communication techniques found natural in humans, making the interface easy to

use,(3) it allows interfaces to be built for a wider range of users and tasks, (4) it

creates interfaces that are user-centered and not device centered, and (5) it has design

emphasis on being a transparent and unobtrusive interface. (Turk, 1998).

What is interesting in this line of research is the development of natural and

intuitive interface methods that make use of body language. A subset of PUI is Vision

Based Interfaces (VBI) which focuses on the visual awareness of computers to the

people using them. Here computer vision algorithms are used to locate and identify

individuals, track human body motions, model the head, and face, track facial

features, interpret human motion and actions. (Porta, 2002) A certain class of this

research falls under bare hand human-computer interaction which this study is about.

Bare hand human interaction uses as a basis of input, the actions and gestures of the

human hands alone without the use of attached devices.

1.2.Statement of the Problem

Information-rich interactive viewing modules are usually implemented as

computer based kiosks. However placing computer peripherals such as touch-screens

and mouse and keyboard controlled computers in a public area would require

significant space and have maintenance concerns on the physical hardware being used

by the common public. Using a projected display and a camera based input device,
would eliminate the hardware problems associated with the space and maintenance. It

also attracts people since projected displays are new and novel.

1.3.Objectives

1.3.1.General Objectives

The general objective of the thesis is to create an interactive projected

display system using a projector and a camera. The projector would display the

interactive content and the user would use his hand to select objects in the

projected display. Computer vision is used to detect and track the hand and

generate the proper response.

1.3.2.Specific Objectives

1.3.2.1. To use a DLP or LCD projector for the display

1.3.2.2. To use a PC camera as the basis of user input

1.3.2.3. To use a PC to implement algorithms to detect hand action as seen

from the camera

1.3.2.4. To use a PC to host the information-rich content

1.3.2.5. To create an interactive DLSU campus map as a demo application

1.4.Scope and Delimitation

1.4.1.Scope of the Study

1.4.1.1. The proponents will create a real time interactive projected display

using a projector and camera.

1.4.1.2. The proponents will use development tools for image/video

processing and computer vision to program the PC. Algorithms for

hand detection and tracking will be implemented using these tools.

 1.4.1.3. A demo application of the system will be implemented as an

interactive campus map of the school.

1.4.1.4. Only the posture of a pointing hand will be recognize as an input.

Other visual cues to the camera will not be recognized.

1.4.2.Delimitation of the Study

1.4.2.1. The display will be projected in a clean white wall.

1.4.2.2. The projector and the camera set-up will be fixed in such a way that

blocking the projector is not a problem.

1.4.2.3. Trapezoidal distortion which results from projecting from an angle

will be manually compensated if present.

1.4.2.4. Lighting conditions will be controlled to not overpower the projector.

1.4.2.5. The system will be designed to handle only a single user. In the

presence of multiple users, the system would respond to the first user

triggering an event.

1.5.Significance of the Study

The study applies a new way of presenting information using projected

displays and allows the user to interact with it. A projected display conserves space as

the system is ceiling mounted and there is no hardware that directly involves the user.

Using only the hands of the user as an input, the system is intuitive and natural-- key

criteria for effective interfaces. It presents an alternative to computer based modules

where space can be a problem.

 Currently there is a high cost of acquiring and maintaining a projector. But it

is still viable when maintaining an information center is deemed to be important. The

system can be comparable to large screen displays that are used in malls and such.

Since the system is also a novel way of presenting information. It can be used to

make interactive advertisements that are very attracting to consumers. The display

can transform from an inviting advertisement into detailed product information. With

this said, the cost of the operation of the projector can possibly be justified with the

revenue generated from effective advertising. Other applications of the system may

be for use in exhibits which may have particular requirements in uniqueness and

attractiveness. An interactive projected display can provide visitors of the exhibit a

higher level of immersion and have a high level of impact to its viewers.

The study is an endeavor towards the development of natural interfaces. The

use of a projector and camera provides a means of producing an augmented reality

that is natural-- requiring no special goggles or gloves that the user has to wear. In

public spaces where information is very valuable, a system that can provide an added

dimension to reality is very advantageous and the use of nothing but the hands means

the user can instantly tap the content of the projected interface. Computer vision

provides the implementation of a perceptual user interface and the projection provide

the means of creating an augmented reality. Further developments in these areas

means the presence of computers can be found throughout everyday life without

necessarily being attached to it. With the development of PUI there is no need for

physical interface hardware, only the use of natural interaction skills present in every

human would be needed.

1.6.Description of the Project

The system is comprised of 3 main components; (1) the PC which houses the

information and control content, (2) the projector which displays the information, and

(3) the PC camera which is the input of the system.

Block diagram of the system

The system would acquire the image from the camera. Preliminary image

processing is done to prepare the frames of the video to be ready for hand detection

and recognition. The system would then detect the position and action of the hands of

the user relative to the screen. Then it will generate a response from a specific action.

Techniques of image/video processing and machine vision will be used to facilitate

these functions of the PC.

As a demo application an interactive map of the school is used. The projector

will project the campus directory of De La Salle University Manila. The camera will

capture the images needed and will upload to the computer. The user will then pick on

which building he/she would like to explore using his/her hand as the pointing tool.

Once the user has chosen a building, a menu will appear that will give information

about the building. Information includes brief history, floor plans, facilities, faculties,

etc. Once the user is finished exploring the building, he/she can touch the back button

to select another building in the campus. The cycle will just continue until the user is

satisfied.
1.7.Methodology

Development of the study will be heavily invested in the software

programming of the PC. On a hardware level, acquiring and setting up the camera and

projector is needed in the initial stages. Hardware requirements will be looked up for

acquiring the camera and selecting the development PC. However cost consideration

for the projector entail a base model projector that will have a lower luminance than

that may be desired. This has led to a delimitation that lighting conditions should be

controlled.

Implementation of the study is primarily based on programming, such that it

is necessary for the researchers to acquire the necessary skills in programming. Quick

familiarization and efficiency with the libraries and tools for computer vision is

necessary for timely progress in the study. Development of the system will be

undertaken in steps. Code will be developed for acquisition, detection, evaluation,

and then event generation. At each stage, the outputs will be tested such that

requirements will be met.

Computer vision tools have acquisition routines that we can use for the study.

A detection routine based on algorithms discussed in literature will be implemented in

programming. After a working prototype is made, the system will be evaluated for

trial by other users for their inputs. Initial adjustments and fixes will be made

according to their inputs.

Seeking advice from different people may be necessary for speedy progress of

the study. Advice in programming will be very helpful, since the implementation is
PC based. Advice from the panel, adviser, and other people about the interface will be

helpful in removing biases the proponents may have in developing the system.
1.8.

1.9.Estimated Cost

Projector P 50,000

PC Camera P 1500-2000

Open Source SDK Free

Development / Prototype Computer Available

Miscellaneous P5000 ___

Estimated budget P 57000

References:

Beardsley, P. (2005). Interactive projection. IEEE Computer Graphics and Applications.

Hardenberg, C., Bérard, F., (2001). Bare-hand human-computer interaction. in Proc. of

Workshop on Perceptive User Interfaces, PUI'01. 2001. Orlando, Florida.

Hewett, et. al. (1996) Chapter 2: Human Computer Interaction. ACM SIGCHI Curricila
for Human Computer Interaction. Available:
http://sigchi.org/cdg/cdg2.html#2_3 retrieved June 2, 2006.

Kjeldsen, R., Levas, A., & Pinhanez, C. (2003). Dynamically Reconfigurable Vision-
Based User Interface. Retrieved from
http://www.research.ibm.com/ed/publications/icvs03.pdf

Lenman, S., Bretzner, L., Thuresson B., (2002, October). Using Marking Menus to
Develop Command Sets for Computer Vision Based Hand Gesture Interfaces.
Retrieved from http://delivery.acm.org/10.1145/580000/572055/p239-
lenman.pdf?key1=572055&key2=1405429411&coll=GUIDE&dl=ACM&CFID=
77345099&CFTOKEN=54215790

Moeslund T., Liu Y., Storring M., (2004, September). Computer Vision-Based Gesture
Recognition for an Augmented Reality Interface. Marbella, Spain. Retrieved from
http://www.cs.sfu.ca/~mori/courses/cmpt882/papers/augreality.pdf

Myers B., et. al. (1996) Strategic Directions in Human-Computer Interaction. ACM
Computing Surveys Vol.28 No.4

Pinhanez, C. et. al. (2003). Creating touch-screens anywhere with interactive projected
display. Proceedings of the eleventh ACM international conference on
Multimedia.

Pinhanez, C. et. al. (2003). Ubiquitous interactive displays in a retail environment.

Proceedings of ACM Special Interest Group on Graphics (SIGGRAPH): Sketches.
San Diego, California.

Porta, M. (2002) Vision-based user interfaces:methods and applications. International

Journal of Human Computer Studies. Elsevier Science

Turk, M. (1998). Moving from GUIs to PUIs. Symposium on Intelligent Information

Media. Microsoft Research Technical Report MSR-TR-98-69