Documente Academic
Documente Profesional
Documente Cultură
Politecnico di Torino
III Facoltà di Ingegneria
Author:
Matteo Giaccone
February 2010
Acknowledgements
This work would not be possible without the help and motivation
received in the years by my family, by Agnese and from a lot of old
time friends.
Finally, thanks to all the users of d-touch that had the courage of
trying this instruments and make this incredible adventure happen.
i
Abstract
ii
Contents
Acknowledgements i
Abstract ii
1 Introduction 1
2 Related Work 5
2.1 Early Work . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Audio TUI . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3 Markers . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.4 TUI User Studies . . . . . . . . . . . . . . . . . . . . 12
2.5 Large Scale User Studies . . . . . . . . . . . . . . . . 14
2.6 Grounded Theory . . . . . . . . . . . . . . . . . . . . 15
3 Audio d-touch 17
3.1 Working Principle . . . . . . . . . . . . . . . . . . . . 17
3.2 The Software . . . . . . . . . . . . . . . . . . . . . . 19
3.3 d-touch Drum Machine . . . . . . . . . . . . . . . . . 20
3.4 d-touch Sequencer . . . . . . . . . . . . . . . . . . . . 21
3.5 Logging System . . . . . . . . . . . . . . . . . . . . . 22
3.6 Distribution . . . . . . . . . . . . . . . . . . . . . . . 24
3.7 Diary of the Launch . . . . . . . . . . . . . . . . . . 26
iii
5 User Generated Content: collection and analysis 37
5.1 Data Collection . . . . . . . . . . . . . . . . . . . . . 37
5.2 Data Analysis . . . . . . . . . . . . . . . . . . . . . . 38
5.2.1 User Generated Content: Text . . . . . . . . . 38
5.2.2 User Generated Content: Videos and Photos . 45
6 Discussion 49
7 Future Work 52
8 Conclusions 54
A UGC texts 56
D Riassunto in Italiano 69
D.1 Introduzione . . . . . . . . . . . . . . . . . . . . . . . 69
D.2 Audio d-touch . . . . . . . . . . . . . . . . . . . . . . 71
D.2.1 Come funziona . . . . . . . . . . . . . . . . . 72
D.2.2 d-touch Drum Machine . . . . . . . . . . . . . 73
D.2.3 d-touch Sequencer . . . . . . . . . . . . . . . 73
D.2.4 Sistema di Registrazione Remota . . . . . . . 74
D.2.5 Distribuzione . . . . . . . . . . . . . . . . . . 75
D.2.6 Il Lancio Online . . . . . . . . . . . . . . . . . 76
D.3 Raccolta e Analisi dei Log . . . . . . . . . . . . . . . 77
D.4 Raccolta e Analisi di Contenuti Creati dagli Utenti . 80
D.4.1 Analisi dei Testi . . . . . . . . . . . . . . . . . 81
D.4.2 Analisi delle Foto e dei Video . . . . . . . . . 82
D.5 Discussione . . . . . . . . . . . . . . . . . . . . . . . 83
D.6 Lavoro Futuro . . . . . . . . . . . . . . . . . . . . . . 85
D.7 Conclusione . . . . . . . . . . . . . . . . . . . . . . . 86
Bibliography 88
iv
Chapter 1
Introduction
1
stantly interacting with intelligent devices, from the personal com-
puter to the VCR and tens of other invisible intelligent companions.
This trend is described as ubiquitous or pervasive computing [2].
TUIs are popular in HCI since the first half of 1990s when Well-
ner, Fitzmaurice, Ishii and Ullmer [5, 6, 3, 7] started developing
these new interfaces. TUIs are debated in the HCI community
since the pioneering works cited above, and since 2007 the ACM
participates to the organization of a specific conference on tangible
interfaces, the TEI conference3 .
2
interesting for certain type of educational tasks and improved cre-
ativity, especially when some kind of cooperation has been made
possible.
3
were the high cost and the high technology that is usually needed by
a TUI, for example a sensing system or a retro-projected surface [13,
11]. Other typical usage of TUIs were in museums or in controlled
environments as school classes with trained teachers [14, 10]. The
suitability of TUIs in everyday environment with untrained users
remain an open question for everyday users’ environments.
4
Chapter 2
Related Work
5
camera and projectors are under the table), the projector that aug-
ment the surface near the camera and the computer vision program
to elaborate the data seen from the camera. This setup has inspired
also the d-touch applications that use the same strategy.
6
The word Tangible User Interface was born with the paper Tan-
gible Bits by Ishii and Ullmer [7]. This paper is synergic with the
Weiser paper on Ubiquitous Computing (UbiComp) [18]. The focus
of the two papers is that with UbiComp and TUIs become possible
the use of natural human abilities like the ability to interact with
physical objects, and the use of background perception abilities of
humans. The Weiser objective is to put as much actions as possible
in the perceptive background. The concept behind is that if some-
one is doing something in his perceptive background, it means that
he can do something else in his focus, so doing more without using
more effort. With this idea in mind, Ishii and Ullmer developed
some experiments, like the ambientROOM, where the background
perception is used to react in case of ambient changes.
The project Urp [20] from Underkoffler and Ishii at the MIT Me-
7
dia Lab explores the possibility of tangible objects that express also
a physical meaning. Urp is a tangible table-top interface for urban
planning and design. The buildings in the applications are repre-
sented with objects with the shape of the real buildings in order
to understand how they look like and to see the shadows projected
on the table after Urp’s calculations. At the end of the paper the
authors explore the possibilities given by objects on TUIs. Here in
Urp objects can have the same shape of real buildings, or the shape
of a clock to regulate time used in shadow calculation and projec-
tion. Starting from these difference they explore the possibilities of
tangibles and their different possible meaning when used in TUIs.
8
the model is directly connected to Representation and Control.
9
the video feedback and the complex interaction are parts of the
project common in several subsequent experiments. Audio d-touch
is different from this approach in the marker tracking (Computer
Vision approach) and in the simple and essential interface, allowing
an easy reproduction and installation.
10
LCD screen used horizontally as the base of the instrument. Over
the screen were placed the electronic circuits and holes that were
closed with steel balls to trigger the music.
2.3 Markers
11
Another framework to build AR applications on desktop is DART [24].
The focus of DART and DART-TUI is to easily prototype Tangible
Interface applications through Adobe Director12 extensions. In the
paper they also illustrate an example application built with their
framework about physics and mathematics experiments in a school.
12
Two user studies on TUIs that we have analyzed [29, 28], were
done with adults and so it was possible the handling of a ques-
tionnaire (seven point Likert-scale or NASA-TLX) and a statistical
analysis on answers. In either cases the results were used to see
that the interface was working as expected and to compare it to a
preexistent model to have a comparison. Instead, all the user stud-
ies analyzed used informal questions, or the analysis of videotaped
experiments to gain a better understanding of the study and to find
problems or possible improvements. We decided then to not use
questionnaires neither to ask people to fill them, since it was an ex-
ploratory study and we didn’t find good previous work to compare
in a large scale. We tried to gain some insight from the users answer-
ing emails or on the d-touch forum13 in form of informal questioning
or just reading the thought that people autonomously sent us.
13
2.5 Large Scale User Studies
Another work that has marked good points using the distributed
intelligence of Internet users is the study on Mechanical Turk [16].
In this study the Amazon Mechanical Turk14 was used to under-
stand if the shared intelligence of the Internet users is valuable as
the intelligence of experts. Obviously the result is not completely
satisfactory, but if the work is thought expressly for the Mechanical
Turk, good results could be obtained. This paper is useful for us to
understand that if searching for the right things, the tests with a
large number of users are a valuable source of informations.
14
A previous large scale user study on a particular interface or
device is the Scent Field Trial [30] by Nokia. This study is peculiar
because it’s held internally in the Nokia Intranet using a specific
Nokia mobile phone, available to 800 employees. The basis of this
study is that the device was already available to the users so that
they didn’t have to spend money. For a novel TUI this approach
is impossible except if the participants decide to buy the product
studied as might happen in a mobile phones company or with other
kind of commercial products.
15
our research this last step has produced two different ideas grounded
in data and not a real theory that could comprehend everything.
16
Chapter 3
Audio d-touch
17
Figure 2.8, that represent one hand and is associated with medium
volume of sample reproduction. Instead the marker in Figure 3.1,
displaying two hands, was used to play sample at an higher volume.
The markers were tested to be recognized at a resolution of 320 by
240 with the A4 board completely visible. The speed of the webcam
is useful only to get a low latency response from the instrument
but anything that could work at 15 frame per second or more is
good. With this low requirements the webcam can be very cheap
and the recognition algorithm was developed in 2003 and roughly
tested for performance in previous papers [25]. We can say that the
recognition algorithm was never the bottleneck of the application,
the speed was regulated always by the camera frame rate.
18
different behaviors and have different uses.
19
The last part of the software, the remote logging engine and the
activation GUI has been written in C++ with Qt3 . All the libraries
are cross platform and we didn’t experienced particular problems
using them.
20
3.4 d-touch Sequencer
21
in the interface but they will reproduce the same sample. The col-
ored arrow, if put pointing at the right, as the play symbol, indicates
the normal speed of reproduction of the sample associated with the
block. The Start block has a twin called End block. The two blocks
can be used together or one at a time and they indicate that only
the subset of the board that they delimit will be looped to get the
sequence. These blocks can be used to break the constant duration
of loop, or in conjunction with the store action, to store in a block
a subset of the played sequence. The Time Control blocks are four,
differentiated by the different times (2 sec, 4 sec, 8 sec, 16 sec), and
are used to change the loop duration of the sequencer. The default
duration of the entire loop is 2 seconds, but can be changed to 16
seconds with 4 seconds steps. To use the Control blocks the user
has to place them in the board and they will change the behavior of
the instrument. If the Time Control blocks are covered or removed
the last time that has been set up is kept, while if the Start or End
blocks are removed the sequence is played completely and not only
in the last subset indicated.
To enable the remote user study, a new part of the d-touch appli-
cations that sends usage logs to a remote server that stores them in
a database has been developed.
22
to input activation data. To limit the possibilities of shared ac-
counts between users a MAC address check on activation has been
implemented. When the user activates the software for the first
time it sends to the server an hash of the username, the password
and the MAC address of the machine, then the hash is stored. The
next time that the application is started the hash is created and
sent to the server. If it’s different another activation is required. So
if someone shares an account with someone else each time that he
starts the application he gets annoyed by the activation. We hoped
that this behavior could push the users to register and use their
own account in order to preserve usage patterns of single users, as
the time span between the first play and the last play, that we’ll
see in the next section. With these solutions we noticed, from IP
addresses belonging to different countries, only one case of shared
account and we removed it from our database.
The registration phase on the web has been useful also to gather
some background information for the users. We asked to users their
age, sex, profession, musical knowledge and practice, TUI knowl-
edge and practice. These data were later used to group logged
data and try to understand how people with different background
could differently interact with the d-touch instruments. With the
applications, as with the registration, we decided to collect only the
essential data interferring as little as possible with the users’ pri-
vacy. The application registers the marker and calibration position
and angle, the timestamp, the user and the IP address of the con-
nection. Every 25 frames the logging part of the application sends
data to the remote server that stores it along with the user iden-
tifier. We never gathered pictures from the webcam to reconstruct
the usage. For the sequencer we decided to log also the registered
sounds, otherwise we could not understand what the user was do-
ing but we thought that this was not an invasive practice since the
audio should only be recorded during playing. In fact we never reg-
istered personal conversations. However we always stated clearly
and as better as possible that the application was free because we
23
were doing a user study and that we were registering user actions
during usage. Also the fact that the application could work only
with an Internet connection raised attention and understanding on
this point. We received a lot of emails and messages of user com-
plaining on the need of Internet connection, but only once a user
asked to be removed from our database.
From the server point of view, the logging was done modifying
an existing system based on TikiWiki4 a PHP based content man-
agement system. A PHP script has been developed to get the data
arriving from the applications in form of HTTP POST calls. The
data is gathered, associated with the user that has sent the data
and then stored in a MySql database. The system has revealed it-
self fairly robust since it gathered simultaneously data from multiple
users apparently without losing data.
3.6 Distribution
The low cost and easy to set-up nature of the d-touch instruments
were design goals from the beginning of the development. No sol-
dering, particular electronic components nor special handcraft is
needed, just a webcam (or a consumer grade DV camera), a com-
puter with Windows, Mac OS X or Linux, an audio card and an
ink-jet printer are needed. The complete package with software,
images with blocks and board to print is downloadable in less than
10 Mega Bytes.
24
unknown hardware and software configurations has revealed itself a
non-trivial technical challenge. The system has been heavily tested
and some deep software bug has been found. We can say that the
public release of the software forced us to clean and review a large
part of the audio and computer vision code of d-touch. Then for
the first time we tested the cross-platform capabilities on OS X
and finally we had to prepare installation packages with external
libraries for Windows and OS X.
To have an easy to
setup tangible interface
we prepared the ready to
cut blocks, visible in Fig-
ure 3.5, the instructions
to build a webcam stand
with the paper outline to
be glued on cardboard
and the image to print
the board. We provided
also the plain markers for
the lazy users, suggest-
ing an easy and quick way
to build the d-touch sys-
tem: the webcam could
be hanged to a lamp, the Figure 3.5: d-touch marker on a ready to build block
plain markers could be
glued on objects like small chocolate bars or nuts and we demon-
strated it with a video that we posted on YouTube5 where we used
chocolate and walnuts as marker support. Otherwise we suggested
to fill the blocks with lentils to give some weight to the blocks and
have a better control of them6 . We did these videos mainly to con-
vey the easy to build and low cost materials needed for the project
as well as to get some media attention on the project.
5 http://www.youtube.com/watch?v=lCv0TvnVUHg
6 http://www.youtube.com/watch?v=AmGP3eCOF5A
25
3.7 Diary of the Launch
26
with the collection of informal comments.
The first step that we did was the creation of the d-touch.org
website to promote and publish the applications. On the site we
made available for download the Windows and OS X applications,
together with the PDF file of blocks and board, after a registration
procedure. In a first phase we made publicly available only the
drum machine while we were completing also the revision of the
sequencer. To explain to people how d-touch works and to promote
the diffusion we did a short video that we published on YouTube8
and we featured it in the d-touch.org homepage.
27
continued to be viewed and totally we received 25000 visits and the
project was featured in more than 30 blogs including 2 hands-on
original reviews10 .
10 http://d-touch.org/coverage/
11 http://www.youtube.com/watch?v=cKd8NXWwvKI
12 http://d-touch.org/audio/material/
28
Chapter 4
29
to try to understand the actions taken by the people. The mixture
of the two approaches, with the the UGC analysis done in the next
chapter will help drawing some conclusion.
30
Sequencer Drum Machine Overall
Avg session length (min.) 8.75, (σ = 10.12) 6.75, (σ = 8.05) 7.34, (σ = 8.76)
Avg no. sessions per user 3.33, (σ = 3.13) 5.45, (σ = 6.35) 5.55, (σ = 6.34)
Max no. sessions per user 15 30 39
Avg blocks in session 1.80, (σ = 1.71) 3.06, (σ = 3.75) 2.69, (σ = 3.33)
Max blocks in session 5.49 7.11 6.64
Sessions with 8 blocks or more 96, (8.35%) 318, (27.70%) 414, (25.43%)
No. of sessions 479, (29.42%) 1149, (70.58%) 1628, (100%)
Minutes of usage 4193.25, (35.10%) 7752.62, (64.90%) 11945.87, (100%)
Table 4.1: Audio d-touch usage data, gathered from the interaction logs. In brackets the
standard deviation of the average values.
the 273 users that successfully started Audio d-touch and used it
for more than one minute, except 7 people that filled forms with
useless data. 27% of our users were under 20 years of age, 73%
under 30 and 94% under 40. Females were very few, only the 2% of
users. Few users reported previous knowledge with TUIs and even
less reported previous experience with them.
31
(a) Boxplot of maximum number of blocks.
Figure 4.2: Comparison between instruments. The boxplot used in these pictures indicates:
with the box the lower and upper quartile (25% to 75%), in red the median and the whiskers
are 1.5 times 50%. Usually in the box will be the majority of data, with the whiskers we see
the minimum and the maximum and with blue crosses are marked the outliars.
32
with the complete A4 paper full of markers that we provided in
PDF. We can say that on average the sequencer sessions showed
more blocks, probably because of the two available tracks to place
markers and to the more interesting instrument that engages more
the users.
Generally usage of d-
touch was infrequent: 21%
of users interacted with
d-touch over a period
longer than 2 days and
11% for more than one
week. We can also see,
from the histogram in
Figure 4.3, that the us-
Figure 4.3: Histogram of number of sessions in a time
age was compressed in slot. The bins are half an hour big in the global image
the large majority in less and 2 minutes long in the zoomed part.
than two hours. All the
quantitative analysis is useful to understand better the pattern of
usage that we will later see and the UGC that will be presented in
the next chapter. In the final discussion, we will review this data
in conjunction with the others to get a complete picture over the
experiment.
33
(a) Usage graph. The number of blocks in the (b) Usage plot. This is a heat map presenting
interface are mapped on the y-axis while the time where markers has been positioned during a session
is mapped on the x-axis. to understand if some movement has happened.
Figure 4.4: Selection principles. The video analysis has been done selecting videos that pre-
sented interesting patterns in above pictures.
drew heat maps of usage, based on where the blocks were more
present during a session. The plotting resulted in images like the
Figure 4.4b. The pixels were more colored if the block was present
for a relevant number of second in a certain position. From this
plots we can see if a user has moved blocks during a session or if the
session was all equal because he forgot the instrument on an he went
away. With this previous analysis we found interesting sessions and
then we produced the videos regarding the history of these users to
see the learning curve and the possibly interesting results. Using
this selection approach we managed to analyze around 24 hours of
videos, instead of the 90 present at the time of video generation and
we were satisfied with the quality of the sessions. The results of the
video analysis will be presented in the next section.
Since we had not analyzed all the videos but only a part of them
we cannot give percentages on what we have seen, also because we
report here mostly observed patterns, so we will give trends of usage
that might be useful for the development of d-touch and other TUIs
with similar setup.
34
saw that everyone managed to produce a basic rhythm with few
blocks or that they enjoyed themselves with one or two blocks try-
ing to understand how the system is working. We will refer to this
latter kind of behavior calling it exploration of the interface. Al-
most everyone started the first time the application and explored
its possibilities moving one block in front of the camera and try-
ing to calibrate the system showing all the markers of the board
to the camera. Only a few minority started immediately with a
rhythm creation, probably users with a background in tangible in-
terfaces. The exploration of the interface might also happen with
the uncut paper with markers that we provided on the website. This
behavior leads to a high number of markers seen (around 30) for a
small amount of time and an indistinct noise. This kind of pattern
was observed multiple time as it is the easiest way to test if the
instrument is able to produce some sound only having printed a
paper. We have observed sometime that after this exploration the
board remains empty and markers appear slowly in the interface,
as probably the user were cutting markers and placing them in the
instrument as soon as they were cutted. These exploration patterns
are visible only in the firsts sessions of the users. Later sessions
start with the already calibrated setup and sometimes with already
some blocks in the interface, as if the setup was left untouched from
the previous experience in a fixed setup.
35
by the applications, leading to incorrect audio generation. Given
that the algorithm has been tested for five years and a tracking
filter actively smooth this behavior, we suppose that the lighting
conditions were really poor in these situations. After having ob-
served this phenomenon, we realized that we didn’t explain clearly
how to check if the recognition was happening correctly, beside a
small sign on markers visible on screen. Related to this lack we
realized also that we didn’t provide a sign on the GUI to indicate
that the calibration of the system was acting correctly.
36
Chapter 5
The focus of our research has been the observation of tangible in-
terface users in their everyday environments. Through interaction
logs we gathered information on their actual usage, but probably
the most interesting informations arrived from their informal com-
ments and from the videos and photos they shared with us. In this
chapter this type of content will be analyzed.
37
real usage scenario. Six videos were posted on YouTube1 , one photo
on Flickr2 , and one photo was sent to us via email. We found also
two hands-on reviews of the drum machine which included various
photos of the used setup. The contents that we have found were
from all around the world, mostly from USA, Germany, UK, Chile,
Brazil, Japan. We tracked the most interesting ones in our website3 .
38
Axial Codes Sentences
Actual Usage Feedback 305
Technical 202
Improve and extend 40
Audio 25
Physicality of the interface 21
Applications and real use 10
Field trial related 7
Generic Comments 286
Viral sharing 153
First impressions 118
Personal data sharing 15
Total 591
39
Actual Usage Feedback
or
40
and
Really enjoyed the glue, lentil and paper time though, re-
minds me how good it can be to get down and dirty with
materials.
41
Everyone in this house has now put together a radio-worthy
beat by pushing little scraps of paper around under a web-
cam.
The audio part of d-touch received the largest part of the criti-
cism. The audio synthesis part was too limited and without enough
features for a long engagement and for real musical use. Several
users requested the ability of sending MIDI or OSC signal out of
the drum machine, or the ability to load custom samples. One post
that we received on our forum is explicative of these problems for
musicians:
or
42
Software doesn’t have to mean virtualizing everything and
letting go of physical objects. On the contrary, it can create
all sorts of imaginative, new ways of mapping musical ideas
to the physical world. And that’s how we wind up with a
walnut drum sequencer.
It’s easy and it’s really fun. All other examples of similar
technologies involved being a computer genius and the step
by step instructions are just pain awesome for everyone, this
is the right way to do this kind of things, by making them
available and enjoyable to everyone.
43
This was often accepted well as the applications were freely down-
loadable and the setup was really cheap. We received also proposals
of small payments to remove the logging system to allow free usage
of the applications.
Generic Comments
44
Figure 5.2: This is the first video that
we received. It’s done with an old ver-
sion of the drum machine and the user
is hearing wrong sounds caused by a
bug in the audio card settings, that
we solved with later versions. We can
see that the user has a very small
space to place the board and it’s in
the middle of two different computers.
The markers have been done folding
the paper blocks that we proposed on
our website. This is a typical case of
technology interested user, since the
music is not reproduced correctly but
no comments are done on this.
The photos and the videos that we received were probably the most
informative part of our study, since we could see where the user
setup the system and how they used it. To analyze them we initially
created a set of questions to answer for each video and then we
observed all of them trying to answer our questions, that were:
What was used for the blocks? What kind of stand? Near the
computer? What kind of room? Are they on table? Audio setup?
Operating system? A summary of the analysis and other material
can be found on Appendix C.
45
Figure 5.3: This user built wooden
blocks to have a polished setup and
we can also hear in the video a very
good rhythm created in few seconds.
The ability and control of the inter-
face are good as if he practiced a bit
before doing the video. He is inter-
ested more in the musical part, since
while speaking in the video he always
refers to the drum machine and never
to the tangible interface.
46
the flat markers printed on paper. Sometimes they even used the
uncut strip of markers to construct repetitive patterns or to test
quickly the interface, as we also observed from the interaction logs.
On the opposite end the DIY lovers built a polished setup with
wooden blocks and professional tripods. We never saw the card-
board stand that we provided as a PDF, users preferred to hung
the webcam to shelves or to camera tripods. In some cases, the
interactive board was raised, on books, to bring it closer to the
camera.
The usage settings were very different, we saw small rooms, bed-
rooms, individual offices, a cubicle and a music studio. The hard-
ware and operating system used was varied: laptops, desktop com-
puter, Windows and OS X. Also audio systems were various, from
laptop speakers to professional hi-fi systems.
47
(a) d-touch drum machine put vertical and used col- (b) The tangible interface subverted and made virtual.
laboratively.
idea. Then the video shows her explaining the complete setup, how
she was using the classroom hardware to complete the setup and it
finishes with two fellow students using collaboratively the setup to
build a drum sequence.
48
Chapter 6
Discussion
We have analyzed so far the data separately, now we’ll try to get
a comprehensive view of all the data and get some inspirations for
future works.
The largest part of the users explored the interface and produced
basic rhythms. The creation of advanced patterns was infrequent
49
as probably the applications were seen as toys more than musical
instruments, as someone pointed out with their comments. The lack
of important features for musicians has been widely communicated
by the users as we reported previously.
50
open source third party libraries that we used to support webcams
and audio cards under different operating systems, together with our
software may create some incompatibility. Moreover we discovered
bugs related to hardware configurations only delivering the software
to our users, that helped us a lot in discovering and fixing them.
To sum up, the web distribution and user study of a tangible in-
terface has never been done before the d-touch experiment and at
the end we can say that it has been a success. Even if the applica-
tions were perceived mostly as toys than proper musical instruments
we received a huge media attention and users were really involved in
the project and helped us a lot to improve the applications. A large
number of users managed to use the instruments, even in presence of
technical difficulties, and they explored the possibilities of tangible
interfaces. About the interface we received very few comments and
all of them were positive and encouraging. We interpret this lack of
comments as an evidence that users found the interface “obvious”
to use, even if very few of them reported previous experiences with
TUIs. Maybe this is due to previous informations about TUIs gath-
ered on medias that in the last years reported a lot of works in this
field with a lot of emphasis as they did with us. Finally the cases
of user appropriation are signals of strong interest and advanced
understanding of this novel type of interface.
51
Chapter 7
Future Work
52
them to play more. The compositions could be shared just in form
of images that could be downloaded and printed as a musical score.
The MIDI, OSC or other output signal is not interesting from the
research point of view because it would lead to uncontrollable usage
that could not remotely studied. In any case, it’s already possible
to create this kind of interface with the d-touch recognition library,
freely downloadable from the web in open source1 .
Now with Adobe Alchemy2 it’s possible to port the entire recog-
nition library to Adobe Flash, allowing to reduce problems in hard-
ware compatibility and to make faster the setup of the system. If the
performance loss due to the Flash porting shows to be sustainable
from the performance point of view, this kind of approach would
make d-touch available to an even broader audience, directly from
their Internet browser.
1 http://sourceforge.net/projects/libdtouch/
2 http://labs.adobe.com/technologies/alchemy/
53
Chapter 8
Conclusions
54
two different approaches, to gain significant statistical numbers us-
ing the large user base and, on the other side, having the direct
exchange with users might be very useful to pose direct questions
and to see common behaviors. In our case we had to hypothesize
what our statistical analysis meant, while if we had the possibility of
asking questions directly to users we could get the precise answers.
But a positive consequence of this approach is that, being com-
pletely absent from the setting we can assure a low invasiveness and
a greater level of ecological validity of the experiments. With both
approaches would be possible to relate analyzed patterns, especially
on interaction log videos, with in-situ observations explaining much
better the gathered data.
55
Appendix A
UGC texts
A part of the analyzed text content with the open code used.
Table A.1: Some sentence collected with the related open code.
Source Text Category
Mail I think with projects like this the educational
kids will having a lot of fun to per-
form handicraft work & making ex-
periments with music.
Blog in pure style Do-It-Yourself (proba- DIY Love
bly the most appealing part of the
project, from the user perspective)
Blog Bored with mouse pushing and knob low cost ap-
twiddling? The d-touch tangible preciation
sequencer/drum machine makes a
cheap interface (with free download-
able software) for assembling se-
quences.
Blog yes, drums are tangible. We know. tangible
What this is, however, is a tangible interface
interface that is a drum machine. awareness
Continued on next page
56
Table A.1 – continued from previous page
Source Text Category
Blog Global economic recession cutting low cost ap-
into your gear purchases? The team preciation
at d-touch.org may have a solution:
a drum machine that you print with
your computer printer.
Mail All the musicians I showed the unit not good de-
to, commented on how sterile and sign
clean the board looked.
Mail And i’d say you have to be on the net Internet
because the software cost $0. availability
issue
Mail I just tried registering on the d-touch personal data
website and it didn’t like my rather complain
too expansive answer to ”how often
do you play”. It still went ahead
and created the account (username:
Dunx)
Mail Most of my friends used the heavier tangible
objects for the lower sounds instinc- appreciation
tively (metal and glass for kicks and
toms) and the wood for snare and pa-
per for the hats and cymbals.
Forum Had some problem setting the light- lighting prob-
ing (some beat inputs skipped) lem
Forum I will try to improve my setup and Try-to-
experiment a bit with velcro... Will improve
keep you posted. Beesh
Forum Hi my names Gideon and I’m from social sharing
England. I live in a small town called
Chichester which is near Brighton.
57
Appendix B
59
Table B.1 – continued from previous page
Category Entries
Audio 25
MIDI 11
use on stage 4
custom sounds 2
music tool 4
tempo selection 1
not good design 1
audio complexity 1
time issuesv1
Field trial related 7
research interest 1
Internet availability issue 5
personal data complain 1
Total 591
60
Appendix C
In this appendix are shown all the sources of our analysis on photo
and video UGCs.
DESCRIPTION:
-What was used for the blocks?
The blocks have been made with the proposed shape.
61
-Are they on table?
Yes, but in a very small and awkward position. The board is placed
diagonally (probably to fit the webcam position) and the space is
very small, between a monitor, a laptop and a multimeter...
-Audio setup?
Unknown probably laptop speakers.
-Operating system?
Windows on a laptop.
DESCRIPTION:
-What was used
for the blocks?
Plain markers has
been glued on mag-
nets.
-What kind of
stand?
The webcam has
been duct taped
to a music stand
that pointed to the Figure C.1: The vertical setup of d-touch
magnetic whiteboard.
62
-Are they on table?
No, the setup is on a vertical magnetic whiteboard. Also the inter-
active board has been put vertical with magnets.
-Audio setup?
External M-Audio FireWire connection.
-Operating system?
Os X on a MacBook Pro 15”.
DESCRIPTION:
-What was used
for the blocks?
Nothing!
-What kind of
stand?
Cup filled with cof-
fee beans...
-Audio setup?
63
Internal audio card, or nothing special
-Operating system?
Windows on a desktop computer.
DESCRIPTION:
-What was used for the blocks?
Wooden blocks with paper glued on it.
-Audio setup?
Laptop speakers?
-Operating system?
OS X on a MacBookPro 15”
DESCRIPTION:
64
-What was used for the blocks?
Paper blocks and simple paper markers. Mostly the simple markers
for actual playing.
-Audio setup?
Not seen but probably with an external audio card. We see some
brand of musical instruments (Alesis).
-Operating system?
OS X on a MacBookPro 15”.
DESCRIPTION:
-What was used for the blocks?
Plain paper, no block is constructed.
65
Yes, on the same table.
-Audio setup?
Computer speakers.
-Operating system?
Windows on a desktop computer.
http://www.flickr.com/photos/85596380@N00/
3689219303/
DESCRIPTION:
-What was used for the blocks?
Plain paper.
66
-Audio setup?
A computer speaker is visible in the top of the photo.
-Operating system?
Windows on a laptop
http://www.forest.impress.co.jp/img/wf/docs/307/274/
html/image2.jpg.html
DESCRIPTION:
-What was used for the blocks?
Simple paper and some blocks. Some marker is glued on some small
object not recognizable.
-Audio setup?
Unknown, probably headphones.
-Operating system?
Windows Vista.
http://www.pc-music.com/content/d-touch-paper-drum-
machine-full-hands-review
67
DESCRIPTION:
-What was used for the
blocks?
Paper blocks and simple
paper markers.
-Audio setup?
Probably connected to an external audio card (visible in a photo).
-Operating system?
Windows Vista.
68
Appendix D
Riassunto in Italiano
Questa tesi descrive il mio lavoro svolto al Media and Design Lab-
oratory dell’EPFL del Prof. Jeffrey Huang tra Marzo e Settembre
2009, supervisionato da Enrico Costanza. Lo scopo del progetto è
stato quello di studiare l’interazione di un largo numero di utenti
con un tipo di interfaccia ancora poco studiato, l’interfaccia tangi-
bile. Per fare questo abbiamo usato due strumenti musicali, basati
su interfaccia tangibile, realizzati nel 2003 da Enrico Costanza e
Simon Shelley, d-touch drum machine e d-touch sequencer, li abbi-
amo resi scaricabili da Internet, facilmente riproducibili dagli utenti
e, dopo aver raccolto log di interazione e materiale prodotto dagli
utenti e distribuito su Internet, abbiamo studiato come le persone
si rapportano a questo genere di interfaccia nel loro ambiente e per
un periodo di utilizzo prolungato.
D.1 Introduzione
69
gettazione, valutazione e implementazione di sistemi computazionali
interattivi usati da persone e anche come lo studio di tutti i fenomeni
che gravitano attorno a questo [1].
70
Le TUI sono sempre state sperimentate e studiate in ambienti
controllati come laboratori e musei e sempre in numeri ristretti per
le difficoltà tecniche di riproduzione. Con d-touch abbiamo cercato
di risolvere questo problema creando e distribuendo su Internet una
TUI a bassissimo costo. Per studiare l’utilizzo di questa interfaccia
abbiamo registrato i movimenti prodotti dagli utenti con le appli-
cazioni e abbiamo raccolto materiale trovato su Internet a proposito
di d-touch, ispirati da un modo di studio già sperimentato in HCI
negli ultimi anni [15, 16, 17].
71
D.2.1 Come funziona
72
D.2.2 d-touch Drum Machine
73
D.2.4 Sistema di Registrazione Remota
74
l’uso di centinaia di utenti, talvolta anche più utenti simultanea-
mente, apparentemente senza perdere dati.
D.2.5 Distribuzione
75
D.2.6 Il Lancio Online
76
meno di quello del primo lancio, ma dati i tempi più lunghi, cioè
fino al 15 Dicembre 2009, gli utenti registrati sono stati 1252, di cui
389 hanno provato l’interfaccia e 273 hanno suonato per almeno un
minuto, risultando in 199 ore di uso registrato.
77
vedere che l’uso è stato compresso nella maggioranza dei casi in
meno di due ore.
78
i marker non tagliati, come se avessero stampato il PDF pubbli-
cato sul sito, senza ulteriore lavoro. Questo comportamento porta
ad un alto numero di marker riconosciuti (circa 30) per un breve
tempo avendo come risultato un rumore indistinto. Questo compor-
tamento è stato individuato più di una volta, in quanto è il modo
più rapido per controllare il funzionamento dell’applicazione pro-
ducendo qualche suono. Abbiamo anche osservato che dopo questo
test spesso si nota un certo tempo l’interfaccia vuota e poi che poco
a poco, lentamente, si riempie, come se l’utente stesse tagliando i
marker e ponendoli sotto la webcam uno alla volta. Questi pat-
tern esplorativi sono stati visti solo nella prima sessione di ciascun
utente. Le sessioni successive spesso iniziavano già calibrate o con
dei marker già presenti, come se il setup fosse stato lasciato intatto
dall’ultimo utilizzo.
79
In generale abbiamo osservato interessanti sessioni ritmiche quando
si aveva la calibrazione e buone condizioni luminose. Negli altri
casi gli utenti erano in grado soltanto di esplorare l’interfaccia rap-
idamente e quindi dopo breve tempo abbandonarla senza ottenere
niente di interessante.
80
Giappone. I più interessanti possono essere trovati sul nostro sito12 .
81
possono essere trovati alla Sezione 5.2.1 e altri esempi nelle Appen-
dici A e B.
82
ottimali i risultati musicali erano poco interessanti e neanche troppo
ricercati dagli utenti, che puntavano più al lato tecnico o divertente
dell’applicazione.
Due video erano di particolare interesse, dato che sono stati os-
servati casi di appropriazione, da parte degli utenti, della tecnologia
in modo radicale. Il primo esempio era in un’università, dove un
utente ha realizzato d-touch 8 volte più grande del normale e l’ha
attaccato in verticale su una lavagna magnetica con dei magneti
da ufficio, come ha fatto con i marker, attaccandoli su piccoli mag-
neti. In questo modo d-touch è stato usato in verticale, in maniera
collaborativa da più persone, come si può vedere nel video.
D.5 Discussione
83
ziona. Un alto numero di persone è riuscita a farlo funzionare, senza
il nostro intervento fisico. d-touch è stato usato in diversi ambienti e
ha dimostrato un elevato grado di versatilità. Il concetto di webcam
e di area interattiva ripresa dall’alto con blocchi mobili funziona ed
è ben compreso. Un punto negativo è stato l’algoritmo di riconosci-
mento che non si è dimostrato sufficientemente robusto. Nonostante
gli anni di sperimentazione, essere usato in ambienti non controllati
è qualcosa che non può essere sperimentato prima. Anche la cali-
brazione dell’area interattiva non ha dato ottimi risultati e spesso
il sistema produceva suoni che non rispecchiavano la configurazione
fisica a causa di questi due problemi. Con il presentarsi di questi
problemi ci siamo resi conto che il sistema forniva piccoli indizi sul
fatto che l’illuminazione fosse corretta o il sistema calibrato, questo
perché per noi erano fatti scontati dei quali non ci siamo preoccu-
pati.
I dati nella Tabella 4.1 supportano l’idea che gli strumenti siano
stati percepiti più come dei giochi che come dei veri strumenti musi-
cali. Si può vedere ciò dal fatto che la drum machine, più semplice è
stata preferita al sequencer (65% contro il 35%) e il tempo trascorso
tra il primo e l’ultimo utilizzo raramente supera i due giorni, solo il
21% degli utenti ha usato le interfacce più a lungo. Inoltre il fatto
che sul sequencer siano stati utilizzati più blocchi spinge a pensare
che lo strumento più complesso è più interessante per usi musicali,
anche se meno interessante dal punto di vista mediatico.
84
lo schermo sul quale visualizzava i marker tramite un programma
di grafica (simile ad Adobe Illustrator). Da questi casi si può capire
che gli utenti hanno capito bene il concetto di interfaccia tangibile
e l’hanno adattato ai loro interessi negli scenari più diversi.
85
prossimo futuro.
D.7 Conclusione
86
Il contributo di questa tesi allo studio dell’interazione uomo-
macchina è dato dal nuovo metodo di osservazione utilizzato, che
nonostante sia ancora da rivedere e migliorare, ha dato buoni risul-
tati tenendo conto dei ridottissimi costi che ha comportato se lo
compariamo ai metodi tradizionali di studio sulle interfacce tangi-
bili. Probabilmente il metodo potrebbe essere migliorato notevol-
mente se fosse affiancato ad un’osservazione più tradizionale degli
utenti, che a noi non è stata possibile per la dimensione ridotta
del nostro gruppo di ricerca. Con l’uso di entrambi gli approcci si
sarebbe potuta eliminare una parte di speculazione sui dati che ha
portato a risultati non sicuri, ad esempio quando i dati statistici
erano contraddittori. Con alcune interviste a campione avremmo
capito meglio dei comportamenti ricorrenti che avrebbero semplifi-
cato di molto l’analisi e che avrebbero portato a risultati più affid-
abili.
87
Bibliography
88
[8] D. Xu, J. C. Read, E. Mazzone, and M. Brown, “Designing and
testing a tangible interface prototype,” in IDC ’07: Proceedings
of the 6th international conference on Interaction design and
children, (New York, NY, USA), pp. 25–28, ACM, 2007.
[9] P. Marshall, “Do tangible interfaces enhance learning?,” in TEI
’07: Proceedings of the 1st international conference on Tangible
and embedded interaction, (New York, NY, USA), pp. 163–170,
ACM, 2007.
[10] T. Drori and M. Rinott, “Pixel materiali: a system for creating
and understanding pixel animations,” in IDC ’07: Proceedings
of the 6th international conference on Interaction design and
children, (New York, NY, USA), pp. 157–160, ACM, 2007.
[11] S. Jordà, G. Geiger, M. Alonso, and M. Kaltenbrunner, “The re-
actable: exploring the synergy between live music performance
and tabletop tangible interfaces,” in TEI ’07: Proceedings of
the 1st international conference on Tangible and embedded in-
teraction, (New York, NY, USA), pp. 139–146, ACM, 2007.
[12] K. Ryokai, S. Marti, and H. Ishii, “I/o brush: drawing with ev-
eryday objects as ink,” in CHI ’04: Proceedings of the SIGCHI
conference on Human factors in computing systems, (New York,
NY, USA), pp. 303–310, ACM, 2004.
[13] J. Patten, B. Recht, and H. Ishii, “Audiopad: a tag-based inter-
face for musical performance,” in NIME ’02: Proceedings of the
2002 conference on New interfaces for musical expression, (Sin-
gapore, Singapore), pp. 1–6, National University of Singapore,
2002.
[14] G. Zufferey, P. Jermann, A. Lucchi, and P. Dillenbourg, “Tin-
kersheets: using paper forms to control and visualize tangible
simulations,” in TEI ’09: Proceedings of the 3rd International
Conference on Tangible and Embedded Interaction, (New York,
NY, USA), pp. 377–384, ACM, 2009.
[15] M. Blythe and P. Cairns, “Critical methods and user generated
content: the iphone on youtube,” in CHI ’09: Proceedings of the
89
27th international conference on Human factors in computing
systems, (New York, NY, USA), pp. 1467–1476, ACM, 2009.
[16] A. Kittur, E. H. Chi, and B. Suh, “Crowdsourcing user studies
with mechanical turk,” in CHI ’08: Proceeding of the twenty-
sixth annual SIGCHI conference on Human factors in comput-
ing systems, (New York, NY, USA), pp. 453–456, ACM, 2008.
[17] M. V. Kleek, M. Bernstein, K. Panovich, G. Vargas, D. Karger,
and M. C. Schraefel, “Note to self: Examining personal infor-
mation keeping in a lightweight note-taking tool,” in ACM CHI
2009, 2009. Nominated, best in CHI Award, ACM CHI 2009.
[18] M. Weiser, “The computer for the 21st century,” Scientific
American, September 1991.
[19] M. Weiser and J. S. Brown, “Designing calm technol-
ogy.” http://www.ubiq.com/hypertext/weiser/calmtech/
calmtech.htm, 1995.
[20] J. Underkoffler and H. Ishii, “Urp: a luminous-tangible work-
bench for urban planning and design,” in CHI ’99: Proceedings
of the SIGCHI conference on Human factors in computing sys-
tems, (New York, NY, USA), pp. 386–393, ACM, 1999.
[21] J. Y. Han, “Low-cost multi-touch sensing through frustrated
total internal reflection,” in UIST ’05: Proceedings of the 18th
annual ACM symposium on User interface software and tech-
nology, (New York, NY, USA), pp. 115–118, ACM, 2005.
[22] P. Bennett and S. O’Modhrain, “The beatbearing: a tangi-
ble rhythm sequencer,” in Proceedings of NordiCHI 2008: 5th
Nordic Conference on Computer-Human Interaction (electronic
proceedings), 2008.
[23] H. Kato and M. Billinghurst, “Marker tracking and hmd cal-
ibration for a video-based augmented reality conferencing sys-
tem,” in IWAR ’99: Proceedings of the 2nd IEEE and ACM
International Workshop on Augmented Reality, (Washington,
DC, USA), p. 85, IEEE Computer Society, 1999.
90
[24] M. Gandy, B. Jones, S. Robertson, T. O‘Quinn, and A. John-
son, “Rapidly prototyping marker based tangible user inter-
faces,” in VMR ’09: Proceedings of the 3rd International Con-
ference on Virtual and Mixed Reality, (Berlin, Heidelberg),
pp. 159–168, Springer-Verlag, 2009.
[25] E. Costanza and J. Robinson, “A region adjacency tree ap-
proach to the detection and design of fiducials,” in VVG,
pp. 63–69, 2003.
[26] R. Bencina, M. Kaltenbrunner, and S. Jorda, “Improved topo-
logical fiducial tracking in the reactivision system,” in CVPR
’05: Proceedings of the 2005 IEEE Computer Society Confer-
ence on Computer Vision and Pattern Recognition (CVPR’05)
- Workshops, (Washington, DC, USA), p. 99, IEEE Computer
Society, 2005.
[27] E. Costanza and J. Huang, “Designable visual markers,” in
CHI ’09: Proceedings of the 27th international conference on
Human factors in computing systems, (New York, NY, USA),
pp. 1879–1888, ACM, 2009.
[28] M. Fjeld, J. Fredriksson, M. Ejdestig, F. Duca, K. Bötschi,
B. Voegtli, and P. Juchli, “Tangible user interface for chem-
istry education: comparative evaluation and re-design,” in CHI
’07: Proceedings of the SIGCHI conference on Human factors in
computing systems, (New York, NY, USA), pp. 805–808, ACM,
2007.
[29] M. Waldner, J. Hauber, J. Zauner, M. Haller, and
M. Billinghurst, “Tangible tiles: design and evaluation of a tan-
gible user interface in a collaborative tabletop setup,” in OZCHI
’06: Proceedings of the 18th Australia conference on Computer-
Human Interaction, (New York, NY, USA), pp. 151–158, ACM,
2006.
[30] Y. Jung, J. Blom, and P. Persson, “Scent field trial: under-
standing emerging social interaction,” in MobileHCI ’06: Pro-
ceedings of the 8th conference on Human-computer interaction
91
with mobile devices and services, (New York, NY, USA), pp. 69–
76, ACM, 2006.
[31] H. Sharp, Y. Rogers, and J. Preece, Interaction Design: Beyond
Human-Computer Interaction. Wiley, 2 ed., March 2007.
[32] E. Costanza, S. Shelley, and J. Robinson, “D-touch: A
consumer-grade tangible interface module and musical appli-
cations,” in Proceedings of Conference on HumanComputer In-
teraction (HCI03), 2003.
[33] E. Costanza, S. Shelley, and J. Robinson, “Introducing audio
d-touch: A tangible user interface for music composition and
performance,” in Proc. of the 6th Int. Conference on Digital
Audio Effects (DAFX-03), 2003.
[34] J. Raskin, The humane interface: new directions for designing
interactive systems. New York, NY, USA: ACM Press/Addison-
Wesley Publishing Co., 2000.
92