MULTIMEDIA APPLICATION WITH INTERACTIVE DIGITAL

ANIMATION FOR MUSIC PERFORMANCES

Nuno Correia

Instituto Superior de Tecnologias Avançadas (ISTEC),

Av. Engº Arantes e Oliveira, 3 r/c 1900-221 Lisboa, PORTUGAL, correia.nuno@netcabo.pt

Abstract: The use of multimedia with performance arts has a rich history. This paper proposes
an innovative approach for the use of a Multimedia Application with music performances,
through the integration of interactive digital animation. The Application and its development are
described. The interface, which is part of the visual experience produced by the Application,
consists of two rows and two columns of controls, positioned in the edges of the screen. The
Application was submitted to an evaluation, which had encouraging results. The evaluation
method is presented and its results are interpreted. Future developments are discussed.

Key words: Multimedia applications, interactive animation, digital tools for performance arts

1. INTRODUCTION

The concept of integrated, interactive media has a long history, spanning more than 150
years [1]. Performance arts have been, throughout this period, privileged laboratories to conduct
this kind of integration experiments. Packer and Jordan [1] trace the origin of multimedia to
Wagner, who proclaimed that “No one rich faculty of the separate arts will remain unused in the
United Artwork of the Future; in it will each attain its first complete appraisement. (…) The place
in which this wondrous process comes to pass is the Theatrical stage; the collective artwork
which it brings the light of day, the Drama” [2]. This citation stresses the importance of the
performance arts to the emergence of multimedia.
The genesis of cinema provided another territory for multimedia explorations: the
“dynamic screen” [3]. And the advent of the computer opened up an even greater number of
possibilities. But they both also provided two more “tools” for performance arts to use in its
artistic “palette”. Throughout the last century, many were the authors that put into practice the
interplay of projected moving images with performance arts. Some, like Bauhaus’ László
Moholy-Nagy [4] and John Cage [5] updated Wagner’s concept of performance arts as
Gesamtkunstwerk [2] (total artwork) to encompass moving images. Others, like Nam June Paik
[6], Roy Ascott [7] and Myron Krueger [8] extended this concept into the computer age.
The last century has been witness to many attempts to explore the new “tools” in the
“palette” of performance arts. In the last decade, for instance, a great number of software tools
were marketed towards the real-time manipulation of video for live performances. What new
territories are there, then, left to explore? One answer could be interactive digital animation – that
is, the real-time creation of visual elements through the juxtaposition of pre-defined and
mathematically generated vector graphics. There are a number of pioneer contemporary artists,
most notably John Maeda [9], who have been exploring this new area, but mainly for Web Art
and Installation Art purposes – not for performance arts. This paper describes and analyses a
Multimedia Application that intends to use digital animation with music performances, adopting
the former to the latter’s “tools palette”. The aim of the Application is to provide flexible visual
resources to music performances, which allow for the real-time creation of animations, based on
pre-existing building blocks.
The next sections will describe how the Application works, how it was evaluated and the
evaluation results.

2. DESCRIPTION OF THE APPLICATION

2.1 The Development of the Application

The development of the Application is a co-operative work between designer and

programmer. Their purpose was to build an interactive Application for controlling digital
animation to use along with music performances, through the use of a video projector. This
Application would allow for the control of different types of animated and “behavioural”
modules that could be combined to create, in real time, a unique visual “experience” for each
event. To achieve this effect, a great number of animation modules had to be created, separated
into 4 types of animation modules: “main animations”, “animated icons”, “curtains” and
“interferences”. These modules could be called up, manipulated and programmed during the
performance by a “user”. Therefore, the “user” of the Application would have a great deal of
creative freedom during the event. This “user” would also have the role of “composing”, or
“sequencing”, in real-time, a digital animation suited to the performance.

2.2 The Interface of the Application

The interface of the Application is presented in the 4 edges of the screen. Each edge has a
different function. The interface is visible to the audience, and is part of the visual experience.
Thus, the audience sees how the “user” of the Application is building, in real-time, the digital
animation. The Application is divided in 4 layers. The main animation is always in the bottom
layer (layer 4) and the animated icons are all in the front layer (layer 1). The interferences and the
coloured curtains can be allocated to layers 3 or 2.
The user can choose between different “sets” of animations. Each set has its own group of
“main animations”, “animated icons”, “curtains” and “interferences”. Each set of animations can
correspond to a “song” or to a “theme”. A set of animation can be assigned to a key on the
computer keyboard.
The top edge contains the controls for the full screen animations – entitled “main
animations”. A row of buttons allows for the choice of animation. Different keys allow for the
control of the playback of the animation (backwards/forwards, acceleration/deceleration).
The bottom edge contains the controls for the “animated icons” – small vectorial
animations that appear on top of other elements in different nodes of a pre-defined grid (9x9).
The animated icons are positioned by choosing the type of icon in the bottom edge and choosing
the location in the grid. Otherwise, the icons can be dragged to their intended positions. Once in
their positions, the icons can be deleted individually. The size and opacity of the icons can be
altered.

Fig. 1 Choosing an animated icon (detail)

Fig. 2 An animated icon after having been dragged into the grid

The left edge contains the controls for the “interferences” – abstract vectorial animations,
composed by clusters of small animated elements, that have behaviours with parameters that can
be defined in real-time. These parameters are: colour (or range of colours); density (amount of
this type of objects on the screen); direction of movement; speed of movement; size of each
element; randomness of position. Examples of “interferences”: rain (clusters of different sized
dashes), TV static.
The right edge contains the controls for the “curtains” – sheets of colours and abstract
textures, with the size of the screen, and customizable opacities, Example: strobe-type curtain;
glowing curtain; 3D texture.
 Fig. 3 An animated icon and a curtain after having been activated

Fig. 4 Animated icons and a main animation after having been activated (detail)

3. EVALUATION METHOD

3.1 The Experiment

At the time of writing, the Application has been concluded and tested with a small
audience, in the context of an electronic music concert performance. The “user” of the
Application was a reputed Portuguese illustrator and animator. The audience consisted of
illustrators, animators, musicians, performance artists and other guests.
At the end of the performance, a small questionnaire was distributed to the members of the
audience, with the following questions: a) “Did the digital animations contribute to the overall
enjoyment of the performance?”; b) “Was the integration of the digital animations in the
performance effective?” c) “What are your suggestions concerning improvements to the
Application?”. The aim of question a) was to test the pertinence of the use of digital animation
together with the performance; the aim of question b) was to test the effectiveness of the
interaction between the Application and the performance; question c) intended to gather
suggestions for further developments. The “user” was also asked questions b) and c).

3.2 The Results

The audience showed enthusiasm during and after the test performance. The answers to
questions a) and b) were overwhelmingly positive. The “user” of the Application stated that the
Application had been extremely effective in its integration with the performance and that it had
allowed him to express his reactions to the performance fluently.
Regarding question c), the “user” suggested that more main animations could be added to
the Application, to facilitate a more flexible artistic expression. The “user” also suggested that the
application should be handled tactfully, in order to avoid using too many elements at a time.

4. CONCLUSION

The Application is innovative and unlike any tool in the market for use with music
performances. This uniqueness is due to: a) the type of media used – digital animation; b) the
degree of control it allows; c) the complexity of the programming of the different animation
modules; d) the interface of the Application, which allows the audience to see how the “user” is
composing the visual experience.
As the test results have shown, the Application has room for improvement. Namely, the
stock of animation modules can be significantly increased. In order to avoid adding too much
complexity to the interface, this increase should be accompanied by a greater modularity of the
software. These modules should be available through downloadable upgrades. These
developments are already taking place.
Apart from these considerations, the Application has showed to be a promising step
towards a new kind of integration between a Multimedia Application and Music Performances.
Its main application is music performances, but it can be suited to theatre, dance, art happenings,
VJing and installations.

5. ACKNOWLEGEMENTS

The authors would like to thank illustrator and animator André Carrilho for his invaluable
ideas and suggestions for the Application, and congratulate him for winning USA Society for
News Design 2002 Gold Award for Illustrator’s Portfolio.

REFERENCES

[1] R. Packer, K. Jordan, Multimedia: From Wagner to Virtual Reality , W. W. Norton &
Company, London, 2001
[2] R. Wagner, Outlines of the Artwork of the Future, in R. Packer, K. Jordan, Multimedia: From
Wagner to Virtual Reality, W. W. Norton & Company, London, 2001
 [3] L. Manovitch, The Language of New Media, MIT Press, Cambridge, 2001
[4] L. Moholy-Nagy, Theatre, Circus, Variety , in R. Packer, K. Jordan, Multimedia: From
Wagner to Virtual Reality, W. W. Norton & Company, London, 2001
[5] J. Cage, Diary: Audience 1966 , in R. Packer, K. Jordan, Multimedia: From Wagner to
Virtual Reality, W. W. Norton & Company, London, 2001
[6] N. J. Paik, Cybernated Art , in R. Packer, K. Jordan, Multimedia: From Wagner to Virtual
Reality, W. W. Norton & Company, London, 2001
[7] R. Ascott, Behaviouristic Art and the Cybernetic Vision , in R. Packer, K. Jordan,
Multimedia: From Wagner to Virtual Reality, W. W. Norton & Company, London, 2001
[8] M. Krueger, Responsive Environments, in R. Packer, K. Jordan, Multimedia: From Wagner
to Virtual Reality, W. W. Norton & Company, London, 2001
[9] J. Maeda, Maeda@Media, Thames & Hudson, London, 2000