Digital media and performance art, with a focus on motion interpretation in virtual spaces through computer vision and

motion tracking By Emma Benany

Digital media and performance art, with a focus on motion interpretation in virtual spaces through computer vision and motion tracking by Emma Benany Supervised by Prof. Daniel Fetzner and co-supervised by Bernd Dudzik

This thesis has been submitted in fulfilment of the requirements for the award of a Bachlors degree in Multimedia and Information Design, the faculty of Applied Scienses and Arts, The German University in Cairo, Spring10

Special thanks and respect: Dalia Hassan, Shaymaa Shoukry, Yves Degoyon, David Pello Gonzalis, Ahmad El Azma for their efforts and contribution.

Table of Contents

Abstract 1] Introduction 1.1] The extension of man into the virtual world 1.2] Digital Media and Performance Art 1.3] Digital motion mapping techniques 1.4] About Computer Vision

1 2 3 5 6 9

4.2] Exp. Two: Basic Motion Printing 4.3] Exp Three: Color filtering and Printing 4.4] Exp Four: Fellow Shadow 5] Methodology of the Final Installation 5.1] Ideas and Sketches 5.2] The final concept 5.3] Documentation of the Design Process 5.3.1] Building and Testing the Prototype 5.3.2] Sound and Software Design 5.3.3] Building and Testing the Installation 5.4] Testing the Installation

28 31 35 41 42 44 46 46 47 52 55

2] Review on Related Projects and Artists. 2.1] David Rockeby 2.2] Biped, 1999 3] The Design of the Artistic Project 3.1] Preparations 3.1.1] Software 3.1.2] Testing the Software 3.2] Conversations with Shymaa Shoukry. 4] Experimental Videos 4.1] Exp. One: Motion Blur

12 13 15 17 18 19 19 21

6] Conclusion 7] References

58 59

24 25

Abstract

The design of this research project was to create a process of experimenting with tools related to computer vision and interactivity along with the human body, in an attempt of creating environments that combine both elements in a harmonious and visually aesthetic ways. Dealing mainly with the keywords space and time on both dimensions, the real and the virtual, later combining them to others such as digital double, fellow shadow and space-less. The main point about the process is to speculate about the relation and attitude we adopt towards the machine according to its level of reality, in terms of visual and audio manipulation and self-representations. At the end of this process yet not more significant than the rest, I was to design an interactive installation to be exhibited at the Townhouse - Cairo, in June 2010. Keywords: interactivity, digital media and performance art, space, time, real, virtual, digital double, sound manipulation.

Introduction

The Extension of Man into the Virtual World Digital Media and Performance Art Digital Motion Mapping Techniques Computer Vision- OpenCV

The Extension of Man into the Virtual World

Through out the last couple of decades computer environments have taken a massive role in everyones daily life, starting from social systems to industry, science, art and design. Man has extended his senses into the technologies that grow more sophisticated each day, and form more complex cybernetic systems that involve not only humans, but also highly intelligent machines. No doubt that the over exposure to virtual platforms either by gaming or social networking like the MUDs, IRC and facebook, has given birth to what is called by the digital double or the digital multiple. Through virtual realities we explore our various identities, which so often disconnect us from our real world. It offers a host to our senses, memory, fantasies, social connections and even our bodies through the cyborg technologies. But our schizophrenic behavior with computers is not new to the human nature; through out our history we have restlessly sought to fantasize about being heroes, gods and living in Utopia. Its clear in an immense amount of literature from

all cultures and eras, in all art forms man has visualized his demons and angels and abstracted his conflicts and fears. Also in our social communications, we develop multiple identities and mold various castes for public self-presentation, as Erving Goffman in his book The Presentation of Self in Everyday Life (1956) suggests that we are continuously gliding in between performances for the best self-presentation. Just like any other medium, computers or artificial interactive environments offer many other complex- playful ways of exploring the identities we behold. Nevertheless, it is considered to be quite tricky that some people substitute our worldly experience with the overwhelming- infinite possibilities of the safe- controlled virtual one. As well we develop new anxieties and conflicts resulting from the excessive selfindulgence to this world, as Marshall McLuhan in his book Understanding Media the Extension of Man stated If the nineteenth century was the age of the editorial chair, ours is the century of the

psychiatrists couch. Steve Dixon in his book Digital Performance categorizes the digital doubles we develop through out the process of interacting with virtual environments or digital performances into four main types. First is the reflection double, which is any type of art or platform that mirrors the exact image of one-self onto another on a screen. Putting the user into a position of a narcissist studying himself, as in an early interactive installation in 1993 by Monica Fleischman and Wolfgang Strauss Liquid Views, (figure 1.1) and facebook the social networking platform. Second is the alter ego type, which is the higher image of one-self, or an alternative double to the real- self that looks exactly the same but still is not. Usually it is the form in which the user draws upon to blame or confront himself. The third type is the spiritual emanation, which mainly releases the body from its materialistic state to a rather abstract form onto the screen, as in a

Group, transforming the body to ASCII characters. (figure 1.2) The last and most commonly used form is the manipulate manikin or the puppet double, like the digital avatar. This avatar is represented glamorously to fit into the fantasies of the user of becoming superhuman or even not human at all. Many examples like Second Life, and i m v u along with other Multi User Dimension or MUD online platforms offering digital residencies for multi-users. (figure 1.3.) (figure 1.4)

Fig1.1 Liqiud Views 1993, Monika Flieschman & Wolfgang Strauss

Fig1.3 World of War Craft, MUD environment

Fig1.2 ASCII performance by the Dancetech group

Fig1.4 Second Life, MUD environment

Digital Media and Performance Art

Upon this radical transformation in media through out the last twenty-five years, a new genre of art has formed with the same aesthetics of being more democratic, user-interactive and hybrid. New media art stretches to include Internet art, digital art, computer graphics, computer animation, virtual art, interactive art, etc. Mainly being concerned with telecommunication, mass media and augmented realities, these practices could be applied in diverse forms of installation art, video art, conceptual or performance art. Now combining the new media approaches or the digital media along with the traditional performance art, which is basically any artwork that constitutes of physical actions of a person or many, in a certain place and time. Broadly identified during the 1960s, and the like sort of works that emerged from the visual arts and imbedded avant-garde ideals and high concept live performance or recorded material on film or video. The result is Digital Performance, which broadly includes all performances that involve

Fig1.5 Merce Cunningham

computer technologies as a key role to the concept of the whole artwork. This could be theater, dance or performances that include projections of computer generated graphics, or images that are manipulated by the participants, robotics or cyborg technologies, virtual performances, installations and theatrical works that involve computer sensing equipments or techniques for different event activation. All these diverse applications of combining new media art and performance art has led to the clear blurring of the boarders between disciplines like interactivity, scriptwriting, installation art, theater, dance, science, telecommunication, design, performance and video. Rather computer environments or applications have become more of a means to an end not the end itself. More Examples about Digital Performance artists are discussed in chapter 2.

Fig1.6 Stelarc, cstom-built robotic Third Hand

Digital Motion Mapping Techniques

As digital media merges with performance, it becomes essential to record or map motion of the performer, for triggering events, manipulating images, abstracting the body movements or mapping the motion history. In 1995, Scott Dyer, Jeff Martin, and John Zulauf explain that motion capture "involves measuring an object's position and orientation in physical space, then recording that information in a computer-usable form. Objects of interest include human and non-human bodies, facial expressions, camera or light positions, and other elements in a scene." In this discipline, many technologies have been developing from the mid 70s up to this day, which have been employed mainly in the fields of music, fine arts, dance/performance, sign language, gesture recognition, rehabilitation/medicine, biomechanics, special effects for live-action films, and computer animation of all types. Motion capturing or mocap in abbreviation, could be done in various techniques deriving from two main approaches, relaying either on electronic hardware based

technologies, or on programming software/computer vision based ones. Following is a summary of the three main types of mocap- hardware based techniques used in entertainment and the ways in which they work. Later, is a more detailed section about mocap- computer vision based techniques used in the practical part of this project. 1. Mechanical Mocap: Mechanical motion capturing systems directly track body joint angles and are often referred to as exo-skeleton motion capture systems. The performer wears a human-shaped set of straight metal pieces that are hooked onto the performer's back; as the performer moves, this exo-skeleton is forced to move as well and sensors in each joint feel the rotations. Other types of mechanical motion capture involve gloves or mechanical arms. What is good about this technique is that there is no interference from light or magnetic fields. Nevertheless, the technology has no awareness of ground, so there can be no

Fig1.7 Bedford Interactive 1994, using overlaied garaphics to trace motion

jumping, as well feet data tends to be inaccurate, the equipment must be calibrated often, and unless there is some other type of sensor in place, it does not know which way the performer's body is pointing, and absolute positions are not known but are calculated from the rotations. (figure 1.8) 2. Optical Mocap: The performer wears reflective dots that are followed by several cameras and the information is triangulated between them. Usually the markers are either reflective, such as a system manufactured by Vicon or MotionAnalysis. This Technique is developed mostly in the fields of biomedical applications, sports injuries, analysis of athletic performance. Optical systems are often preferred over the mechanical ones for several reasons; performer feels free to move due to no cables connecting the body to the equipment, larger volumes are possible, multiple performers are possible and very clean, detailed data input. However, it is prone to light interference, reflective dots

Fig1.8 Mechanical Mocap suite

can be blocked by performers or other structures, causing loss of data, but compensated for with software which estimates the position of a missing dot. Also rotations of body parts must be solved for and are not absolute, performer must wear a suit with dots and balls ranging from 30-20 for one body, which may be uncomfortable. Information has to be post-processed or 'tracked' before viewing so performer cannot see his or her image and so cannot be as creative or identify potential problems. Furthermore, these systems have higher costs than magnetic ones, and tracking can take up to 2-1 minutes per captured second for straightforward data, and 30-15 minutes for the more complicated ones. 3. Electromagnetic (magnetic) Mocap: The performer wears an array of magnetic receivers, which track location with respect to a static magnetic transmitter. One of the first uses of this technique was for the military, to track head movements of pilots. Via magnetic mocap systems positions and rotations are absolute;

Fig1.9 Optical Mocap technique

orientation in space can be easily determined. Processing can be real-time, which allows immediate broadcast as well as the opportunity for performers to puppeteer themselves with instantaneous feedback. The system is relatively cheaper than optical and mechanical systems. However, magnetic distortion occurs as distance increases, and data can be noisy. The system is prone to interference from magnetic fields, cement floors usually contain metal, and so stages must be built. Moreover, performers wear cables connecting them to a computer, which limits their freedom.

Fig1.10 Magnetic Mocop suystem

Computer Vision- OpenCV

Computer vision refers to a broad class of algorithms that allow computers to make intelligent assertions about digital images and video. Recently, improvements in software development tools for student programmers and interactive media artists, in combination with the rapid growth of open source code sharing communities, predictable increases in PC processor speeds, and decreasing costs of digital video hardware, have made widespread artistic experimentation with computer vision techniques a reality. The result is a proliferation of new practitioners with a great potential of new application ideas, and the incorporation of computer vision techniques into new artworks, games, home automation systems, and other fields. The first interactive artwork to incorporate computer vision was also one of the first interactive artworks. Myron Krueger's legendary Videoplace, developed between 1969 and 1975, was motivated by his belief that the entire human body ought to have a role in our interactions with computers. In the Videoplace installation,

a participant stands in front of a backlit wall and faces a video projection screen. The participant's silhouette is then digitized, and its posture, shape and gestural movements analyzed. In response, Videoplace synthesizes graphics such as small critters, which climb up the participant's projected silhouette, or colored loops drawn between the participant's fingers. Krueger also allowed participants to paint lines with their fingers and entire shapes with their bodies; eventually, Videoplace offered over 50 different compositions and interactions. Computer vision- based systems work with algorithms of three elementary techniques to track people of interest. Tracking is done at a wide variety of spatial scales, from extremely intimate studies of facial expressions, to the gestures of the limbs, and to movements of the entire bodies. The first technique for distinguishing pixels is the frame differencing, which attempts to locate features by detecting

their movements. Each pixel in a video frame F1 is compared with its corresponding pixel in the subsequent frame F2. The difference in color and/or brightness between these two pixels is a measure of the amount of movement in that particular location. These differences can be summed across all of the pixels' locations, in order to provide a single measurement of the aggregate movement within the video frame. In some motion detection implementations, the video frame is spatially subdivided into a grid of cells, and the values derived from frame differencing are reported for each of the individual cells. For accuracy, the frame-differencing algorithm depends on relatively stable environmental lighting, and on having a stationary camera unless it is the motion of the camera that is being measured. Second is the background subtraction, which locates visitor pixels according to their difference from a known background scene. The technique operates by comparing each frame of video with a

stored image of the scene's background, captured at a point in time when the scene was known to be empty. For every pixel in the frame, the absolute difference is computed between its color and that of its corresponding pixel in the stored background image; areas that are very different from the background are likely to represent objects of interest. Background subtraction works well in heterogeneous environments, but it is very sensitive to changes in lighting conditions, and depends on objects of interest having sufficient contrast against the background scene. The last technique is brightness threshold, which uses differences in luminosity between foreground objects and their background environment. With the aid of controlled illumination (such as backlighting) and/or surface treatments (such as high contrast paints), it is possible to ensure that objects of interest are considerably darker than, or lighter than, their surroundings. In such cases objects of interest can be distinguished based on

their brightness alone. To do this, each video pixel's brightness is compared to a threshold value, and tagged as foreground or background accordingly. In this respect, many developer have been continuously enhancing and adding libraries to interactive platforms such as Max/MSP and open-source communities such as PureData. Two of the most influential contributors to the open-source computer vision platforms are Yves Degoyon and Lluis Gmez I Bigrda, by creating the openCV library for PureData. The pix_opencv library includes many approaches to motion capturing and motion tracking, also relying on the three elementary techniques mentioned before. Examples from objects in the library include pix_opencv_contours_convexity, which looks for the convexity curves of the biggest contour of an image, and defines each curve by three points: the starting point, the ending point and the depth point. pix_opencv_motempl, This object tracks movement of detected objects

Fig1.11 pix_opencv_contours_convexity

Fig1.12 pix_opencv_motempl

10

using the history of motion on a variable number of frames. It operates internally with binary images; thresholding the incoming frames and build a history of image changes with timestamps. Another one is the pix_opencv_haarscascade, this object loads a Haar's cascade decision tree in the form of an XML file, that is based on Haar's technique of dividing an image in squares and calculating some pixels sums on these squares, and the differences between them. Using this technique, it is possible to detect some peculiar form in an image, like faces, profiles, eyes, etc. Other examples form the library include color filtering and tracking, background subtraction, edge detector and line detectors.
Note: pictures in the openCV section are low resolution due to webcam capturing.

Fig1.13 pix_opencv_haarcascade

Fig1.13 pix_opencv_bgstats

Fig1.12 pix_opencv_colorfilt

Fig1.14 pix_opencv_lk

11

Review on Related Projects and Artists

David Rokeby BIPED, 1999

12

David Rockeby

David Rokeby is a Canadian artist who has been creating interactive sound and video installations since 1982. His work directly engages the human body or involves artificial perception systems and intends to explore time, perception, issues of digital surveillance and the relationships between humans and interactive machines. In 1982 Rokeby started developing Very Nervous System, a real time motion tracking system, which monitors the user's action via video camera, analyses the data in the computer and responds to the participants input. With the Very Nervous System approach, Rockeby has created a series of installations that create a dialog between the participants body and the invisible environment of the installation. As the user enters the active space of the installation, he/she starts to generate sounds or music through different gestures and movements of their limbs. Referring to Rokeby in his article Harmonics of Interaction, The central aspect of the work is neither the 'sound'

Fig 2.1 David Rokeby, A Very Nerous System

nor the 'dance'. It is the relationship that develops between the sounding installation and the dancing person that is the core of the work. Also as described by him in The Construction of Experience: Interface as Content, the early developments to the system was to try and reflect as many parameters of the action as possible. I worked out ways to map velocity, gestural quality, acceleration, dynamics, and direction onto as many parameters of sound synthesis as I could. What I found was that people simply got lost. Every movement they made affected several aspects of the sound simultaneously, in different ways. Ironically, the system was interactive on so many levels that the interaction became indigestible. Peoples most common response was to decide that the sounds from the system were not interactive at all, but were being played back on a cassette deck. Rokeby concluded that by reducing the level of interactivity or limiting it, gave empowerment to the whole work by making the users feel more in control

13

over the system. The problem seems to have been a linguistic one: people were unfamiliar with the language of interaction that I gave them. Through limiting the interactive variables, the user starts to be conditioned to and appreciate the given level of interaction. Rokebys work has been shown in exhibitions, shows, festivals and performances across Canada, the United States, Europe and Japan, including the Venice Biennale in 1986, Festival des Arts Electroniques (France) in 1988, the SIGGRAPH88 Art Show (USA), artware:kunst und Electronik (Germany) in 1989, the Kanagawa International Arts Festival (Japan) in 1990,and Arts Electronica (Austria) in 1991.

Fig 2.2 System operation, Very Nervous System

Fig 2.3 Video analysis, Very Nervous System

14

BIPED, 1999

BIPED is a forty-five minutes long hybrid performance, produced in collaboration between the renowned choreographer/dancer Merce Cunningham, the digital artist/writer Paul Kaiser and the digital artist/computer graphics/ human motion artist Shelly Eshkar. BIPED is one of the early approaches in performance to use mocap and projected computer generated imagery of abstracted human figures during the performance. Cunningham is considered to be one of the leading avant-garde artists for more than 50 years in pursuing a career in dance and choreography. He is known to apply new technologies to his work, in the 70s he combined film and dance, and later in the 90s combining computer graphics to his performances. Kaiser, originally a filmmaker and illustrator and later developing in the interactive and multimedia installations. Through out the 90s he has produced several collaborative woks in the field of hybrid media with
Fig 2.4 Motion abstraction, BIPED

renowned artists. Eshkar, relatively a young digital artist whose work mainly relies on computer graphics and interactivity. Eshkar is rather interested in motion capturing and projecting his technology by abstracting the human form into a generated virtual space. In the performance the stage decor is minimalist, with dark side and back curtains and a few vertical reflective materials placed against the back curtain. Between the front of the stage and the audience lies a transparent, reflective scrim. Animated real and abstracted dance characters projected onto the scrim create the illusion of the animations moving with and among the real dancers; they become part of the set. In fact, each element, choreography, music, decor and projections are created separately and united at the rehearsal for the first time as the known technique and tradition for Cunningham, which gave a feeling of randomness, although the final mix is quite deliberate and reproducible. The

15

The animated projections vary from simple dots or straight lines driven by clear human movement, to ghostly human forms appearing to dance with the dancers on stage. The animations derive from a complex process, beginning with computerized motion capture sessions by using Motion Analysis optical motion capture technology. Wearing a collection of strategically placed optical sensors, two of the Merce Cunningham Dance Company dancers, performed a series of short choreographed movements, sometimes alone, sometimes together. Two distinct methods generate the animations based on the bipeds movements. Earlier efforts used 2D animation by Paul Kaiser and Shelly Eshkar who rotoscoped (traced frame by frame) a series of highly gestural, nonsolid 2D hand drawings intended to capture the expression and emotion in the virtual choreography.

Fig 2.5 Projectiontions among dancers

Fig 2.7 Optical Mocap, Modern Uprising Studios, NY

Fig 2.6 A Merce Cunningham Dancer

Fig 2.8 Generated dancers

16

The Design of the Artistic Project

Preparations Software Testing the Software Conversations with Shaymaa Shoukry

17

Preparations

As I was starting to design the practical part of the project or the final installation to be exhibited in June, all what I knew at the time was that Im supposed to come up with something related to the topic of my interest. I had some keywords in mind in which Im concerned with: time, reality, space, virtual, human body, computer-vision and the relation of our worldly experience to that of the machines. It was not so interesting to me just sitting on a desk behind a blank paper and drawing a sketch of the installation at an early stage like this. Rather, I decided to just dive into the process with an open-mind and flexible decisions, to explore available techniques, styles and possibilities. I was interested in playing around with my tools, which basically were the computer, my programming skills, webcams, projectors, screens and my own body. Later through out this process of experimentation or simply playing, I was to produce a series of experimental videos, which approach my topic in different ways, each time focusing on one or more of the keywords. At the

end of this process, I was to design the final installation based on my conclusions from the videos. It then came to my mind extending the pool of the artistic input, also approaches, by collaborating with a performer or a dancer. Someone who has an understanding of the human structure, knows how to express topics through gestures and most importantly is aware of what I will be talking about, has an artistic insight and interested more or less in the same topics. Accordingly, I approached Shaymaa Shoukry, a performer / dancer / choreographer, also a friend of mine, to see if she would be interested in working along in those experimental videos. Fortunately, she was very excited in collaborating, as this to her was a very good chance to experience this new approach to performance. As well, a great opportunity to express, experience and interact with the machine and her-self in a totally new perspective and dimension. So the first step afterwards was to sit together and decide on mutual topics. At

the beginning of the brainstorming sessions, the end result seemed pretty vague; for me I only knew my end result, but had no idea how to get there still, and for her the process was so interesting but still a bit confusing; drawing upon the ceiling of technically whats possible and whats not. However, weve put lots of effort into not getting confused with the endless possibilities and where will this end. Rather, keeping the process flexible, open-ended, minimizing the tools and strongly believing that ideas will come on the way towards the end. Shoukry and I had lots of mutual interests, such as time, time lapse and spaces, on both dimensions, the real and the virtual. Also she was interested in interacting with projected imagery, in ways of manipulation and distortion of all types, in the transformation of forms, in what is a matter and what is beyond that state, in the relation between humans, nature and urban spaces, in electronic sounds that were not so dry and emotion-less. However, out of all the topics that each one of us suggested, we chose to be

18

be working on the bridges between real and virtual spaces that conveyed time, space and diverse motion interpretation.

Testing the Software

I then started building up some rough patches as a way of testing and trying out divers ways in expressing time delay, motion interpretation and color filtering for possible visual aesthetics. 1] Abstracting motion In this example, I was trying to abstractly represent motion on to the screen. This effect represents bodies in motion in a particle like form, the higher the acceleration of the movement the more the body turns into particles farther from each other, and the less motion the more the particles attract each other in the original form of the body. (Fig 3.1)

Software
The first thing after drawing the rough outline for the progress of the project, was improvising with the machine to create different visual styles for the videos. The software in which I was using through out the whole project is PureData, a real-time graphical programming interface designed originally for electronic music, then developing into the interactive media art for translating any kind of input to manipulate different graphical, video or audio events. PureData was designed by Miller Puckette in the 1990s, as an open-source software to be used by programmers, musicians and media artists, supporting the copy left philosophy. For more about PureData and copy-left visit www.puredata.info
Fig 3.1 Motion to paticle abstraction

19

2] Motion Blur This is a trial in dealing with time dimension on the machine. The effect done is just blurring motion by storing up a custom number of the past frames, in this case 8, and then blending the past and present frames all together. The result is interesting on many levels; there are many ways in customizing the effect, in means of stored frames to the intensity of mixing them, to the decay of the past stored frames. (Fig 3.3)

3] Color Filtering and Abstraction Another interesting approach was combining the motion-blur patch to another of color filtering. In this example, I over-exposed my hand and a white object and selected the colors of both my hand and the object, so they stand out in a black environment. Afterwards, I applied the patch of the motion-blur, and played with the presents in extreme variations through out the capturing time. (Fig 3.4)
Fig 3.3 Motion Blur Fig 3.4 Color filter and abstraction

20

Conversations with Shaymaa Shoukry

What is worthy of being mentioned or documented, are my long conversations with Shoukry through out the process. Myself coming from a media background, and herself dealing with body, materials and physical aesthetics, gave a good potential for interesting results to come out, as well as good chances for both of us to get lost and confused. After Ive made the rough patches or software tests, we watched the results together and sat down to discuss and come up with ideas. At first, she was so exited to test and try out these approaches (that wouldve been fine from my side) but as we moved on she started to lose the reasons for doing so. As a dancer, it was not enough for her to keep on improvising to the same interaction, which after a while she got used to and lost the motivation to continue. To her having a storyline or even choreographed improvisation would give the whole process a twist of emotion or expectations, which would make it much more interesting. As for myself, I had no problem with just creating experiments that had no dramaturgy, rather exploring

possibilities that could then turn into a choreographed piece. Moreover, I had no time, or couldnt think back then of all disciplines of the project, plus I have never choreographed before. However, we kept asking ourselves some questions, as a means to reach a compromise or a solution that works out for both of us. Here are excerpts from our recorded conversations: Shoukry: Improvisation is my school of dance, but every time I do it, I do it in a different way, related to emotional, conceptual and visual reasons. For example, if I write the choreography of a piece with the concept of random probability, I would divide the space in to invisible paths, across the duration of the performance, and my moves or dance will only be inspired from the music, which could change each time. So there would be a concept, tool, and a way of applying these two variables together through a performance or a human element. So lets say we have got six patches, each creating a different effect, and I do one

Fig 3.5 Myself and Shoukry

Fig 3.6 Snap shots from the conversation

21

phrase of movement in all of them, which for instance starts smoothly and ends violently. Like this we have taken an element, which is the performance, and put it into many different contexts so that each time it produces a different result. Accordingly, I would know how to integrate myself though out all the patches, and how to play with the effect, based on the rules or parameters of each one of them. But still there is a central idea that Im talking about. Experimenting with computer and performance only, seems to me as cleanproper movements and very well handled tools. But it does not send me anything more. As an audience, I want something that looks nice, as well to touch me in a way. So what I want to find out, or for us to try and find out, is the human being in the space and time between the real and the virtual. What is the topic now? What we are doing is using a new tool or a medium, which is

the computer, and we are so excited about the possibilities that are still fresh and interesting. So what happens is that we are making the topic this medium itself. As for myself, whenever I have something to say, I search for the appropriate tools that would help me convey my message. However, I am already so seduced to play with these tools, which are the effects done by the patches, but somehow Im looking for a justification for doing so. It doesnt have to be a complicated answer, but just simply why do I use it, and through using it what am I sending? What you are trying to do is combining two variables, or having a conversation between a language and a tool. Benany: But both variables, which are you and the computer, are as well languages and tools at the same time. Shoukry: Yes, thats true. We are both tools in a way, trying to make or speak the same language.

Fig 3.7 Snap shots from the conversation

22

So what I want to know, is how to speak with the machine towards one language, and how would this language look like in the end from the outside, apart from any emphasis of one tools over the other? Moreover, why interact at all? Benany: From a subjective view to the language created though combining the two tools (you and the computer), it's would observed that there are different levels of reality. This reality has its own rules and pool of expectations, in which you condition the viewer to what is possible and what is not. But before you go on with the storyline, you have to introduce the level of reality. So for example this could be a person waving his arms and leaving traces of his motion on a wall or in the air, or even manipulating sounds of different events. This would be one level of reality that people accept due to our inherited talent of fantasy. Nevertheless, what would be the one perfect answer to why do we interact with the machine at all? At the moment I dont

have that single answer, but I can conclude couple of possibilities, which all boil down to our human nature of being curious, selfish, narcissistic, lazy, creative and innovative. I guess if you follow each route of these characteristics you would find that humans have endlessly invented, adapted and got conditioned to so many things, just like our condition with computers nowadays, you would keep on asking yourself endlessly, why do that at all? Shoukry: Sure, just like dance as well, we chose to use some elements and neglect some other. For example why dont we speak in a choreographed piece? Why are all people so silent? I dont know. All what I want is to find a motive, and playing with the tool or the patch to be the extension that came out of that motive.

These were interesting parts of the conversation we had, nevertheless, we came to a conclusion of keeping these questions in mind such as: speculating about the relation between the human element and the computer element, as well findig the something that preserves the emotional part of the work in the interaction between the both elements, at last trying to combine these elements towards one language, which has a motive and aesthetical sense in the final visiual outcome. Nevertheless, we've decided to start the practical part, as so much is revealed in the process, as well so many questions were only making us more confused and distracted from the original topic and motivation which was exploring at the first place.

23

Experimental Videos

Exp. One : Motion Blur Exp. Two: Basic Motion Printing Exp. Three: Color Filtering and Printing Exp. Four: Fellow Shadow

24

What seemed highly interesting to me is the subjective point of view that combines parts from the reality and parts from the virtual in one frame. Which instantly creates some kind of an illusion and a twist to the perception of the level of reality that is presented on the screen (which is still virtual).

Exp. One: Motion Blur

In this first example, the main keywords or ideas targeted were time, self-perception and machine time. The construction of the location, (Fig 4.2) the performer (Shoukry) stands in front of a mirror, seeing herself in real time and studying her reflection. In front of that mirror there is a webcam mounted on a screen, which is showing the processed image from the webcam. The processing of the image is done by a patch that blends present and past frames, which creates a phenomenon in motion.
Fig 4.1 Motion blur exp.

We have recorded many trials or variations for the approach, and in each time Shoukry was free to act according to what she felt from the effect and the whole setup. Weve spent so much time that day in a single small room with a big mirror, and for Shoukry standing for more than five hours in front of her so called real reflection plus another one, which is acting not in a usual way on the screen was another interesting effect. She started to feel extraordinary exhausted from the lack of calibration between reality and playing with the more interesting image of herself on the screen. As well as her increased level of self-perception and narcissism, seeing only herself many times and for a long time. Her movements were affected by her self-observation on more than one dimension plus the limited space in which we were working in, also taking a long time in the setup and preparations that day made her feel a little bored and tired, which could be seen on her movements and expressions as well.

25

Conclusion: This first trial of approaching real and machine time was important on different levels. One thing is that I further tested for the aesthetics of the visual look of the effect, which seemed to be appealing. Another thing is the bridge or the confrontation of reflected reality and virtual spaces, which even seemed to be even more appealing. It is after this trial that Ive decided to stick more or less to the idea of framing reality along with virtual spaces, and let the viewer re-calibrate or bridge the two worlds together. As well, the feeling funneled to the user or performer after experiencing such different dimensions. I was to arrange the setup and software preparations prior engaging Shoukry, to keep her motivation and energy for the recording period.

Fig 4.2 Floor plan of the setup

Fig 4.3 Snap shot from the patch

26

Fig 4.4 Early trials with the patch

Fig 4.5 Further trials and arrangements

Fig 4.6 Different arrangements

27

Exp. Two: Basic Motion Printing

For this trial, the main keywords were space and motion in real and virtual dimensions. The idea was simply and roughly tracking motion and printing it abstractly on a vacuum environment on the screen. The construction of the location was the same as the previous example, but this time the patch was designed to detect moving objects or bodies and print down strokes in the route of the movement. Stroke size and color could be manipulated and movement history could be re-configured.

During the recording period, Shoukry was again standing in front of the mirror the whole time, but here it had a different impact on her than the previous example. Over here she had no problem calibrating her reflection in the mirror, as her actions had a direct impact on the screen. As well, there was no big problem with self-perception in the same way as seeing herself multiplied. Rather, there was something else, after talking with her to see how does it feel like, she said: When the image is doubled, like the screen and me in a reflection, I cannot place my real- self. At that moment I'm experimenting the tools to create something, but the way that I experiment is not creating anything. My actions only have influence on my extension, which is the screen, but in reality I am actually doing nothing. This is why I lose the focus from my real-self. To me this seemed rather an interesting conclusion, and not so much of a problem to try and solve.
Fig 4.7 Motion print exp.

28

Conclusion: For me what is interesting out of this approach is Shoukrys statement at the end, which seemed to be said in a rather negative way back then. But however, I think it was an interesting starting point that has lots of potential for research, and interesting work to be produced under its topic. As for the technical part, I noticed that the tracking system was not working so well. Sometimes it tracks movement and sometimes not, also the scale or the calculation of the motion in real space was not the same on the virtual one, which was a bit problematic. Somehow this gap made the user loose the connection or the bridge between himself and his impact on the virtual environment. Referring to Shoukry, the imprecise tracking system made her movements very limited, as her body was bound to the technical ceiling of the software. She was too conscious about her decisions in moving at a certain speed and direction, so that the machine would track her simple movements.

However, we both were interested in the idea of drawing or printing on virtual environments, but in a more free, diverse and visually stronger way than this trial. For our next meeting I was supposed to enhance the software, and fill in the leaks that were in the tracking system.

Fig 4.8 Snap shot from the patch

29

Fig 4.10 Variations from motion print

Fig 4.11 Variations from motion print

Fig 4.12 Variation from motion print

30

Exp. Three: Color Filtering and Printing

As for this approach the main keywords were motion tracking, printing motion and color filtering along with time. I chose to enhance the motion printing system through elective color filters, for example the reds only or the blues or both together. Also combing this to some of the objects from the patch of the first trial, and adjusting the parameters in certain relativity to each other, gave the possibility to draw or print the history of the tracked color. This approach gave a more vivid look to the motion print on the virtual environment, since the strokes were easily manipulative by simply dealing with it as if it was a calligraphic pen. Moreover, the calibration of the motion in real space was the same in the virtual one. The technique was flexible and easy to handle and variable possibilities could be done each time the parameters are altered slightly. In most of the trials of this type, the color filtered was the color of the skin, so that the projected visuals would be the revealing parts of the performer's body, tracking and abstracting motion. (Fig 4.13)

The set up of the location in this video was in a larger room, which gave the possibility of visual projections and more free movements. Shoukry stands in front of a wall in which the visuals are projected upon, in front of her is the webcam capturing and sending to the computer, and from the computer the processed visuals are sent to the projector which stands behind the whole setup. Meanwhile, the performer could see the processed image on a monitor through out the whole time. For me it was of important for Shoukry to see the processed image while performing, which has her abstracted self, or manipulation on the virtual space. Referring to the previous trials, one seems to disconnect or reconnect in another way to his real-self while exploring new levels of reality, which could seem more exiting at the beginning.

Fig 4.13 Skin color filter and print

31

Fig 4.15 Floor plan of the setup

Fig 4.14 Snaps from the working progress

Fig 4.16 Snap shot from the patch

32

In the first variation of this approach, weve decided to shoot a trial in which Shoukry was just exploring the system, and getting used to it more or less, in order to know its aesthetics and leaks. The postproduction shows three perspectives to the same event. Top right is the subjective point of view, showing the capturing camera, the performer and the projections at the background. Bottom right, is the objective point of view, streaming the video input from the appearing camera in the top right frame. The whole left section of the total frame is showing the processed image captured from the objective camera. The combination of the three perspectives together, somehow draws a bridge between what is virtually subjective, objective and virtual, since at the end the edited video is being streamed on another monitor. Choosing the capturing camera had an interesting effect on the final outcome. When choosing a low-resolution webcam, the final visuals seemed to be rough, distorted, pixilated and more abstract than

Fig 4.17 Variation one, subjective/objective/virtual points of view

Fig 4.18 Variation two, wall projections

33

that of the final out come of a DV cam for example. The processed image out of a DV cam seems to be much smoother and relaxed. However, both results are aesthetically appreciated in their own way. Conclusion: From these trials I found a nice combination between color filtering and motion blur or motion history. Resulting in an interesting material of visuals, related to motion mapping and printing with certain colors or body parts. Also the software seemed to be working fine and in a satisfying way more or less. Through out the videos the performer seemed to be having some kind of conversation with her abstractions that are projected largely on the wall behind her, which helped in disconnecting more from the real-self. At this stage, I got more concerned with creating something that disconnects oneself more, and rather transcends us in to the projected image, and the interaction, and to try and create a space-less moment.

Fig 4.19 Webcam video input

Fig 4.20 DVcam video input

34

Exp. Three: Fellow Shadow

At this stage Shoukry and I got even more interested in the digital double, along with the self-less, space-less phenomenon that occurs as you bridge your influence to the virtual events. We were as well fascinated by the relation to our shadows, which is considered a strange one with the same respect to the virtual reflection. For this example the keywords were space, space-less, digital double, fellow shadow and relation to virtual self. Drawing upon art as an experience, and interactive installations relying heavily on that concept, I was quite excited about designing a simple installation that could make you feel space-less and self-less for even a second. In this experimental I was to try and test along with Shoukry if interacting with real / virtual shadows that are yourself, but still not, could eventually send this feeling over during the performance.

The set up of the location in this example was very influential to the visual result, since the effect relied mainly on capturing the shadow, processing and projecting it, all at the same time. The elements used were a rear projection frame, webcam, projector, and the computer. So basically the performer stands behind the frame, which is backlit by the light coming from the projector. A webcam is placed on the other side of the frame, capturing the shadow of the performer, the webcam sends the captured image to the computer to process and send to the projector, which then projects the image inverted and delayed, to be re-captured with the webcam, re-processed and getting into the loop over and over. The patch used for this approach is rather a simple one; using the same motion blur with a frame history or 8 past mixing with present.

Fig 4.21 Fellow Shadow exp.

35

However, interesting things happen within the loop of capturing, processing and projecting, also the sequence in vise-versa. For example certain shades of a color appear as a result of the interference of the webcam and the projector being placed opposite to each other. Mixing the RGB channels of the projector gave a halo like shape spreading from the edges of the frame, tightening up when the frame is empty and loosening up to create a rather light space whenever a person walked into the middle of it. At that time we wanted to try and design a simple dramaturgy for our concept along with the tool, which was the installation. We were talking now more about the relation to the digital double, or the fellow shadow, with no idea about whether the space-less or self-less effect would happen or not. Shoukry suggested to write a storyline, or basically steps in which the interactions would happen as performance. What and how would be the relation to yourself in another space? To a being that seems to be you but isnt; its

Fig 4.22 Floor plan of the setup

Fig 4.23 Snap shot from the patch

36

inverted and following your movements but milliseconds delayed and seems as if its looking right at you. The video shot was from a subjective point of view as usual, but here this camera had an influential role in the dramaturgy of the video as well. The video was arranged to show the rear-projection frame in the middle; with a central focus weight, and around you could see a small window at the top bringing in some light, as well some dark details of the room and rough features of the performer before entering the frame. This composition was to stress on the bridge between real and virtual, as the frame was representing the virtual world created by projected elements from reality in an unusual way, but still in a totally normal room and details that we are conditioned to. Starting by the performer standing outside of the frame or the virtual space in the composition, then entering slowly and exiting again as if being terrified. Again she enters, this time taking a little longer to have a look at what seems to be

herself, and then hesitatingly walks out. Re-entering again with a more curious attitude towards that fellow shadow; she starts to explore that world and to study her reflection or herself on new dimensions. Both shadows, the real and the virtual start getting in harmony as one-self gets conditioned to the level of reality set by the installation; dancing and creating patterns together and being more relaxed. At a certain point in the timeline, the so-called real shadow starts to develop some anxieties related to self-perception, and starts blaming or confronting its fellow in a way. But somehow the other fellow shadow seems to be developing a character on its own, and throwing back the blame as well, forming a violent conversation between the both sides. Each side is looking for something to embrace, as many failing attempts in doing so results in aggression and frustration. Afterwards, both get tiered and worn out from their confused conversation, and reside in the middle of the frame, to have

a deep breath. As for the ending of the storyline, Shoukry was free to end it the way she likes, either by staying in the middle of the frame or deciding to walk out again, that only depended on what she felt at that moment. In the recorded video, she decides to walk out.

37

Fig 4.24 Snap shots from the timeline, Fellow Shadow

38

After shooting the video also many prior trials so that Shoukry gets acquainted with how the installation worked, it was so important to get her comments and feelings out of performing with the installation. As she says: Somehow while standing behind the frame and watching my shadows interact, gave me the feeling that my direct shadow (the darker) seemed to be the real element in the performance, and the opposite shadow was the virtual, but my real-self didnt exist, as if I was not real! My extension in the machine became my real-self and the reflection of the machine became the virtual, and I became a factor that is not really existing! Benany: But while performing and disconnecting from the real-space and getting into this sort of fantasy, which is projected on the frame, somehow you feel that the other inverted being is also a part of you that you want to embrace and be tender to, by trying to touch and reach for all the time. Also in other times you

feel totally the opposite. Shoukry: Yes, and I think this is what simply happens to us when put in certain situations with ourselves, we get confused, freak out, blame and sometimes hate ourselves, but then we relax and try to understand and get back in harmony to obtain a peace of mind. But over here this fellow shadow made you visualize or think that it is that part of you in which you confront and have dialogs with.

Fig 4.25 Shots from the back-stage

39

Conclusion: I conceder this experimental to be successful in fulfilling the intentions of detaching the performer or the user for couple of seconds behind that fantasy like world of the frame. Now, I got more interested with combining the keywords real space, virtual space and space-less, also fellow shadows. Also what I think helped in the disconnection of the real-self was the sound. In case of this experimental, the music choice was something to be ambient like, not to be the element of emphasis in the performance, smooth and mystical in a way. The track playing during the video is called Cold by Sigur Ros, which help quite well with the overall mood and dramaturgy. However, for the next step it would be interesting even combing the keywords sound, real space, virtual space, space-less and time.

Fig 4.26 Shots from the back-stage

40

Methodology of the Final Installation

Ideas and Sketches The Final Concept Documentation of the Design Process Building and Testing the Prototype Sound and Software Design Building and Testing the Boards Testing the Installation
41

Ideas and Sketches

Reaching now the point where a final idea has to be developed and the installation has to be done, was a less confusing task relatively to where I was at the starting point of the project. For me drawing on the final idea was a matter of cross matching the keywords from the experiments and listing down in priority the conclusions through out that period. So what I had in mind at that moment was designing a pubic installation that could combine my points of interest and findings. Including the relation between real and virtual space, real and virtual time, the conversation with the digital double, the dimensions of the space-less and mystical sound generation. Something that could detach the user from his real-self and transcend him into a fantasy and simulation of being virtual with actions that has an interesting impact on various events. An early idea for the final installation was the Laseroom. The main idea in this installation was to create some kind of platform or stage that would act like a

Fig 5.1 Sketches of the Laseroom

virtual instrument for producing sounds with different notes. On one side of the arena was a vertical series of photocells or light sensors lining up on the wall, and opposite to it another vertical series of simple red laser beams. So basically as any body interferes in the path of the laser to the directed sensor, a certain sound is produced. Supposedly, four or three of those columns were to be places across the whole arena, creating a wave of emitted computer generated sounds. I was not sure whither to place a monitor so that the user could see himself performing or not, and if yes where will it be? Also a fog machine was planed to be placed at one corner, producing a small amount of fog every couple of minutes in order for the laser beams to be visible to the user. For me this idea was fine at the beginning, but then I had some problems with it. First of all it was not satisfactory to most of the planed keywords I was working on. For example there was neither this strong bridge between real

42

and virtual space nor time. It might give a sense of space-less environment, though not on high expectations. There wasnt a relation to the virtual self, besides you could see the laser beams with your naked eye, which signified the real-self as the main actor. Sure the beams could have been invisible just by simply removing the fog machine, but then why sensors, lasers and physical computing at all? Since this could be done through computer vision as well. However, many elements were unsatisfactory and problematic for me in this idea so I decided to keep on brainstorming. Many ideas crossed my mind of working with shadows, silhouettes, sounds, and something that didnt look the same on the monitor as it looked in reality. Searching through physical components that have characteristics of acting in such a way, I thought of the infrared light emitting diodes. These infrared LEDs are simply small light bulbs that when connected to the proper electric circuit their light could only be seen on screen via

a camera with infrared filter. For me this was very inspiring and with potential for promising results. I started going further into this direction of combining physical properties that were on the same line as the keywords that Im currently dealing with.

Fig 5.2 Infrared LED

Fig 5.3 Infrared night vision camera

43

The Final Concept

After recombining the tools and elements that I had in hand, I came out with the final concept to be working upon for the next 5 weeks at least. InfraShadows is simply a wall with pixel like infrared LEDs placed evenly through out the wall. Drawing upon the invisibility of the light of these LEDs to the naked eye, the person who walks into the relatively dimed space, and across this wall, will see himself on a monitor as a silhouette in front of small lighten LEDs. As the user moves different parts of his body, he changes tones and notes in the sounds produced. The setup of the space of the installation would be as following: one side of a room will have the board (200x170 cm) in which the LEDs will be inserted, opposite is a rear projection frame with an infrared camera mounted on top. The frame will picture the image captured from the camera showing only the lighten LEDs. The user will enter in the space between the wall and the frame to stand in front of his silhouette that is pictured in an unusual environment. Ambient and mystical

sounds start to be produced as the user crosses certain sections of the virtually lighten LEDs. Now having a closer look at the concept of the installation and the setup. Referring to the conclusions and interesting points from the previous stage, I wanted to represent a relation or bridge from our world into the different levels of realities that are possible on the screen. Over here the infrared characteristic of the LED gives a basic and simple relation between the lag between the two levels of reality: the room and room on the screen with the LEDs activated. Just by this simple confrontation, or fact, one starts to get a twist of curiosity, and gets on a ride of wanting to know what is this? How and why are they only lighting up on the screen? For me this was a great trigger that took me to the next step of using this element. Another keyword of mine was the relation to the digital double, or our fellow shadow, a thing or being that somehow looks and acts the same, but isnt.

Normally, if any object stands in front of a light background, only the silhouette appears. So what I though of as a possible combination between the keywords: real / virtual space and digital double, was the combination of the infrared LED characteristic of virtual light only, and extracted silhouettes of people standing in front of that virtual light. Then projecting the image in real size opposite to the user with the same interesting result of the motion blur or delay from the experimental videos, which gives a sense of ghost like motion, infra-presence and somehow mystical experience. As for further enhancing the experience of interacting with another dimensional space, some sounds were to be added or triggered as the user activates certain parts of the virtual grid of the whole space. By this approach, it is though to be a support in creating a space-less environment by empowering the bridge of virtually manipulating diverse events.

44

Fig 5.4 Floor plan of the setup

45

Documentation of the Design Process

Following is a cited documentation of the building up of the installation. However, please refer to the accompanied DVD for more details of the whole process from start to end.

Building and Testing the Prototype

In order to start testing for the possible technical applications for the final installation, I first had to test the software and the electronics part on a small prototype.For this I used a proto-board, which measured 10x10 cm and soldered infrared LEDs in an aspect ratio of 4x4. After connecting the proto-board to a voltage of 9v, I tested it to see how it looked on camera when the infrared filter is turned on, and whither it appears with the effect that I had in mind or not. The look on the camera seemed to be pretty well, and doing its role of extracting the silhouettes of the forms direct to the LEDs.
Fig 5.5 Snap shots from building the prototype Fig 5.6 Testing the prototype on camera

46

Sound and Software Design

My chosen approach towards motion tracking was the motion grid, which is simply dividing the space into an invisible gird of X and Y axis. The grid could be adjusted according to the aspect ration of the desired tracked space, in the case of the prototype 4x4, so that each LED on the proto-board sits in a square on its own. The software then sends a variable number of the blocks that have detected motion within its territory. I chose that approach because it was the best suiting my situation of assigning different events to blocks on the LED wall, regardless of the number or size of people within the active zone of the installation. Early trials from the sound manipulation include basic phasors, sign-waves and white noise, in which motion changed the frequency or the parameters of each variable. Next was a trial to import some dummy samples and assign their activation to certain spatial blocks by using the playlist object. But with this patch, lots of bugs came out as the samples

overlapped in a disturbing way; cutting through the same block several times produced an error in the play count. However, I was to further try other ways of playing and mixing samples in a more calibrated and smooth way. As for the samples to be played in the final installation, I preferred to have some designed samples and not to use downloaded sounds that would not be suiting the case so much. Also looking for something that helps in transcending into the mood of the installation, something thats spacy, mystical, ambient, electro, groovy, a little melancholic and could even be a little daunting.

Fig 5.7 Grid motion analysis

Fig 5.8 A patch for early sound trials

47

Accordingly I approached a friend of mine Ahmad El Azma a young visual artist also a sound designer to ask for his help in the sound part. We sat together couple of times trying different things that would fit all together. To help deciding about the assigned sounds through out the board, I made a general sketch for the map of sound manipulation through out the board. Basically the sounds were divided into variations of bass (variable volume), rhythms (variable tones) and ambient (variable tones). The idea was to minimize the level of interaction and to assign clear blocks for the tones produced so that the user would easily get the impact and be conditioned to his manipulation on the sounds. My sessions with El Azma resulted in producing various samples for each section, but still there was the feeling that something was missing. So I kept on trying different patchs and samples, also applying real time filters to the variables of the manipulation, in vain that the sound produced is harmonious

Fig 5.9 Sound sessions with Ahmad El Azma

Fig 5.10 General map for sound manipulation

48

harmonious and again enhancing the whole installation. However, coming across some tracks from Karim Lotfi also a young graphic and sound designer also a musician, added something to the sounds produced. After notifying Lotfi about using samples from his tracks Fum Catura and Caffeine Bullet, I started dividing and organizing the samples in a progressive pattern along with some of the samples produced from the sessions with El Azma as well. Getting back to the patch for the sound manipulation after the errors in the approach of the first trial. Now I used table 0$ as a playback for the samples, as it was much easier to handle and real time filters could be applied and manipulated, in this case a reverb filter. The sounds were now divided into two main families, and each family is composed of 6 different samples of the same aesthetics. Each family was assigned to one virtual half of the board, for example on the Y axis starting from block number 0 all the way to block

number 12 (a total of 25) each 3 blocks will trigger one sample. On the other half of the board, which is from the block 13 till 25 also each 3 blocks approximately will trigger one sample from the other family. As for the X axis, its blocks will be mainly manipulating the volume and pitch of the constant ambient noise.

Fig 5.12 Testing the motion grid patch along with the sound

Fig 5.11 Early patch designs

49

Fig 5.14 A sample from the first family, Zone1

Fig 5.13 Final sound assingments, Zone1 / Zone2

Fig 5.15 A sample from the second family, Zone2

50

Fig 5.16 Final Software design / Patch

51

Building and Testing the Boards

The dimensions of the installation measured a total of 200 cm height and 170 cm width, divided upon 4 separate foam boards. I divided the total scale of the boards to fit to the aspect ratio of 20x14 LEDs, meaning one LED each 10 cm, with a total of 280 LEDs. Starting off by spraying the boards in black paint, then pinning the LEDs to the correct aspect ratio, afterwards wiring all the LEDs cathodes to the anodes and bottom cathodes to resistors of 47 ohms to be connected to the ground pin, and the top anodes to be connected to the voltage or positive pin of a power supply of 48v and 1 ampere. There isnt much to say about the process except the technical details, however, this section looks small, but out of all the rest this took the longest time and the most effort to be done.

Fig 5.17 First steps in building the installation

52

Fig 5.18 Steps from the building up

53

As for trying out the software and sound design in a primitive way, I tested the installation on myself first then on some friends. My first tests made me see that there is still a lot of work to be done on enhancing further the software and sound manipulation. But in general, the ghost like motion effect seemed to be doing the job quite well, as I got that from my trial and others comments and reactions which seemed to be fascinated over all by the virtual light of the infrared LEDs and their silhouettes which danced in a ghosty effect against the boards. As well the sounds were helping a lot in creating a rugged character for the installation. One of the comments that I got was to minimize the divers sounds so to feel the actions impact more. But no one gave comments regarding the space-less feeling, however, my interpretation would be because of the small monitor in which the user could see himself in, which minimized the connection. As well the ambience or the surrounding, which was somehow distracting from getting fully into the mood (putting into consideration

that the setup was in my room with limited space for divers movements) and the sound manipulation that needed to be further enhanced. However, I was to work on these leaks and retest again on a larger number of participants.

Fig 5.19 Infrared boards

Fig 5.20 Snap shots from testing the boards

54

Testing the Instalation

The next step was to get further comments on how the installation worked from diverse people and in a more suitable environment, to make the final enhancements for the exhibition. For this I used the media lab in my university, which is very spacious and could be dimed more or less. The four boards were hung on the wall, and right in front of them was a rear-projection screen, about five meters away from the boards. As I finished the setup, I noticed some problems that could affect the whole point of the installation. First was that some LED columns were not working and others got burnt, which distorted the shape of the silhouettes standing in front of them. Second, the projection on the screen was not big enough to reflect the real dimensions of the bodies in front, which I though was problematic. One more thing was that the room couldnt get as dark as I though, so much uncontrolled light was coming in making the projection faded and the user feeling quite observed. Nevertheless I was to test how people

react to the current circumstances, and if they have further comments I couldnt notice. As some people started to walk into the room for trying out the draft setup for the installation, they first look curious and skeptical from the first sight of the setup, and didnt know what to do exactly until told to walk into the active zone. As they do, sounds start to be generated and react to their movements, which draws their attention quite fast. Nevertheless, most act in a rather restricted way due to the conscious feeling of being observed mainly by myself in order to see their reactions. From the common comments of the users was the size of the projection, which was too small and not so clear, as well the missing LEDs in the board, which somehow abstracted the whole form of the body making it harder to connect to the image on the screen. Other comments related to the type of sounds generated included that they were creepy, and too much noisy, yet others

Fig 5.21 Shots from testing the installation

55

said that they were fine and had nice details in them. However, most people gave positive comments about the interaction, which seemed to be working quite well and didnt have lousy bugs in it. Someone said that her manipulation over the sounds made her feel as if she was a magician. It was interesting for most to see the contrast between the inactive LEDs on the boards behind them, and the virtually lit ones on the screen. As I have noticed that the average time of the users trying the installation was five minutes of continuously performing different movements, before feeling embarrassed, include walking across to both ends, diverse arm movements, finger movements, bending down and running across for those who were more relaxed. But still there seemed to be many missing elements, which made them not get across the whole point and rather deal with it as something just for fun. After discussing with Prof. Daniel my supervisor, I started seeing that the room had the greatest impact on the mood or

feeling being delivered to the user, since the boards where hung on a long wall leaving the whole installation looking very small, cold and as a minor element in the space. As well the big distance between the projection screen and the boards (making it look like a corridor to the other side of the room) along with the different dimensions of the screen which is more of a landscape to that of the boards which is more of a portrait, somehow gave less possibilities to form a strong connection and distorted the dialog between the two parties or elements, and released the tension and contrast that is supposed to rise between both sides, which is the whole point of installation. For the final amendments in the installation and setup for the exhibition, I was to fix the LED patterns on the screen, to reflect the body more. As well to recompose the setup in a manner that supports the concept of the installation. Moreover to have a high visual characteristic resembling the dialog between the physical and virtual boards,

Fig 5.22 Shots from testing the installation

56

by making them both have the same size, to give a chance for comparisons and tension to rise. Accordingly, test if the installation would succeed in making the user have a conversation with his double and sense a moment of space-lessness or not, through observing the time spent in the installation and the level of consciousness directed to the physical surrounding.

57

Conclusion

In this research project I was to create a process of experimenting with tools related to computer vision and interactivity along with the human body, trying to create environments that combine both elements in a harmonious and visually aesthetic way. At the end of this process yet not more significant than the rest, I was to design an interactive installation to be exhibited at the Townhouse- Cairo in June 2010. Starting of with broad keywords: space and time, real and virtual, later narrowing down to bridging real and virtual, space-less and the digital double. I proceeded in the experimentation process along with the dancer/ performer Shaymaa Shoukry who was collaborating with me in the experimental videos, which speculate about the relation and attitude we adopt towards the machine according to its levels of reality, in terms of visual and audio manipulation or self representations. After the experimental stage, a lot of interesting questions and findings came out regarding self-perception when reality

and virtual collide together with shifted time calibration. As well when actions in real have no effect but in the virtual are abstracted and enhanced in an interesting way; one tends to detach from his real-self and focus more on his extension on the screen. Moreover, the profound relation that is generated towards our digital double, a being that could look and act the same, but is observed to develop another character on the screen. Post testing the draft installation prior the exhibition, I drew some conclusion that had an impact on making the work fully effective and sending my point without leaks. For the environment to fulfill effects such as self-less, space-less moments and generated conversations with a virtual double, its important to put into consideration elements such as: the setup of the location, the size of the space, the size of the screen showing the virtual environment, the overall ambience, sounds enhancing connectivity, software calibration and the company of the direct user (alone or accompanied), which could

make him conscious about others presence. Still further tests and amendments are to be done till the opening of the exhibition, later documenting and testing for any upgrades.

58

References

> Erving Goffman 1956 The Presentaion of Self in Every Day Life > Marshal McLuhan Understanding Media the Extension of Man > Steve Dixon Digital Performance > Victor B. Zordan and Nicholas C. Van Der Horst Mapping Optical Motion Capturing Data to Skeletal Motion Using a Physical Model University of California > Maureen Furniss Motion Capture MIT Communication Forum > Levin, G. Computer Vision for Artists and Designers: Pedagogic Tools and Techniques for Novice Programmers Journal of Artificial Intelligence and Society > Yves Degoyon, Lluis Gmez I Bigrda Enhancing Pure Data Interactivity With Computer Vision ( Open CV )

> David Rokeby The Construction of Experience : Interface as Content > David Rokeby The Harmonics of Interaction > www.puredata.info

59