Documente Academic
Documente Profesional
Documente Cultură
Acknowledgements:
I would like to thank Prof. Franois Penz for his expertise, guidance and humour;
Amir Soltani for his enthusiasm and support;
And all the friends who volunteered to be experimented on:
Adam, Sohanna, Matt, Max, Lucy, Song, Jazz, Lauren, Katie, Mica, Iain, Immy, Fiona,
Daniel, Elly, Miranda, Charlie, Sam, Freddy, Emma, Phoebe, Sophia, Alice, Kitty, Livvy
and Francois.
9353 words
Abstract.
Key words: Embodied music cognition, sound, space, interaction, association
This dissertation hypothesises that our perceptions of music and space are
cognitively connected through embodied interaction with the world. These
perceptual connections will be initially revealed by looking at our structures of
language and speech, and then examining physical and audible precedents in the
work of Iannis Xenakis, Oskar Fischinger and Mark Applebaum. The hypothesis will
be further developed by exploring the shared neurological processes in listening
to music and the navigation of spatial environments. The theoretical framework
developed from the research is tested with a series of spatio-musical interactions
tested by 24 volunteers. The results are discussed in relation to their application to
musical composition, architecture and urbanism, speculating the potential for a new
paradigm in Music Aided Design.
Contents.
Introduction:
Precedents:
Music Signification:
Animated Music:
13
Sound Gestures:
15
18
Experimental Research:
21
22
Sonic Compass:
23
Sounds in Space:
24
Spaces of Sound:
25
26
Haptic Music:
27
Experiment Data:
29
Discussion:
30
List of Illustrations:
33
Bibliography:
34
Appendix:
36
Introduction.
To most of us music suggests definite mental images of form and colour. The
picture you are about to see is a novel scientific experiment - its object is to convey
these mental images in visual form. - Oskar Fischinger 1938
It is now becoming clear through scanning technologies that the various senses
also share higher order cerebral networks, or perceptual supramodalities that
engage a crossover of sensory inputs from one sense to another
- Harry Francis Mallgrave 2010
Over the past hundred years, increasingly refined theories of perception have
emerged. The top down philosophical theories are beginning to coalesce with the
bottom up scientific theories giving us greater insight into the study of experience1.
The relatively recent theory of Embodied Cognition has shed light on how the mind,
the body and the world form a conscious system of dynamic symbiosis. This recent
view of perception implies that experiences originally considered to reside solely
in the human mind such as music are in fact intrinsically connected to our bodies
and environment. Marc Leman describes this concept of embodied cognition:
In contrast to dualism, the concept of mind is seen as an emerging effect of the
brain perceiving its own actions in relation to a physical environment.2 From that
perspective the subjective world of mental representations is not an autonomous
category but a result of an embodied interaction with the physical environment.3 He
also suggests how our perception of music is tied to other embodied experiences:
The multimodal aspect of musical interaction draws on the idea that the sensory
systems auditory, visual, haptic as well as movement perception form a fully
integrated part of the way the human subject is involved with music. 3 This
dissertation will not focus on specific emotional processes involved in making and
listening to music. The primary aim of this discussion is to reveal and test how our
embodied interaction with the world develops perceptions of music and sound that
are fundamentally connected to our perceptions of space.
The history of phenomenology through to embodied cognition is described in further detail in
Dourish (2004) p.102-126
2
Ernelling and Johnson (2005)
3
Leman (2008) p.13
1
6
To develop this hypothesis, I will examine the works of Iannis Xenakis, Oskar
Fischinger and Mark Applebaum. I will observe how these artists have been able to
transpose between sound, image and gesture by utilising and adapting the intrinsic
spatial inferences that music creates. These works will be examined based on the
principles of embodied cognition to find the theoretical basis for the vocabulary and
intersubjectivity of each medium. The discussion will also look briefly towards shared
neurological processes in music and space perception, particularly the narrative
and memory functions of the hippocampus necessary for both spatial navigation
and musical experience.
The research is consolidated with a series of spatio-musical interactions in the form
of volunteer tested experiments. The experiments use the theoretical framework set
out in the first chapters to actively engage the volunteers spatio-musical ability and
develop new kinds of intersensory interactions. I will then discuss the potential value
and opportunities given by this frame of mind towards architectural discourse.
For the purposes of this text I will use a broad definition of music as: any sequence
of sounds that are arranged within a larger temporal structure, and the term sound
to describe any audible form.
Music Signification.
Listening to music is a vastly complex mental phenomenon that activates many
different parts of the brain. However, we can start to reveal the spatial aspects of
music perception by examining the ways in which we signify musical concepts.
The genealogy of music perception has consistently been related to the evolution
of language.4 Verbal communication, as well as refining the sophisticated pitch,
rhythm and pattern recognition necessary for music appreciation, uses higher level
representations to signify different experiences.
Linguistic descriptions of music (such as Hattens texts about Beethovens works
1994) rely heavily on metaphors to create intersubjective representations of melodies,
dynamics and timbres. These metaphors (such as Escalating, Swirling, Collapsing
and Sweeping) are not simply means to describe music but actively display how
we perceive music through the language of other experiences. George Lakoff and
Mark Johnson illustrate how our entire language is constructed from cross modal
inferencesand that these associations form the essential rudiments in which we can
conceptualise and think about the world.5,6 They describe the metaphor of affection
is warmth as emergent from a childs concurrent experiences of affection and warmth
in the embrace of a parent.6,7 This association between conceptual and sensory
inferences forms what Lakoff and Johnson call an Embodied concept.8 Our ability
to think is facilitated by both the brain and the nervous system7,8,9. The prevalence of
physical and spatial metaphors use to describe musical attributes and features, such
as dancing melodies, sweeping and swirling phrases, rising and falling scales etc.
allow us to see that our conceptions of music rely heavily on spatial understanding
and embodied concepts.
Brown (1999)
Mallgrave (2010) p.175-180.
6
Lakoff and Johnson (1980) p.255-256
7
The neurological basis for metaphors in language is described by Lakoff and Johnson (1999) using
Donald O. Hebbs theory of concurrent sensory information and synaptic growth (Hebb 1949), This is
explained fully in Mallgrave (2010) p.175-180.
8
Lakoff and Johnson (1999) p.20
9
Tversky (2008)
4
5
8
The words high and low used to denote audible pitch as well as vertical
position is a prime example of an audio-spatial metaphor that we can suggest a
possible etymology from an embodied perspective. Perhaps the most direct way
we interact with sound from the moment we are born is through the use of our
vocal chords. In producing or replicating different pitched sounds we must perform
certain biomechanical operations that are coordinated with our auditory sense.
Our vocal production of sound affects our spatial perception of pitch through the
perceived location of vocal noises relative to the ears. The position of the larynx
in the throat moves vertically to allow a higher or lower vocal range, this physical
change in position affects the perceived location of the produced sound. Singers
use the terms Chest Voice, Middle Voice and Head Voice to represent these
different positions as this is where the sound is felt to be coming from, this clearly
reinforces the perception that lower pitches are physically positioned below higher
pitches in space. This embodied explanation displays that the signification of
the words high and low to audible pitch is not an arbitrary selection of words
but is heavily reinforced by the spatial perception of our voices, catalysing this
association between pitch and height over time. This is one example of how an
embodied action produces cross modal inferences between seemingly disparate
sensory perceptions. This multimodal perception is consolidated and symbolised
as polysemous homonyms in language. This vertical mapping of pitch in space was
initially observed by C.C Pratt in 1930 after observing that the specific succession
of tones in a musical phrase can generate a sensation of vertical movement.10
These ideas have since been experimentally demonstrated by Lidji et al. 2007 and
Rusconi et al. 2006 and show the potential that audible properties can be mapped
successfully into spatial representations.11,12 I have highlighted just one example of
how spatial terms are used to describe audible characteristics in speech; however
the discipline of music has developed other means of signification to represent and
translate musical attributes in visual and spatial terms.
music, the contemporary Einsteinian view had shown that time was relative to mass
and energy, in that any change in the contents of time would change time itself.
Xenakis aimed to emulate/explore this abstract idea in his music through the act of
capturing it in a static space.14 This negation of real time allowed the comprehension
of the whole in the instance of seeing the score. Xenakis used the example of
gunshots in a battlefield to describe the nonlinearity of these particular musical
Linguistic descriptions and conventional notations are critiqued by Bengtsson, I & Eggebrecht, H.
(1973), and Leman, M. (2008)
14
Hofmann, B. (2005)
13
10
experiences; the exact order of each bullet fired is irrelevant as the resulting sound
as a whole will undoubtedly be that of gunfire. This concept of the musical form
being something other than a sum of its parts is particularly evident in Xenakis
Mycenes Alpha (1978).The graphical score of Mycenes Alpha is comprised of many
(often straight) lines which collectively define larger, curved undulating masses (Fig.
2). The image that results is a collection of undulating conglomerate forms. The
graphical score was translated into audio by Xenakis own UPIC system15, whilst
Fig. 2
the system translates each individual line into a note; the resulting cacophony of
whole. Xenakis was fascinated by the form of the Hyperbolic Paraboloid (Fig. 3)
all the combined sounds also generates the perception of an undulating dynamic
which too features this characteristic explored in his music: whilst every element
in a hyperbolic paraboloid is a straight line, the perceived whole is a twisting and
smoothly curved surface. These ideas were manifested spatially in the design of the
Fig. 3
Example Hyperbolic
Paraboloid.
Philips Pavilion (Fig. 4). This is an example of how a perceived audial experience
has become effectively formalised to generate similar visual (and then spatial)
experiences. However, Xenakis was not the only proponent of the graphical score
and whilst his methods of representation reveal some ways in which the experience
of music can be perceived in static representations, others began to extend the
boundaries of how 2 dimensional spatial representations in fact become sonic
experiences in themselves.
Mark Applebaums Metaphysics of Notation (2010) rejects the standard
use of dimensional axis to represent specific attributes of sound and time.
His work instead relies solely on subjective associations and metaphorical
Fig. 4
Philips Pavilion
Xenakis & Le Corbusier.
The computerised UPIC system translated the vertical position of a mark into pitch and its
horizontal length into duration.
16
Mallgrave, H. F. (2010)
17
Arnold, R. (2010)
15
11
associative language are revealed by intrinsic similarities between the performers
interpretations of certain passages of shapes.17 Some of these are simply carried
across from the performers traditional training of scores; sweeping and curving
lines become long notes changing in pitch respectively, and sections of repetitive
Fig. 5
Bouba.
Fig. 6
95% of subjects assigned Kiki to the spiky form and Bouba to the rounded blob.
Speculatively this could be connected to the visual graphemes used to represent
the verbal sounds. For example the letters of the word Kiki feature sharper, more
angular lines than those in the word Bouba, which ubiquitously features rounded
letters. This implies that there may be some aspect of reading the shapes in the
graphical score using the musicians functions of verbal language, developed in
parallel to their musical training. However, further work on sound-shape correlations
by Daphne Maurer in 2006 observed that the Kiki/Bouba effect was also prevalent
in toddlers as young as 2.5 years old.19 Whilst the effect was slightly less prominent,
the fact that the toddlers had yet to develop reading skills shows that the shapesound inference is primarily based on the audible sound rather than inferences back
to a visual representation of the words. Ramachandran and Hubbard hypothesised
Khler, W. (1929)
Maurer D, Pathman T & Mondloch C.J. (2006)
20
Ramachandran, V. S., & Hubbard, E. M. (2001)
18
19
12
that this association is built through the use and shape of the mouth in creating
the sounds of the nonsense words: the angular figure mimics the sharp phonemic
inflections of the sound kiki, as well as the sharp inflection of the tongue on the
palate.20 From this we can see how our ability to translate shapes into sounds
is also derived from an embodied application of using our mouths and lips to
produce different sounds. However, many musicians have refined interactions with
sound (other than using their vocal chords) that they use to interpret Applebaums
scores. For these examples we must look at the coupling of body to instrument,
and then to the affordances of the instrument in terms of the corporeal motion used
to produce sounds. The limitations and facilitations of the instrument will invariably
affect the interpretation of the visual form. The bodily motion that changes the
pitch of a trombone for example is very different to that of playing a cello or flute;
as such, associations of movement across the score could yield different audible
results. The empathic association that allows the symbols to be read as sound
generating movements is another factor in interpreting these scores, and can be
generated through the very act of drawing the score. The signification of the musical
expression has emerged from a motion, such as a sweeping brush stroke or flick
that has imprinted its temporal history into a static symbol. The intentionality of the
artist in interpreting the symbol can be read through their empathic comprehension
of how the mark was made, this can then be manifested as corporeal musical
expression.21 We have seen how forms on a 2 dimensional plane can infer audible
responses, through convention, embodied concepts and even inferences to
motion and gesture that inform the instrumental output. This inferred motion and
mental animation of the symbols on the score as they are interpreted displays the
limitations of 2D graphical representations. Music is dynamic and it appears that our
spatial representation and cognition of music can be further augmented through a
dynamic spatial medium.
Corporeal imitation, articulation and expression are further investigated in Leman, M. (2008)
p.103,123,141
21
13
Animated Music.
The medium of film and in particular the animations of Oskar Fischinger address
some of the dimensional abstractions of graphical scores as the visual space and
animation occupy the same temporal dimension as the music itself. Whilst the
visual metaphors between shape and sound such as size to volume and shape
to timbre etc. are still used to great effect in Fischingers films, the possibilities of
movement are released by the animated medium. Musical objects now exhibit
speed and characteristics of movement that bring with them their own physical
and aural associations. Objects that exist within the dimension of time (unlike the
images of Applebaum and Xenakis) can swoop, shake, dart, fold in on themselves
and perform a wide variety of dynamic behaviours; as a result the animations get
closer to synthesising these intuitive connections between sound and visual form.
Susannah Knights writes of Oskars animations: Both media unfold and engage the
audio and visual senses over time, similarly capable of evoking an automatic sense
of narrative expectation. Secondly, both create an illusion of a gestalt, through the
audience psychologically connecting component parts which run in a temporal
sequence.22 The dynamic properties of Fischingers films reveal how the viewers
expectations of imminent musical and visual activity can become synchronised to
further enhance the perception of an audio-visual gestalt. Our ability to expect future
events based on current conditions is based on observing recurring phenomena
to build an understanding of how objects behave. If we continue to use embodied
experiences and interactions as a basis for generating audio-visual associations
22
14
we can start looking at our spatial expectations and see how these interact with our
musical expectations. For example, the resolution of objects in motion is for them to
fall downwards under gravity and come vertically to rest. We use this expectation
and understanding of the mechanics of our environment to be able to perform
actions, such as catching a falling ball. The same kind of resolution also appears
in chord sequences. Sustained chords and perfect cadences also create a tension
that is traditionally expected to be resolved. This effect of expected resolution is
evident in our melodies of speech, for example an upwards inflection at the end of a
Downbeat 00:37
Downbeat 00:38
Expectation 00:39
Resolution 00:40
23
15
Sound Gestures.
Fischingers films visibly correlate to our embodied perception of musics motions
and dynamic patterns far more effectively than static graphical symbols. However,
both mediums achieve their audio-spatial inferences through embodied concepts
and common sensory experiences. Mark Applebaums piece Aphasia (2010)
combines hand gestures with music to create an audiospatial gestalt that is created
around the body itself. The medium of hand gestures not only moves the experience
of the music into a dynamic 3 dimensional realm but immediately generates an
implication of action to the audience. We use our hands as a primary means to
interact with the world, therefore a movement of a hand implies an action is taking
place in which there will be an effect. Unlike the dancing shapes and sprites
of Fischingers films, that are simply characterised by motional behaviours that
complement the music, the movements of the hands are perceived to have a direct
causal relationship to the music that they accompany. The electronically altered
vocal samples and abstract audio snippets used to create the audio of the piece
do not remind us of any traditional instrumentation, however the hand gestures
appear so perfectly matched to the audio that it is difficult to imagine which is
derived from the other, or if they were composed in parallel. Applebaums advanced
and multidisciplinary instrumental experience has given him a sophisticated
understanding of how motions and hand actions produce sounds with objects. His
search for new musical interactions has led to the invention of bespoke instruments
such as the Mousketeer (Fig. 8). Applebaum avoids the motional associations that
Fig. 8 - Mark Applebaum playing his Mouseketeer.
16
and gradually learns the vocabulary of the piece as it progresses and new sounds
and motions are revealed. By producing the illusion that the motions actively create
the sounds, the actions that Applebaum uses appear to imbue the sounds with a
physical presence and spatial attributes. The piece opens with several disparate
beats to the chest, creating a short percussive knocking sound. After the sixth
instance of this action, instead of returning his hand to his lap his arm opens out
away from his body (Fig. 9 - Aphasia.mp4 - 00:34), the knock appears to echo
around this new space that the gesture has created. This perception localises the
Fig. 9
Echo Gesture.
sound of the knock to the space around the body that is perceived to be its source.
Similarly at (Fig. 10 - Aphasia.mp4 - 02:12) Applebaum strenuously pulls his hands
apart emulating a tension that is audibly reflected with a series of strained rubbery
squeaks. The perception that the sound is being tangibly stretched apart is a very
successful example of the sounds assuming a spatial and tangible form in the mind.
It must be noted at this point that a core aspect of embodied cognition is that for
any intentionality or experience to be embodied it must also be situated. Peoples
conceptions of space differ for different spaces and are a joint product of perception
and action appropriate for those spaces24 This is the idea that behaviour is situated
in its architectural environment.25 A person might be more likely to dance if music is
played in a dance studio however would perhaps act differently in a church or office.
Fig. 10
Stretch Gesture.
The gestures of Applebaums Aphasia are not only afforded by the biomechanical
trajectories of the human body but also influenced (or indeed generated) by the
wider spatial context and the perception and action appropriate for the environment
itself. The piece is primarily viewed on a blank stage or in front of a black
background as shown in the filmed version. There are no visual clues to the spatial
context in terms of architecture, however the piece is by nature a performance so
there is an intrinsic factor of Audience that informs the spatial representations of the
music that Applebaum creates. This audience is either a real audience in the stage
context or the camera in the filmed context. This directional connection between
audience and performer changes how the gestures are interpreted, it would
certainly not have the same effect if viewed from behind. We can see how the study
of spatio-musical chiasm in a dynamic 3 dimensional space demands embodied
and situated means of analysis; the spatial context and relative perspectives of
people within the system deeply affect the resulting musical interpretation.
Tversky, B. (2008) p.202
Behaviour and language are spatially situated but not environmentally determined Dourish, P.
(2004) p89
24
25
17
By initially looking at examples of prior art we have seen how static and dynamic 2
and 3 dimensional spaces have been used to naturally mediate our perceptions of
sound, music and space. I have also described how these gestalts are formed in
the mind by utilising the approach of embodied cognition26,27 and the action oriented
ontology of music described by Leman. The precedents have not only examined the
spatial inferences of individual sounds, but also touched on how the sounds (and
indeed shapes and gestures) in the music relate to each other in time and generate
expectations, moments and contrasts that are attuned to our embodied experience
of the physical world. This aspect of music distinguishes it from being simply
sequences of sound or noise and deserves greater examination as it displays more
fundamental ways in which space and music are shared in the brain.
26
27
Tversky, B. (2008)
Shapiro, L. (2010)
18
Minsky M. (1981) In Clynes M, ed. Music, Mind and the Brain: The Neuropsychology of Music p.29
Stages of learning outlined in Piaget. (1970)
Minsky M. (1981) In Clynes M, ed. Music, Mind and the Brain: The Neuropsychology of Music p.30
Minsky M. (1981) In Clynes M, ed. Music, Mind and the Brain: The Neuropsychology of Music p.37-38
19
20
to their practiced experience and reliance on both spatial and temporal narratives
and memory. 43,44,45,46 The increased hippocampal neuroplasticity clearly illustrates
the essentiality of the hippocampus in perceiving narrative and temporal contrast,
and thus in facilitating our higher level emotional responses to sensitively curated
temporal experiences.
This associative understanding of human navigation through space is currently in
common use by architects and urban planners to generate consistent elements
and extensions of urban form. The reason for this is not neurologically based
but has evolved as a refined human sensitivity to city making. Our experience of
the city is built on a framework of how spaces relate to each other. We learn the
relationships between roads, streets, alleys and open spaces as we move through
the city, these relationships are used as a subconscious mental framework that
we use to comprehend and operate within other cities and spaces. By examining
the processes and functions of the hippocampus, we have seen that music and
dynamic navigation of space share this cognitive process, yet the parallels between
sensitive urban design and musical compositions are rarely consciously used.
Many metaphors such as harmony and rhythm are used to discuss and design
architectural proposals and their relationships to their context, yet few actively
engage the processes of musical composition to augment the embodied experience
of the city that they are developing. Perhaps by further understanding the deeper
complexities of music and its interaction with mind, body and place we could use
Music Aided Design (MAD) to develop deeper and more emotionally engaging
experiences of architecture and the built environment. This specific area of spatial
musicality deserves a lot more research but we are beginning to see how musical
experiences and spatial experiences are not as distinct as previously thought.
Maguire, E.A. Gadian D.G, Johnsrude I.S, Good C.D, Ashburner J, Frackowiak R.S, Frith C.D. (2000)
Gaser, C., & Schlaug, G. (2003)
45
Rodrigues, A. C., Loureiro, M. A., & Caramelli, P. (2010)
46
Herdener, M., Esposito, F., di Salle, F., Boller, C., Hilti, C. C., Habermeyer, B. & Cattapan-Ludewig, K.
(2010)
43
44
21
Participant
Body Tracking Camera
Experimental Research.
We have examined how recent research and development in neuroscience, music
cognition and embodied cognition provide a framework for a spatial interaction
based ontology of music, this has been illustrated through basic observations
about how our human physiology affords particular experiences of the world and
also by examining similarities in neurological processes. A series of 6 interactive
experiments were developed in order to directly expand upon the multimodal
experiences described in the first chapters. The information and precedents
discussed suggest that we should also be able to form a symbiotic perception of
Fig. 12 - Experiment Setup.
Custom Software
space and music that uses our knowledge and experience of both to produce an
intuitively interactive gestalt.
Surround Audio
The 6 experiments featured 3 focussed on sound and 3 focussed on music
and were followed by a short questionnaire about the different interactions and the
users experiences (Fig. 12-13). The experiments were tested on students ranging
in musical ability and spatial awareness and were placed in a category of either
Architect (architecture students with less than 5 years musical experience),
Musician (non architecture student with over 5 years of musical experience),
Musician Architect (Architecture student with over 5 years musical experience)
or Non Musician (does not study architecture and has less than 5 years musical
experience). The main body of the questions required an X on a spectrum between
Agree and Disagree; the data of each questionnaire was calculated by measuring
the distance on the spectrum and dividing it by the length to create a coefficient of
Fig. 13 - Speaker placement and Camera field of view.
22
23
Sonic Compass.
Experiment 2
The second experiment used a similar mechanic of reaching away from the body to
Fig. 15 - Reaching into Sound Compass.
produce a sine wave pitch yet the responsive area was arranged radially around the
center of the body as a ring at arms reach. The Highest pitches pointed towards the
front of the lecture room and the lowest pointed towards the rear of the lecture room.
The users were able to tap into this audial sound compass by extending their hands
out and feeling for direction with the sound (Fig. 15). The users were not told how
the sounds were oriented but were given 2 minutes to explore their affective audible
environment. The users were then blindfolded and disorientated by revolving on
the spot a number of times. The blindfolded, disoriented users were asked to
simply point towards the lectern at the front of the room. Despite only experiencing
this interaction for 2 minutes, 88% of the subjects were able to successfully and
instinctively use the sound as an audible sense of orientation and point in the correct
direction. This effectively shows how sound information can be imbued with spatial
orientation information and that even with only 2 minutes of exposure to the system
the users were able to use their augmented senses of proprioception and hearing to
make confident spatial decisions about their orientation in the space.
The speakers used for this experiment completely surrounded the user such that no sense of stereographic orientation could be used
47
24
Sounds in Space.
Experiment 3
This ability to map the experience of a sound into a 3 dimensional position through
bodily interaction illustrates the potential of sensory chiasm when perceiving spatial
objects and sounds. The aural connections between sound and shape discussed in
the first chapter imply that the perceived shape of the sound would be affected by
the timbre and texture as well as just the interactive spatial experience. In the future,
this experiment could be expanded by using different types of sound as well as just
different positions, pitches and sizes to test if perceptions of shape and form were
25
smooth transition between each piece when the users moved between zones. The
users explored the room to discover the new audible significance of the different
spaces, most of them developed new preferences for particular positions in the
room. 96% of the users found that the music significantly changed the character of
the spaces, and 96% were able to perceive the spatial presence of a modulating
sound field. The experiment also caused some users to jump from one zone to
another at the end of a bar to further control the music as well as passively observe
the changes. The questionnaire required the users to draw a map of the sound
spaces on a floor plan of the room. 67% of the users accurately illustrated the
spatial layout of the music zones (Fig. 19). The remaining participants produced
less accurate diagrams of how the room was divided, yet still picked up some
attributes of the zones such as the fact that the zones overlapped (Fig. 20). Only
54% of all architecture students we able to successfully illustrate the zone divisions,
whereas the other 46% had close representations but generally overcomplicated
the spatial arrangement; this suggests that the architects refined skill for thinking in
plan had not necessarily expanded their ability to map the sounds into space but
perhaps added extra complexity from other factors in the music. However, 70% of
the musicians were able to draw the zones in plan with near perfect accuracy. This
is evidence that instrumental experience builds and reinforces strong associations
26
27
insulation will appear muffled and
Haptic Music.
Experiment 6
28
In this experiment the subject was asked to stand in the middle of the space whilst
repetitive nature of the beat and chord progression was chosen in order to introduce
the music began playing through the speakers. The subject was then simply asked
the subjects to the temporal framework of the music quickly such that they were able
to compress the music but was given no visual or gestural cues of how this
might be done. The verbal cue to compress the sound elicited various different
body movements. By compressing the sound and reducing the music to the
responses from the subjects, some extended all their limbs before crouching
bassline and underlying beat, the subjects were able to contribute to the dynamic
and curling into a ball (Fig. 24), others raised their hands up then pressed them
changes of the music, generating a literal tension and anticipation of resolution that
together towards the floor and most of the subjects brought their hands together as
is felt both musically and physically. This is a similar effect to the stretching action
if squeezing a large ball (Fig. 25-26). These initial responses indicate the success
and audio used in Applebaums Aphasia. 83% of the musician subjects agreed that
the structure of the music influenced their choice of motion, yet only 40% of the non-
subjects realised that their initial motions affected the music in an expected manner
musician non-architect group found this to be the case. The musicians had a clearly
they almost immediately began to interact with the music as an object or field in
space. 96% of the subjects agreed that the interaction between body and music
the primary aim of the non-musicians actions was simply to examine the effect of
felt natural with a strong average agreement of 0.89. The experiment was not only
the resulting tone of the sound. 100% of the Architect musician group agreed that
devised to test whether a spatial sense of music could be achieved through intuitive
the structure of the music influenced their choice of movement with an average
interaction but also to test how the temporal structure of the music would influence
the subjects choice of movement once the perceptual coupling between action and
The music was of a popular genre to the subject demographic yet not well known enough to distract
the subject from the spatial interaction
49
Experiment 6:
Haptic Music
100
100
60
100
92
0.94
0.86
0.65
0.84
Architects
Musicians
Architect Musicians
Non Either
% Agreement
0.84
88
83
100
80
88
0.79
0.77
0.87
0.64
0.77
Agreement Coefficient
Experiment 5:
88
33
80
40
63
0.79
0.64
0.85
0.39
0.68
75
50
80
60
67
0.80
0.61
0.75
0.61
0.71
75
83
100
80
83
0.66
0.75
0.87
0.75
0.74
60
26
45
32
47
40
59
60
58
60
34
43
37
21
30
40
60
59
40
47
40
54
50
30
100
83
100
100
96
0.91
0.69
0.96
0.97
0.88
60
46
15
57
43
40
42
59
60
60
56
60
47
55
50
60
60
59
57
60
60
44
52
60
100
83
100
100
96
0.87
0.82
0.87
0.93
0.87
60
57
60
57
46
30
42
60
60
60
33
60
36
55
50
60
45
59
59
57
60
44
53
49
88
100
100
100
96
0.89
0.79
0.93
0.94
0.88
58
47
43
60
46
50
59
60
60
60
47
60
40
30
50
60
60
59
60
55
58
40
60
49
5
8
5
5
17
0
0
42
2
0
0
0
2
0
15
0
6
0
7
12
0
4
5
12
5
10
0
17
0
0
0.02
0.24
0.09
0.08
0.10
45
11
15
60
30
52
59
46
60
44
26
60
17
58
52
24
60
59
60
42
12
54
30
40
Spaces of Sound
59
44
58
51
44
15
33
60
60
60
57
60
45
56
50
60
52
42
60
57
55
45
60
48
10
100
83
100
100
96
0.89
0.69
0.92
0.93
0.85
75
83
100
40
75
0.80
0.74
0.93
0.48
0.75
58
56
15
60
30
54
47
30
11
60
57
60
40
56
60
30
60
59
46
11
13
53
58
52
11
100
83
100
100
96
0.95
0.86
0.96
0.91
0.92
57
54
30
59
60
50
45
59
60
60
57
60
50
56
60
60
60
60
60
54
52
56
53
53
88
100
100
80
92
0.85
0.84
0.98
0.82
0.87
60
55
50
51
42
52
46
16
60
60
58
60
54
56
60
60
60
60
59
50
30
45
53
53
General
12
13
60
30
30
60
30
30
59
0
50
30
34
60
45
58
50
60
60
54
60
4
47
55
13
3
4
6
Experiment 4:
Architecture
Physics
Engineering
Architecture
History
Chemistry
Architecture
Architecture
History of Art
Architecture
Architecture
Architecture
Architecture
Architecture
Architecture
Architecture
Architecture
Theology
Architecture
PPS
History
Engineering
PPS
Biological Sciences
63
83
40
60
Successfully orientated
1
1
1
0
1
1
0
0
0
1
0
1
1
0
0
0
1
1
0
0
0
1
0
0
Musician Subject
Sample size
8
6
5
5
F
M
M
M
M
F
M
F
F
M
M
F
F
M
F
F
M
F
F
F
F
M
F
F
M/F
59
54
60
37
0
57
59
51
10
59
47
60
37
58
8
60
45
60
60
45
45
45
52
40
Sounds in Space
Experiment 3:
Sonic Compass
Experiment 2:
3
5
1
6
5
3
6
2
5
6
3
5
5
4
6
6
6
6
3
4
6
6
4
4
2
6
3
3
1
2
4
3
3
4
6
6
1
5
4
4
4
1
6
6
4
3
6
6
6
2
2
2
2
4
2
4
2
2
5
3
4
3
3
1
5
2
2
1
1
2
5
3
5
3
5
4
4
6
3
5
6
3
2
4
6
6
2
3
3
3
4
3
3
4
2
2
1
1
4
1
3
1
1
1
1
1
1
1
3
1
1
2
1
4
1
2
2
1
1
1
6
4
6
6
4
6
3
4
5
3
1
5
3
2
4
1
2
5
5
2
1
1
2
5
4
4
6
5
6
5
5
6
4
5
4
2
2
2
5
5
2
5
5
5
5
5
3
5
1 2 3 4 5 6
60
53
60
50
46
57
59
60
34
0
50
60
15
58
60
54
60
37
60
57
55
55
50
55
2
2
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
1
1
Experiment 1:
Experiment >
Question Number >
Subject Number v
29
30
Discussion.
The results of the experiments clearly show that we have the ability to perceive
both music and sound as spatial forms. We naturally use metaphors to mediate
meaning between musical and spatial experiences yet rarely actively engage our
experiential knowledge of one with the other. This ability is not reserved to those with
neurological synaesthesia, it is built through our everyday experience. It is clear that
we have moved beyond the question Are music and space cognitively connected?
to How can we apply this unified understanding of embodied perception to
design? and particularly How can the diciplines of music and architecture
augment and inform each other?
I have demonstrated that by using emerging technologies and systems, both
spatial and audible presence can be combined to augment a users navigation and
experience of architecture. The results of the experiments speculate that these forms
of navigable music could become an architecture unto themselves. In experiments 3
and 4 the subjects used words such as columns, corridors, walls and rooms
to describe the sounds they interacted with. This shows the potential to create
an immaterial architecture made of music that is still spatially navigated. I have
illustrated that the instrumental experience of the participants heavily influenced
their audio-spatial ability; however the architecture itself now has the potential to
become the instrument that musicians and non-musicians alike will inadvertently
play (and thus practice) with their bodies as well as their eyes. This implies that even
non musicians could become musicians of space as they are exposed to this type of
architecture.
31
The experiments have not only shown that we perceive spatial inferences in sounds
but also that music can have an affective role in our spatial interaction when the two
are coupled. This affective quality of music could extend the idea of an architecture
made of sound to become active navigational sound architecture for the blind,
where sounds are naturally imbued with spatial characteristics. Ascending scales
of notes could have clear spatial connotations to obstacles such as stairs, or
volumes of sounds with different textures could be used to generate perceptions
of proximity to various objects in a room. The refined musical ability of temporal
pattern recognition could be attuned to the motions of the body as it moves through
space. Phrases of music can now actually become musical places which develop
and change as we move from one to another. This combines both uses of the
hippocampus discussed and could be used as a framework to distinguish different
spaces and their relationships to each other. In the same way as Minsky described
musical learning, this system of navigable music would perceptually build into
intuitive wayfinding devices that cue memories of what lies ahead as inhabitants
move through them. These are just a few examples of embodied music concepts
that could contribute to a much larger vocabulary and temporal grammar of intuitive
architectural sound elements.
The implementation and curation of these elements need not be functional and
could become new kinds of architectural detail. Experiment 4 illustrated how our
visual perceptions of space are overridden by the audible characteristics, in the
same way that film soundtracks significantly enhance or alter the visual narrative.
Pockets of music can now be placed in buildings that change and augment the
qualities of that particular space. The study of how and what music makes us feel is
a larger avenue of research but the architect has the opportunity to change the way
their buildings are listened to as well as seen. The music and the architecture have
the potential to be composed together: Michael Gondrys film for Star Guitar by
the Chemical Brothers is a fantastic example of how music changes our perception
of images, and also how features of space could be heard and augmented in
this way (StarGuitarExperiment.mp4). Every element in the landscape of the train
journey is highlighted by a different sound; attention is drawn to every element in
the scene as it is revealed and audibly signified in the music. Architecture could
use similar technology to literally become audible music in this way, or alternatively
(and perhaps more interestingly) sequences of spaces could be designed with
these principles of Music Aided Design to actively stimulate the same temporal
continuities, contrasts and empathic motions that move us so deeply in music.
32
There is the opportunity to apply this musical intelligence during the appraisal
and brief stages of architectural design. During the contextual analysis of a
site, particular attention could be given to the details, rhythms and forms of the
surrounding spaces and buildings, analysing the developments and changes of
a dynamic perspective through the area. Musical composition could be used to
understand how these rhythmic and dynamic textural changes lead to a dynamic
architectural language of the area as well as just a static, stylistic or materialistic
language. The gap (or silence) of the undeveloped site in the routes through the
area could be conceptually developed based on the expectations, contrasts and
rhythms (to name a few) of the temporal moments either side. This could even be
achieved by composing the music of the context and experimenting with potential
surprises or continuities in the currently empty site. The approaches and departures
from the building could therefore be actively engaging the visual and spatial aspects
of the site with a larger musical narrative. This would continue inside the building
where languages of exposition, development and recapitulation (taking sonata
form for example) could be used to create gradual changes, moments of surprise,
steady continuities or moments to reconfirm or change our perceptions of the whole.
The results of this discussion also have implications in other disciplines. Similar
technologies and bespoke software could allow sounds to be actively sculpted,
stretched, smoothed, textured and manipulated as spatial objects. The sounds
could even be passed between multiple collaborators or moulded and developed
together. This could have huge potential in creating music through dance, and also
in electronic sound design and musical composition.
The precedent studies, neurological studies and experimental studies have all
revealed that they may be explored in far greater depth than I have done here.
The experiments deserve a greater sample size to critically examine the changes
between refined spatial awareness and musical ability. The study would also benefit
from exploring the spatial responses to many different types of music and sound
in each experiment. To extend this research, empathy and empathic involvement
with space, motion and music will also need more comprehensive analysis. This
could even extend into how mirror neurons activate the sensorimotor system just by
perceiving the sounds and motions of other objects and people. This dissertation
has been a proof of concept in what clearly has the potential to be a much larger
practical and theoretical pursuit in the disciplines of both architecture and music.
33
34
Ekstrom A.D, Kahana M.J, Caplan J.B, Fields T.A, Isham E.A, Newman E.L, Fried I. (2003).
Cellular networks underlying human spatial navigation. Nature 425:184188.
Erneling, C. E., & Johnson, D. M. (Eds.). (2005). The mind as a scientific object: between
brain and culture. Oxford University Press: USA.
Bibliography.
35
Knight, R (1996). Contribution of human hippocampal region to novelty detection. Nature
Pratt, C.C (1930). The spatial character of high and low tones Journal of Experimental
383:256259.
Knights S, (2012). Visual Music: the films of Oskar Fischinger. Cambridge: History of Art
dept. Dissertation
Ramachandran, V. S., & Hubbard, E. M. (2001). Synaesthesia - a window into perception,
Kumaran D, Maguire EA (2007). Which computational mechanisms operate in the
MIT Press
Sacks, O, (2008). Musicophilia; Tales of music and the Brain, New York: Knopf
Lidji, P., Kolinsky, R., Lochy, A., & Morais, J. (2007). Spatial associations for musical
stimuli: A piano in the head?. Journal of Experimental Psychology: Human Perception and
17(20), R892-R893.
Paris 157:791794.
Strange B.A, Fletcher P.C, Henson R.N, Friston K.J, Dolan R.J. (1999). Segregating the
Maguire, E.A. Gadian D.G, Johnsrude I.S, Good C.D, Ashburner J, Frackowiak R.S, Frith C.D.
(2000). Navigation-related structural change in the hippocampi of taxi drivers. Proc Natl
Acad Sci USA 97: 4398-4403.
Strange B.A, Dolan R.J. (2001). Adaptive anterior hippocampal responses to oddball stimuli.
Hippocampus 11:690698.
36
Appendix:
www.felixfaire.com