Experimental research about the correlation of sound and

image in Motion Graphics

YouJeong PAIK*, Mahnoo KWON**, Adam SINYKIN***, JaeMyoung KIM****

*Graduate School of Digital Design, KyungSung University & 110,Daeyeon-dong, Nam-gu, Pusan, 608-736,
KOREA, amidalapaik@hotmail.com
**Graduate School of Digital Design, KyungSung University & 110,Daeyeon-dong, Nam-gu, Pusan, 608-736,
KOREA,, mahnoo@star.ks.ac.kr
***Graduate School of Digital Design, KyungSung University & 110,Daeyeon-dong, Nam-gu, Pusan, 608736, KOREA,, adam@star.ks.ac.kr
****Graduate School of Digital Design, KyungSung University & 110,Daeyeon-dong, Nam-gu, Pusan, 608736, KOREA,, jmgim@star.ks.ac.kr

Abstract:

The purpose of this study is to highlight aspects of audio/visual synergy within the general field of motion graphic

technology.

Contemporary research suggests that the preeminence of image in motion graphic production is giving way to a

strategic fusion of sound and image as the paramount carrier of information and emotional impact.
Visual information is conveyed through a perceptual media.

Audio information is conveyed through a physical medium, (the

body actually resonates with incoming acoustic vibrations) and it is primarily through physical means that subliminal,
emotional and linguistic information is transmitted.

This was not always the received view.

visual information was the signature carrier of media data and media effect.

For years it was accepted that

Recent data, especially IT and neuro-linguistic

research, advance the notion that acoustic effects are actually more effective than information conveyed by purely visual
means. Motion graphic technology is time based media with distinctive spatial characteristics, and is well suited as a
laboratory for experiments in media effect with regard to sound and image.

This paper will outline a series of prototypes

and experiments utilizing MindSensor II software and brain wave topology interface (ECTI) in the relative effectiveness of
various sound/image combinations conveyed through motion graphic media.
This paper will conclude with an executive summery of the experimental data, and a review of recent developments in the
synergistic effects of sound and image.
Key words: Motion Graphics, Cognitive information, Physical Information

1. Introduction
Motion Graphics is one of the most persuasive multimedia genres. It has the potential to fuse audio and visual
stimulus more effectively than any other digital media. Within this genera, as the technology improves, audio data
will become as important a carrier of information as visual data. Motion Graphic work can express powerful
emotional content, exploiting the dynamic interrelation of visual kinetics and acoustic texture and rhythm.[1] As
Marshall McLuhan argues, the mechanics of the media itself is as important a carrier of emotional effect as is
the cognitive content.[2] Acoustic material, the prime driver of physical and emotional effects in multimedia,

will gain increasing importance in the deployment of motion graphics technology.

In this paper, experimental research will focus on physical information (i.e. audio.) and its interrelationship
with a series of stable parameters of visual (i.e. cognitive) information. This study will establish an experimental
methodology in order to gage the varying effects and correlation of cognitive and physical information on human
brain waves.
This paper will conclude with an executive summery of the experimental data, and a review of recent
developments in the synergistic effects of sound and image.
2. Method
The first step of this research is making audio-visual prototype samples. The samples consist 8
categories.<Table1> Visual classification lend its concept from JAGDA(Japan graphic designers association)
model.[3] Next step, every subjects are exposed these 4 audio-visual samples. Finally, gathering brainwave data
and results were analyzed with statistic software. To make up for the weak points of quantitative analysis in-depth
interview followed.
The topography of cognitive activity and emotional responses to external stimuli can be studied by analyzing
the variation of EEG potentials in brainwaves. Hans Berger, the discoverer of brainwaves, was the first to
demonstrate that as aspects of the psyche can be measured using electronic apparatus. Berger argued that brain
waves constituted a rough analog to thought and emotion.[4] This research will explore the correlation of sound
and image in a scalable study of brainwaves in the context of motional graphic stimuli.
Human emotions can not be error-freely measured into quantitative data because they have marked tendency of
subjective aspects. Emotion can be defined as strong psychological change and inevitably accompanied with
physiological symptoms.[5] Brainwave is the most salient of those symptoms. So this research tried to capture
meaningful data about subjective emotion through objective detect device like brainwave measurement system.

Table.1 Experiment design

Audio

visual

Accidental

Mechanical

Organic

Freehand

Audio

Sample 1

Sample 3

Sample 5

Sample 7

No audio

Sample 2

Sample 4

Sample 6

Sample 8

3. Background
3-1 Motion Graphics
Motion graphics refers to a synthesis of elements common to animation and traditional visual design. It is
distinguished from other design media in that it moves by timeline enabling design professionals to compose
images using musical and painterly techniques.[6]
Motion graphics achieved preeminence in the design world in a surprisingly short period, and is now
considered the privileged genre within the field of visual design. Applications for motion graphics include
advertising, music video, cinema, TV, Web and DVD. With the integration of UI technology, motion graphics
has become an interactive genre, incorporating hypertext and non-linear elements.[7]

3-2 Characteristic of the brain waves

There are four types of brainwaves; alpha (8~13Hz), beta (14~50Hz), theta (4~7Hz), and delta (3~5Hz). Alpha
and beta waves are generally referred to as fast waves and theta and delta waves as slow wave in reference to
their electronic amplitude, but in general, the stimulus response of various brainwaves tend to be interrelated.
Normally, alpha waves are characteristic electronic signifier in routine activities. In case of tension or excitement
beta waves become dominant. During rest or relaxation and deep sleep, theta and delta waves appear.[8]
Certain tasks that require high levels of attention tend to suppress alpha or non-synchronous waves and
stimulate beta waves.
3-2-1 alpha waves ()
Characteristic of routine activities.

Frequency range 8~12Hz and amplitude of frequency 50V.

3-2-2 beta waves ()

Indicative of activity that requires attention or immersion. Frequency range 14~50Hz and amplitude 5~10V. 33-2-3 theta waves ()
Theta waves appear during strong excitation or shallow sleep condition. Frequency range 4~7Hz and amplitude is
high and regular.
3-2-4 delta waves ()
Delta waves are called sleeping waves. Delta waves are frequently symptomatic of trauma to the cerebrum,
especially in children. Frequency range 3~5Hz and amplitude 200v.
3-3 Physical information and cognitive information
Human beings have five senses; sight, hearing, smell, touch and taste. The sense of sight and hearing are most
important for operative communication in multimedia. For this reason, computer systems rely on sight and
hearing, almost exclusively. Even here, the role of sound is underexploited. If computer systems or human
perception lack audio stimulus, information redundancy increases.[9] UI research indicates that with enhanced
deployment of audio interface, information efficiency could be maximized.
The sense of sight is associated with cognitive processes. The sense of hearing is actually an emotional process
because acoustic vibrations directly stimulate the human body.[10]<fig.1> Motion graphic designers utilize sound
elements because they are the most important emotional element of the transmission process. However, visual
and audio elements are interrelated.

Fig.1 Cognitive, Physical information

4. Experiment design
4-1 laboratory setting and participants
The experiment is conducted in soundproof environment. A random selection of subjects with set age and

educational parameters participated.<fig.2>

Ten subjects participated pre-experiment and 10 males and 11 females participated in the main
experiment.<fig.3> Participants are students of Kyungsung university.

Fig.2 Soundproof walls

Fig.3 subject interview

4-2 Equipment
Normally medical brain wave detectors consist of more than 20 sensors and each sensor has a sucker. For this
reason all participants have to shave their hairs for experiment. And the purpose of medical sensor is to catch
microscopic abnormal signs of brain, for example epilepsy. So they are not fit for measuring emotional changes.
But thanks to the radio frequency technology development it is possible now to record overall change pattern of
brainwave with only one hair band sensor.
This experiment uses a single-wide sensor system called Brain Builder Unit. The software is called Mind
Sensor II, which is commonly used to gather and analyze brainwaves. Both analytical applications operate in
Windows XP environment. Statistical analysis used SPSS 10.0 (Statistical packages for Social Sciences)
4-2-1 Interface: Brain Builder
4-2-2 Software: Mind SensorII
4-2-3 EEG spectrum: frequency range 1 ~ 24 Hz / pressure 1Vpp / density 0.6V / sampling frequency 128Hz /
channel variant: 1:1
EEG band is set from anterior to posterior lobe for brainwave signature.
4-3 Visual Variables
In the context of contemporary perceptual and color perception research, four motion graphic animations are
utilized; accidental, mechanical, organic, and freehand. Visual prototypes were classified into four groups as
follow figure. <Fig. 4> Each type has opposite visual meaning by vertical and horizontal direction.

Fig.4 Visual category

According to these visual classification principles four motion graphic works were made below.
4-3-1. Accidental: images without perceptible morphology.

Fig.5 Accidental Image sample

4-3-2. Mechanical: refers to mathematically or mechanically generated images.

Fig.6 Mechanical Image sample

4-3-3. Organic: morphology generated through natural processes/complex environmental forces.

Fig.7 Organic Image sample

4-3-4. Freehand: an image generated by human gesture(s).

Fig. 8 Freehand Image sample

4-4 Audio variables

The audio variables were made by arbitrary criteria because which sound is exactly matched with the visual
images can not be scientifically verified. Sound matching is not a realm of science but of art. We made four types
of sound samples fit for visual images using various audio samples and sound effects. The coordinates and
sampling procedures are analogous to the methods used in the composition of the visual files.
4-5 Procedure
The measurement of brain waves was conducted in 2 stages: Pretest and Main test. Pretest cataloged EEG
characteristics and subject bias. Each participant was exposed to stimulus of varying density for trial periods of 5
minutes, forty seconds. Presentation classification included accidental, mechanical, organic, and freehand visual
files. Each motion graphic animation lasted for 20 seconds.
The Main Test was repeated 8 times per person. There were a total number of eight AV files. Each person was
tested for 1 minute and 20 seconds per AV file.

5. Results
Five categories of brainwave data were collected; Beta, fast alpha, mid alpha, slow alpha, theta waves.

Statistical analysis determines mean differences between two variables per subject/Total base.(T-test and F-value
test) Analysis results say that T-test between male and female has no significant differences in brain wave
frequencies. And there are no statistical differences between blood types depend on F-value test. These are
partially caused by small sample size of subject. To analyze effects of audio-visual types this research calculated
dominant brain waves in each stimulus.
Main results bellow. Each cell represents a subject who has the dominant waveform profile for each variety of
AV file. (multiple response.)

Table.2 Number of subject with predominant wave(audio)

wave
Stimulus

Fast

Mid

Slow

accidental

10

15

mechanical

11

organic

11

14

freehand

11

11

Table.3 Number of subject with predominant wave(No audio)

wave
Stimulus

Fast

Mid

Slow

accidental

18

mechanical

16

organic

15

freehand

13

According to the analysis, beta and fast alpha waves were dominant in sound added situations and slow alpha
waves are distinctively notable in the silent trial. These present that every subjects exposed to sound have much
more attention to stimulus. <Table2><Table3> It was evident that the subjects perceive more information when
visual stimulus is accompanied by sound.
Organic and Freehand files elicited minimal beta activity when projected without audio. With accompanying
audio, mechanical and organic files elicited high response in beta and fast alpha waves.

6. Conclusion & Limitation of this study

As a result of our experiment, we can accept the following: If there is sound, the attention of subjects is
considerably higher. Among 4 kinds of visual stimuli, mechanical / organic has the most significant impact on
active and high-density brain waves. In other words, if sound is merged with visual data, brain wave activity is
accelerated. Each of the four visual stimuli have different wave pattern.

The variations of the 4 brain wave patterns would be the subject for continuing research. These variations
could be a factor of demographics, gender, or any number of other personal or social characteristics.

Further

research should use standardized visual files from encoded visual genres. Standardized criteria for audio files
are also central to further EEG research.
Emotional effects can hardly be measured with quantitative tools, but this experiment seeks to move toward a
methodology that could explore human emotions and motion graphics in the context of cognitive and
experimental psychology.

Such research could lead extraordinary innovations within the field of motion graphic

production. Further research could open startling vistas in the convergence of art, industrial psychology and
science.
Effective motion graphic producing is a matter of arts rather than sciences. Superb motion graphic works was
mostly made by creativity or artistic inspiration. But good designer always pursues both art and science together.
Good motion graphic design also lies in the middle of arts and science. This paper tried to handle arts
scientifically. And this kind of efforts can produce scientific method or predictable way to produce effective
visual and sound combination someday.

Reference
1. Matt Woolman, Sonic Graphics Seeing Sound, Rizzoli(2000)
2. Marshall McLuhan, Understanding Media : The Extensions of Man, McGraw Hill(1965)
3. JAGDA Hwaseon Kang trans, JAGDA textbook vol1 Visual Design, seoul : artbook(1995)
4. Tonoike, M., Yamaguchi, M., Nakayama, N. and Ikemi, Y., Brain topography study of aromacological
relaxation using EEG and MEG, in Brain Topography Today, ed. by Koga, Y., Nakata, K. and Hirata, K.
Elsevier, Tokyo, 733-739(1998)
5. Hyung-Gun Yun, Emotion and Emotional Engineering, Design Research Vol 3, Korea Design
Society(1998)
6. Peter Hall & Andrea Codrington, PAUSE: 59 MINUTES OF MOTION GRAPHICS, Universe(2000)
7. Kathleen Ziegler, Nick Greco, Tamye Riggs, Motion Graphics Film + TV, HBI(2002)
8. Daesik Kim & Jangwook Choi, Electroencephalogram, Korea Medical Books(2001)
9. David Sonnenschein, Sound Design, Michael Wiese(2001)
10. Abeles, H.F., Responses to music, In D. A. Hodges(Ed.), Handbook of music psychology, Lawrence, KS:
National Association for Music Therapy(1980)