CHAPTER ONE

INTRODUCTION TO MULTIMEDIA

Multimedia Systems

1
 CONTENTS

 Multimedia Systems Design  Multimedia Applications

 Analogue representation vs  Image processing and
digital representation recognition
 Reconstruction  Electronic messaging

 File formats  Video conferencing

 Compression  Multimedia Architecture

 Multimedia Elements  High resolution graphics display

 Images  Network architecture

 Voice commands / synthesis  Data transfer, data duplication and

 Audio / Video messages
data replication
 GIS
 Multimedia data interface
 File formats (e.g. RTF, TIFF,
 Holography
MIDI, JPEG, MPEG, AVI)
 Fractal
 Compression of data 2
 WHAT IS MULTIMEDIA?
 Multimedia can have many definitions these include: (A computer
system perspective)
 Multimedia means that computer information can be
represented through audio, video, and animation in addition to
traditional media (i.e., text, graphics, drawings, images, video,
audio).
 Multi + Media
 Multi : many; much; multiple
 Medium : An intervening substance through which something is
transmitted or carried on; A means of mass communication such
3
as a newspaper, magazine or TV
 CONT…
Franklin Kuo:
 “Multimedia concerns the representation of mixed modes of
information – text, data, image, audio and video – as digital
signals”
 “Multimedia Communications concerns the technology
required to manipulate, transmit, and control these audiovisual
signals across a communications channel”
Guojun Lu:
 “A system capable of handling at least one type of continuous
media in digital form as well as static media.” 4
MULTIMEDIA APPLICATION DEFINITION

 A Multimedia Application is an application which uses a

collection of multiple media sources e.g. text, graphics, images,
sound/audio, animation and/or video.

5
 MEDIA TYPES
Two broad classes:

 Static, time-independent discrete media: Text, graphics,

images. Information in these media consist exclusively of a
sequence of individual elements without a time component.

 Dynamic, time-dependent continuous media: Sound, video.

Information is expressed as not only of its individual value, but
also by the time of its occurrence.

6
 WHAT IS HYPER TEXT / HYPER MEDIA?

 Hypertext is a text which contains links to other texts. The term

was invented by Ted Nelson around 1965. Traversal through
pages of hypertext is therefore usually non-linear.

 HyperMedia is not constrained to be text based. It can include

other media, e.g., graphics, images, and especially the
continuous media – sound and video

7
 MULTIMEDIA APPLICATIONS
 World Wide Web

 Multimedia Authoring, e.g. Adobe/Macromedia

Director

 Video-on-demand

 Computer Games

 Virtual reality

 Digital video editing and production systems

 Multimedia Database systems

8
 CONTENTS (CONT.)

 Data Compression and Decompression

 Lossless vs lossy compression

 1-D compression

 2-D compression

 JPEG standard

 MPEG coding methodology

9
 Human behavior – reacts according to external stimuli. Five senses of human –
vision, hearing, touch, taste and smell

 A medium refers to different type of data representation such as text, images,

graphics, speech, audio, video.

 A user manipulates multimedia data with multimedia systems.

 Multimedia communication system enables the integration of more than one of

the media for the purpose of transmission, storage, access and content creation.

 Interestingly, with this definition, reading newspaper is a multimedia

experience since it integrates text and halftone images.

 TV is another example that integrates video and audio signals.

10
THE NEED FOR MULTIMODAL COMMUNICATION

 Human communication is multimodal in

nature.

 Human speech is bimodal in nature

ba ga da
 Others; hand gesture, facial expression

 McGurk effect
pa ga ta
 When humans are presented with

conflicting audio and video stimuli, the

perceived sound may not exist in either Example

modality due to the dominance of

visual sense.
11
 CONT…

 Reverse McGurk occurs when the sound produced by an object

influenced its visual interpretation.

 Visit the epsych site of the Mississippi State University

 http://epsych.msstate.edu/descriptive/crossModal/mcgurk/mcg
urk_desc.html

12
ANALOG SYSTEMS VS DIGITAL SYSTEMS
 Naturally audio and video signals are analog continuous signals; they vary
continuously in time as the amplitude, pressure, frequency or phase of the
speech, audio, or video signal varies.

 In general, an analog multimedia system is relatively simple with less building

blocks and fast response when comparing to a digital system.

 An analog system is more sensitive to noise and less robust than its digital
counterpart.

 An analog media is more difficult in editing, indexing, search and transmit

among systems.

 Analog signals must be digitized by codec before computer manipulation.

13
DIGITAL REPRESENTATION OF MULTIMEDIA CONTENTS

 Text or Data

 Text or data is a block of characters with each character represented by a fixed

number of binary digits knows as a code word; ASCII code, Unicode, etc. The
size of text or data files are usually small.

 Graphics

 Graphics can be constructed by the composition of primitive objects such as

lines, circles, polygons, curves and splines.

 Each object is stored as an equation and contains a number of parameters or

attributes such as start and end coordinates; shape; size; fill color; border color;
shadow; etc.

 Graphics file take up less space than image files. 14

 For data visualization, illustration and modeling applications

 CONT…

 Images
 Continuous-tone images can be digitized into bitmaps
 Resolution for print is measured in dots per inch (dpi)
 Resolution for screen is measured in pixels per inch (ppi)
 A digitized image can be represented as two-dimensional matrix
with individual picture elements.
 Each pixel is represented by a fixed number of bits or N-bit
quantization.
 24-bit quantization provides 16,777,216 colours per pixel.
 Dots are digitized in 1-bit quantization 15
BITMAP GRAPHICS & VECTOR GRAPHICS

Bitmap Graphics Vector Graphics

Display Speed X

Image Quality X

Memory Usage X

Ease of editing X

Display X
Independence

Different method serve different purpose.

Neither method is better than the other but depending on
16
the application
DIGITAL REPRESENTATION OF SPEECH / AUDIO SIGNAL

 1-dimensional but need high dynamic range - large file size

 For telephone lines, the channel bandwidth is 200Hz-

3.4kHz for a sampling of 8-bit per channel.

 The sampling rate is 8kHz for slightly oversampling.

 Nyquist sampling rate = (3.4-0.2)x2 kHz = 6.8kHz.

17
SPEECH/ AUDIO ENCODER

18
 SPEECH COMPRESSION

 The digitization process is known as pulse code modulation or PCM.

Typical bit rate is 64kbps - (8 bit x 8kHz) per channel.

 This involves sampling the (analog) audio signal/waveform at a minimum

rate which is twice that of the maximum frequency component(4kHz) that

makes up the signal. – Nyquist Criteria.

 The sampling rate is limited to conserve the bandwidth of the

communications channel.

 A compression algorithm is normally used to achieve comparable

perceptual quality (as perceived by the ear) to conserve bandwidth.

19
 DIGITAL SPEECH / AUDIO REPRESENTATION
 The ear is more sensitive to noise on quite signals than it is on loud signals.
In a PCM system the quantization intervals are made non-linear with
narrower intervals for low level signals. This is known as companding.

 In practice, there are 2 different compression-expansion characteristics;

 A-law for China and Europe and μ-law for Hong Kong and North America.

 The use of companding gives a perceived level of performance with 8-bit

PCM data that is comparable with that of 12-bit uniform quantization.

 For music, the audible bandwidth is from 15Hz to 20kHz and is normally
sample at 44.1kHz. CD-Digital Audio (CD-DA) standard requires 16-bit
ADC.
20
 DATA RATE OF DIGITAL AUDIO APPLICATIONS
Type Approx. Sampling Bits per Uncompressed
Freq. Rate (kHz) Sample Bit Rate
(Hz)

Telephone 200-3200 8 8 64kbps

Speech
Wide Band 50-10000 16 16 256kbps
Speech
CD Audio 15-20000 44.1 16x2 1.41Mbps

21
 DIGITAL VIDEO
 Composed of a series of still image frames and produces the illusion of
movement by quickly displaying one frame after another.

 The Human Visual System (HVS) accepts anything more than 20 frames per
second (fps) as smooth motion.

 The challenges are the massive volume of data involved and the needs to meet
real time constraints on retrieval, delivery and display.

 A digitized video comprises both images and audio and need to be synchronize
in time.

 The solution is to reduce the image size, use high compression ratios; eliminates
spatial(3D and 4D) and colour similarities of individual images and temporal
22
(Time-Based or sequential) redundancies between adjacent video frames
PICTURE FORMATS SUPPORTED BY H.261 & H.263

Parameter Sub- QCIF CIF 4CIF 16CIF

QCIF

Number of pixels per line 128 176 352 704 1,408

Number of lines 96 144 288 576 1,152

Uncompressed bit 4.4 9.1 37 146 584
rates(Mb/s)

23