Multimedia Systems

CS-502 Operating Systems

Fall 2007
(Slides include materials from Operating System Concepts, 7th ed., by Silbershatz, Galvin, & Gagne and
from Modern Operating Systems, 2nd ed., by Tanenbaum)

CS-502 Fall 2007

Multimedia Systems

Outline
What is multi-media?
Requirements and challenges for audio and video in
computer systems
CD devices and formats

Systems for multimedia

Compression and bandwidth
If time:
Processor and disk scheduling
Network streaming and management

Silbershatz, Chapter 20
Tanenbaum, Chapter 7
CS-502 Fall 2007

Multimedia Systems

What do we mean by multimedia

Audio and video within a computer system
CDs & DVDs
Computer hard drive
Interactive

Live broadcast & web casts

Radio stations, Network TV, Webcams, Skype,

Video on demand
Pause, fast forward, reverse, etc.

Interactive activities involving audio, video, images

Meetings and presentations with 2-way audio
Teleconferencing

Handheld devices
iPod, MP-3 players; personal video; mobile phones,

CS-502 Fall 2007

Multimedia Systems

Multimedia Data and Delivery

Stored in file system like ordinary data.
Must be accessed with specific timing
requirements.
E.g., video must be displayed at 24-30
frames per second.
System must guarantee timely delivery

Continuous-media data
Data with specific rate requirements
CS-502 Fall 2007

Multimedia Systems

Multimedia Characteristics
Multimedia files can be very large.
Continuous media data may require very high data
rates.
Multimedia applications are usually sensitive to
timing delays during playback.

Note: human ear is more sensitive to jitter in audio

than eye is to jitter in video!
CS-502 Fall 2007

Multimedia Systems

Requirements
Smooth audio and video
Deterioration in quality jerky playback

Multiple concurrent streams

Video & multimedia servers
TiVo, etc.

Wide range of network bandwidths

Audio/video while PC is doing something
else
CS-502 Fall 2007

Multimedia Systems

System and OS Challenges

Bandwidths and Compression
Jitter

Processor Scheduling
Disk Scheduling
Network Streaming
CS-502 Fall 2007

Multimedia Systems

Some System Architectures

Simple:
Data paths for audio/video that are separate
from computational data paths

Modern
Fast system bus, CPU, devices

Video server
Disk farm and multiple streams
CS-502 Fall 2007

Multimedia Systems

System Organization (a decade ago)

Memory

CPU

memory bus
audio stream

Device

CD-ROM
drive

Sound
card

Separate data path for audio stream

Or headphone jack and volume control on CD drive itself

Main system bus and CPU were too busy/slow to

handle real-time audio
CS-502 Fall 2007

Multimedia Systems

System Organization (typical Pentium PC today)

video stream via
ISA & bridge to
graphics card

Main
Memory

AGP Port

Level
2
cache

CPU

Bridge

PCI bus

Ethernet

SCSI

ISA
bridge

IDE
disk

Sound
card

Printer

ISA bus
Keyboard

Modem
Multimedia Systems

Monitor

audio stream via

ISA bridge to
sound card

USB

Mouse
CS-502 Fall 2007

Graphics
card

10

Digression
CD and DVD devices and files

CS-502 Fall 2007

Multimedia Systems

11

Compact Discs (CDs)

Not in Silbershatz
See Tanenbaum, pp. 306-310

Audio CD
Molded polycarbonate; easy to manufacture
120 mm diameter with 15 mm hole
One single spiral track
Starts in center, spirals outward
22,188 revolutions, approx 5.6 kilometers long
Random access

Constant linear velocity under read head

Audio playback: 120 cm/sec
Variable speed motor: 200 530 rpm

ISO standard IS 10149, aka the Red Book

Audio CDs
CS-502 Fall 2007

Multimedia Systems

12

CDs for data storage

Problem for adapting to data usage
No bad block recovery capability!

ISO standard for data: Yellow Book

Three levels of error-correcting schemes: Symbol, Frame, Sector
~7200 bytes to record 2048 byte payload per sector

Mode 2: less error correction in exchange for more data rate

Audio and video data

Sectors linearly numbered from center to edge

Read speed
1x ~ 153,000 bytes/sec
40x ~ 5.9 megabytes/sec

ISO standard for multi-media: Green Book

Interleaved audio, video, data in same sector
CS-502 Fall 2007

Multimedia Systems

13

CD-ROM File System

ISO 9660 High Sierra
Write once contiguous file allocation
Variable length directories
Variable length directory entries
Points to first sector of file
File size and metadata stored in directory entry
Variable length names

Linux, Windows support iso9660 file system

Several extensions to standard for additional
features
E.g., adding files, replacing directories

See Tanenbaum, pp. 310-313

CS-502 Fall 2007

Multimedia Systems

14

DVD devices
Similar in concept, different in details
More data, higher bandwidth
Track spacing:
Bit spacing:
Capacity:

0.834
vs.
0.4
vs.
4.7 Gbytes vs.

1.6
0.74
650 Mbytes

Details not provided here

Tanenbaum, pp. 313-315
CS-502 Fall 2007

Multimedia Systems

15

Compression
An essential part of audio and video
representation as files

CS-502 Fall 2007

Multimedia Systems

16

Why Compression? CD-quality audio

22,050 Hz 44,100 samples/sec
16 bits per sample

Two channels 176,400 bytes/sec

1.4 mbits/sec
Okay for a modern PC
Not okay for 56 kb/sec modem (speed) or iPod (space)!

MP-3 0.14 mbits/sec (10:1)

Same audio quality!
Compression ratio varies with type of music
CS-502 Fall 2007

Multimedia Systems

17

Why Compression? Video

Standard TV frame = 640 480 pixels @ 25-30
frames/sec (fps)
9,216,000 pixels/sec = 27,648,000 bytes/sec

Hollywood standard movie 133 minutes

approx. 210 gigabytes (standard resolution)!

HDTV = 1280 720 pixels @ 30 fps

82,944,000 bytes/sec (and rising!)

DVD holds ~ 4.7 gigabytes

average of ~ 620 kilobytes/sec!

Standard movie of 133 minutes requires serious

compression just to fit onto DVD
CS-502 Fall 2007

Multimedia Systems

18

Video Compression Requirements

Compression ratio > 50:1
i.e., 210 gigabytes:4.7 gigabytes

Visually indistinguishable from original

Even when paused

Fast, cheap decoder

Slow encoder is okay

VCR controls
Pause, fast forward, reverse
CS-502 Fall 2007

Multimedia Systems

19

Video Compression Standards

MPEG (Motion Picture Experts Group)
Based on JPEG (Joint Photographic Experts Group)
Multi-layer
Layer 1 = system and timing information
Layer 2 = video stream
Layer 3 = audio and text streams

Three standards
MPEG-1 352240 frames; < 1.5 mb/sec ( < VHS quality)
Layer 3 = MP3 Audio standard
Typical uses: video clips on internet; video for handhelds

MPEG-2 standard TV & HDTV; 1.5-15 mb/sec

DVD encoding

MPEG-4 combined audio, video, interactive graphics

2D & 3D animations
CS-502 Fall 2007

Multimedia Systems

20

JPEG compression (single frame)

1. Convert RGB into YIQ

Y = luminance (i.e., brightness) ~ black-white TV

I, Q = chrominance (similar to saturation and hue)

Reason: Human eye is more sensitive to luminance

than to color (rods vs. cones)

2. Down-sample I, Q channels

i.e., average over 22 pixels to reduce resolution

lossy compression, but barely noticeable to eye

3. Partition each channel into 88 blocks

CS-502 Fall 2007

4800 Y blocks, 1200 each I & Q blocks (for 640 480)

Multimedia Systems

21

JPEG (continued)

CS-502 Fall 2007

Multimedia Systems

22

JPEG (continued)

CS-502 Fall 2007

Multimedia Systems

23

JPEG (continued)
4. Calculate Discrete Cosine Transform
(DCT) of each 88 block

What is a Discrete Cosine Transform?

5. Divide 88 block of DCT values by 88

quantization table

Effectively throwing away higher frequencies

6. Linearize 88 block, run-length encode,

and apply a Huffman code to reduce to a
small fraction of original size (in bytes)
CS-502 Fall 2007

Multimedia Systems

24

JPEG (concluded)
7. Store or transmit 88 quantization table
followed by list of compressed blocks

Achieves 20:1 compression with good visual

characteristics

JPEG algorithm executed backwards to recover

image

Higher compression ratios possible with visible degradation

Visually indistinguishable from original @ 20:1

JPEG algorithm is symmetric

CS-502 Fall 2007

Same speed forwards and backwards

Multimedia Systems

25

MPEG
JPEG-like encoding of each frame
Takes advantage of temporal locality
I.e., each frame usually shares similarities
with previous frame
encode and transmit only differences

Sometimes an object moves relative to

background
find object in previous frame, calculate
difference, apply motion vector
CS-502 Fall 2007

Multimedia Systems

26

Temporal Locality (example)

Consecutive Video Frames

CS-502 Fall 2007

Multimedia Systems

27

MPEG organization
Three types of frames
I-frame: Intracoded or Independent.
Full JPEG-encoded frame
Occurs at intervals of a second or so
Also at start of every scene

P-frame: Predictive frame

Difference from previous frame

B-frame: Bidirectional frame

Like p-frame but difference from both previous and next frame

I BBBPBBBPBBBPBBB I BBBPBBBP
CS-502 Fall 2007

Multimedia Systems

28

MPEG Characteristics
Non-uniform data rate!
Compression ratios of 50:1 80:1 are readily
obtainable
Asymmetric algorithm
Fast decode (like JPEG)
Encode requires image search and analysis to get high
quality differences

Cheap decoding chips available for

graphics cards
DVD players, etc.
CS-502 Fall 2007

Multimedia Systems

29

MPEG Problem Fast Forward/Reverse

Cannot simply skip frames
Next desired frame might be B or P derived
from a skipped frame

Options:
Separate fast forward and fast reverse files
MPEG encoding of every nth frame
Often used in video-on-demand server

Display only I and P frames

If B frame is needed, derive from nearest I or P
CS-502 Fall 2007

Multimedia Systems

30

Movie File Organization

One MPEG-2 video stream
Possible fast forward, fast reverse sub-streams

Multiple audio streams

Multiple languages

Multiple text streams

Subtitles in multiple languages

All interleaved
Possibly in multiple files!
CS-502 Fall 2007

Multimedia Systems

31

CS-502 Fall 2007

Multimedia Systems

32

Questions?
(skip to reading assignment)

CS-502 Fall 2007

Multimedia Systems

33

Operating System Challenge

How to get the contents of a movie file from disk
or DVD drive to video screen and speakers.
Fixed frame rate (25 or 30 fps)
Steady audio rate
Bounded jitter

Such requirements are known as Quality-ofService (QoS) guarantees.

Classical problem in real-time scheduling
Obscure niche become mainstream!
See Silbershatz, 19.119.5
CS-502 Fall 2007

Multimedia Systems

34

QoS Guarantees
Building a system to guarantee QoS effects
the following:

CPU processing
Scheduling and interrupt handling
File systems
Network protocols

CS-502 Fall 2007

Multimedia Systems

35

Requirement of Multimedia Operating

Systems
There are three levels of QoS
Best-effort service: the system makes a best
effort with no attempt to guarantee
Soft QoS: allows different traffic streams to be
prioritized, but no QoS guarantees are made
Hard QoS: system ensures QoS requirements
are always met
Prioritization
Admission control
Bounded latency on interrupt handling, processing,
etc.
CS-502 Fall 2007

Multimedia Systems

36

Parameters Defining QoS

Throughput: the total amount of work
completed during a specific time interval
Delay: the elapsed time from when request
is first submitted to desired result
Jitter: delays that occur during playback of
a stream.
Reliability: how errors are handled during
transmission and processing of continuous
media
CS-502 Fall 2007

Multimedia Systems

37

Further QoS Issues

QoS may be negotiated between the client
and server.
Operating systems may use an admission
control algorithm
Admits a request for service only if the server
has sufficient resources to satisfy the request

CS-502 Fall 2007

Multimedia Systems

38

Two common methods

Rate Monotonic Scheduling
Earliest Deadline First

Many variations
Many analytic methods for proving QoS
(Quality of Service)

CS-502 Fall 2007

Multimedia Systems

39

Processor Scheduling for Real-Time

Rate Monotonic Scheduling (RMS)
Assume m periodic processes
Process i requires ti msec of processing time
every pi msec.
Equal processing every interval like
clockwork!

CS-502 Fall 2007

Multimedia Systems

40

Example
Periodic process i requires the CPU at specified intervals
(periods)
pi is the duration of the period
ti is the processing time
di is the deadline by when the process must be serviced
Often same as end of period

CS-502 Fall 2007

Multimedia Systems

41

Processor Scheduling for Real-Time

Rate Monotonic Scheduling (RMS)
Assume m periodic processes
Process i requires ti msec of processing time every pi
msec.
Equal processing every interval like clockwork!

Assume

i 1

ti
1
pi

Assign priority of process i to be

Statically assigned

1
pi

Let priority of non-real-time processes be 0

CS-502 Fall 2007

Multimedia Systems

42

Rate Monotonic Scheduling (continued)

Scheduler simply runs highest priority
process that is ready
May pre-empt other real-time processes
Real-time processes become ready in time for
each frame or sound interval
Non-real-time processes run only when no realtime process needs CPU

CS-502 Fall 2007

Multimedia Systems

43

Example
p1 = 50 msec; t1 = 20 msec
p2 = 100 msec; t2 = 35 msec
Priority(p1) > Priority(p2)
Total compute load is 75 msec per every 100 msec.

Both tasks complete within every period

25 msec per 100 msec to spare

CS-502 Fall 2007

Multimedia Systems

44

Example 2
p1 = 50 msec; t1 = 25 msec
p2 = 80 msec; t2 = 35 msec
Priority(p1) > Priority(p2)
Total compute load is ~ 94% of CPU.

Cannot complete both tasks within some periods

Even though there is still CPU capacity to spare!

CS-502 Fall 2007

Multimedia Systems

45

Rate Monotonic Theorems (without proof)

Theorem 1: using these priorities, scheduler can
guarantee the needed Quality of Service (QoS),
provided that
m

1
ti
m(2 m 1)

i 1 pi
Asymptotically approaches ln 2 as m

ln 2 = 0.6931

Theorem 2: If a set of processes can be scheduled

by any method of static priorities, then it can be
scheduled by Rate Monotonic method.
CS-502 Fall 2007

Multimedia Systems

46

Example 2 again
1
t1 t 2 25 35

0.9375 2 2 2 1 0.828

p1 p2 50 80

Note that p1 pre-empts p2 in second interval, even

though p2 has the earlier deadline!

CS-502 Fall 2007

Multimedia Systems

47

More on Rate Monotonic Scheduling

Rate Monotonic assumes periodic processes
MPEG-2 playback is not a periodic process!

CS-502 Fall 2007

Multimedia Systems

48

Processor Scheduling for Real-Time

Earliest Deadline First (EDF)

When each process i become ready, it

announces deadline Di for its next task.
Scheduler always assigns processor to
process with earliest deadline.
May pre-empt other real-time processes

CS-502 Fall 2007

Multimedia Systems

49

Earliest Deadline First Scheduling (continued)

No assumption of periodicity
No assumption of uniform processing times
Theorem: If any scheduling policy can satisfy
QoS requirement for a sequence of real time tasks,
then EDF can also satisfy it.
Proof: If i scheduled before i+1, but Di+1 < Di , then i
and i+1 can be interchanged without affecting QoS
guarantee to either one.
CS-502 Fall 2007

Multimedia Systems

50

Earliest Deadline First Scheduling (continued)

EDF is more complex scheduling algorithm
Priorities are dynamically calculated
Processes must know deadlines for tasks

EDF can make higher use of processor than

RMS
Up to 100%

There is a large body of knowledge and

theorems about EDF analysis
CS-502 Fall 2007

Multimedia Systems

51

Example 2 (again)
Priorities are assigned according to deadlines:
the earlier the deadline, the higher the priority;
the later the deadline, the lower the priority.

CS-502 Fall 2007

Multimedia Systems

52

Reading Assignments
Silbershatz, 19.119.5
Real-time systems & Real-time scheduling

Silbershat, Chapter 20
Multimedia systems

CS-502 Fall 2007

Multimedia Systems

53

Questions?

CS-502 Fall 2007

Multimedia Systems

54

Network Management
General methods for delivering content
from a server to a client across a network:
Broadcasting: server delivers content to all
clients, whether they want it or not.
Multicasting: server delivers content to group
of clients who indicate they wish to receive the
content.
Unicasting: server delivers content to a single
client.
CS-502 Fall 2007

Multimedia Systems

55

Network Management (continued)

Broadcasting
Like cable TV
Fixed portion of network bandwidth dedicated to set of
broadcast streams; inflexible

Multicasting
Typical webcast
Multicast tree set up through internet to reach customers
Customers can come and go

Unicast
Dedicated connection between server and client
Streaming version of familiar transport protocols
CS-502 Fall 2007

Multimedia Systems

56

Streaming
Delivery of Multimedia Data over Network
Two types of streaming:
Progressive download: client begins playback
of multimedia file during delivery.
File is ultimately stored on client computer
(Hopefully) download speed > playback speed

Real-time streaming: multimedia file is

delivered to, but not stored on, client computer
Played back at same speed as delivery
Limited amount of buffering to remove jitter
CS-502 Fall 2007

Multimedia Systems

57

Two Types of Real-time Streaming

Live streaming: to deliver a live event while
it is occurring.
Broadcast or multicast

On-demand streaming: to deliver archived

media streams

CS-502 Fall 2007

Movies, lectures, old TV shows, etc.

Events not delivered when they occur.
Playback with pause, fast forward, reverse, etc
Unicast (usually)
Multimedia Systems

58

Network Streaming
Traditional HTTP
Stateless
Server responds to each request independently

Real-Time Streaming Protocol (RTSP)

Client initiates a push request for stream
Server provides media stream at frame rate

CS-502 Fall 2007

Multimedia Systems

59

Pull (HTTP) vs. Push (RTSP) server

CS-502 Fall 2007

Multimedia Systems

60

RTSP States
SETUP: server allocates resources for client
session.
PLAY: server delivers stream to a client
session.
PAUSE: server suspends delivery of a
stream.
TEARDOWN: server releases resources and
breaks down connection.
CS-502 Fall 2007

Multimedia Systems

61

RTSP state machine

CS-502 Fall 2007

Multimedia Systems

62

Bandwidth Negotiation
Client application provides feedback to
server to adjust bandwidth
E.g.,
Windows Media Player
RealPlayer
Quicktime

CS-502 Fall 2007

Multimedia Systems

63

Video Server
Multiple CPUs
Disk farm
1000s of disks

Multiple high-bandwidth network links

Cable TV
Video on demand
Internet

CS-502 Fall 2007

Multimedia Systems

64

Multimedia File & Disk Management

Single movie or multimedia file on disk
Interleave audio, video, etc.
So temporally equivalent blocks are near each other

Attempt contiguous allocation

Avoid seeks within a frame

Audio
Frame
CS-502 Fall 2007

Text
Frame
Multimedia Systems

65

File organization Frame vs. Block

Frame organization

Small disk blocks (4-16 Kbytes)

Frame index entries point to starting block for each frame
Frames vary in size (MPEG)
Advantage: very little storage fragmentation
Disadvantage: large frame table in RAM

Block organization

CS-502 Fall 2007

Large disk block (256 Kbytes or more)

Block index entries point to first I-frame of a sequence
Multiple frames per block
Advantage: much smaller block table in RAM
Disadvantage: large storage fragmentation on disk
Multimedia Systems

66

Frame vs. Block organization

larger

smaller

CS-502 Fall 2007

Multimedia Systems

67

File Placement on Server

Random
Striped
Organ pipe allocation

Most popular video in center of disk

Next most popular on either side of it, etc.
Least popular at edges of disk
Minimizes seek distance

CS-502 Fall 2007

Multimedia Systems

68

Resources on a file server

CS-502 Fall 2007

Multimedia Systems

69

Conclusion
Multimedia computing is challenging
Possible with modern computers
Compression is essential, especially for video

Real-time computing techniques move into

mainstream
Processor and disk scheduling

There is much more to this subject than fits

into one class
CS-502 Fall 2007

Multimedia Systems

70

Reading Assignments
Silbershatz, 19.119.5
Real-time systems & Real-time scheduling

Silbershatz, Chapter 20
Multimedia systems

CS-502 Fall 2007

Multimedia Systems

71

CS-502 Fall 2007

Multimedia Systems

72

Digression on Transforms
Fouriers Theorem
Every continuous, periodic function can be
reduced to the sum of a series of sine waves
The Fourier transform is a representation of that
function in terms of the frequencies of those sine
waves
Original function can be recovered from its
Fourier transform
Fourier transforms occur frequently in nature!
CS-502 Fall 2007

Multimedia Systems

73

Nyquists Theorem (1924)

If a continuous function is sampled at a frequency
2f, then the function can be recovered from those
samples provided that its maximum Fourier
frequency is f.

CS-502 Fall 2007

Multimedia Systems

74

Discrete Cosine Transform

A form of the Fourier Transform
When applied to an 88 block of samples
(i.e. pixel values) yields an 88 block of
spatial frequencies
Original 88 block of samples can be
recovered from its DCT.
Assuming infinite arithmetic precision

CS-502 Fall 2007

Multimedia Systems

75

More on Nyquist
If arithmetic precision is not infinite, we get
quantization error during sampling
Recovered signal has quantization noise
i.e., a lossy transform

CS-502 Fall 2007

Multimedia Systems

76