Department of Computer & Information Science

Technical Reports (CIS)

University of Pennsylvania Year 2001

Synthesis and Acquisition of Laban Movement Analysis Qualitative Parameters for Communicative Gestures
Liwei Zhao Norman I. Badler

of Pennsylvania of Pennsylvania, badler@seas.upenn.edu This paper is posted at ScholarlyCommons.

University

University

http://repository.upenn.edu/cis reports/116

Discourse model situation knowledge encyclopaedia

Long!term memory

Others
Propositional Spatial/ Dynamic

Working memory

Conceptualizer preverbal message Formulator Grammatical encoder

Spatial/dynamic feature selector spatial/dynamic feature Motor planner motor program Motor system lexical movement Kinesic monitor

Lemma Word forms Lexicon

Phonological encoder phonetic plans Articulator overt speech

Auditory monitor

user input

speech recognition

gesture recognition

spoken language interpretation

gestural language interpretation

multimodal interface

semantic filtering temporal filtering unification

embodied conversational agent models

statistical weighting neural computing policy and planning

psychological or cognitive models

speech generation

gesture generation

user

neck clavicle shoulder

wrist elbow

spine

waist hip

knee

Shoulder Joint S ^ n

R c ^ v

!
^ e

"

^ u ^ a

^ w

Wrist Joint W

Elbow Joint E

^ ew

o x y z

y z

Y
vdy

vdz

b Z

w w y a z s Y
vdz

$ "
s

"
Y

vdz

z w y

w b Z ver<0 && 0<="<=# Z ver>0 && 0<="<=# b

w y w a
vdz

"
Y
vdz

z s

"
s

$
w

z y w

b Z ver<0 && Z ver>0 &&

#<="<2#

#<="<2#

Y X Z

Shoulder Joint

^ t ^ s
Wrist Joint

W ^ u W" ^ w Z X ^ v Y ^ c O origin
Center of Mass

Pe ^ n

^ P

c ^ u

~ P

e( )

0.9 0.8 0.7 0.6

Spontaneity

Bound!Flow Free!Flow Neutral!Flow

0.5 0.4 0.3 0.2 0.1 0

10

11

12

motion samples

Direct!Strong 0.5 0.4 curvature 0.3 0.2 0.1 0 0 100 Neutral 0.5 0.4 curvature 0.3 0.2 0.1 0 0 200 400 600 curvature 400 300 200 100 0 200 300 curvature 800 600 400 200 0

Direct!Strong

100 Neutral

200

300

200

400

600

Indirect!Light 0.5 0.4 curvature 0.3 0.2 0.1 0 0 200 400 600 frame number 800 0 0 curvature 100 150

Indirect!Light

50

200 400 600 frame number

800

Shoulder Joint S ^ n

R c ^ v

!
^ e Elbow Joint

"

^ u ^ a

^ w

Wrist Joint W

^ e w

Example 1

Example 2

Example 3

100
swivel angles

100
Direct Neutral Indirect Direct Neutral

80 60 40
Direct Neutral Indirect

80 60 40 20 60 0

50

Indirect

500
Example 4 Direct Neutral

1000

0 100

500
Example 5 Indirect

1000

20 80 60 40

100

200

300

Example 6

swivel angle

40 20 0 60

80 60
Indirect

Direct Neutral Indirect

40 300 20 80 60 0 100

Direct Neutral

20 300 0 60 0 100

100

200

200

200

300

Example 7

Example 8

Example 9 Indirect

swivel angle

40 20 0

Direct

Indirect Neutral

Direct

40 20 0 100

Indirect Neutral

40 20

Direct Neutral

100

200

300

0 50

200
Example 11 Neutral

400

100

200
Indirect

300

Example 10

Example 12

100
Indirect swivel angle Direct Neutral

40 30 20

50

Direct Indirect

50
Neutral Direct

100

200

300

10

100

200

300

100

200

300

frame number

frame number

frame number

Example 1

Indirect

100

100

80
swivel angles

Direct Neutral

50
Indirect

60

0 40

indirect normalized low freq high freq 0 200 400 600 800

20

200

400
Neutral

600

800

!50

Indirect vs Neutral

100 80
swivel angles

20 10 0 !10 !20 !30

Indirect_hfreq neutral_hfreq

60 40 20 0 !20 0 200 neutral normalized low freq high freq 400 600
frame number

50

100

150

frame number

Example 1

Example 2

Example 3

80
wrist angle

80
Direct Indirect Neutral

80 60
Direct Neutral Indirect

60 40 20 0 80 0

60 40 20

40
Indirect

20 0 200 400 600 0 100

Indirect Indirect Direct Neutral

200

400

600 100

100

200

300

Example 4

Example 5

Example 6

wrist angle

60 40 20
Direct Indirect

50
Neutral Direct

50

Neutral Direct

0 100

Neutral

100

200

300

0 100

100

200

300

0 80 60

100

200

300

Example 7

Example 8

Example 9 Neutral Indirect

Indirect wrist angle Neutral

50

Direct

50
Indirect

40 20 0 0 100

Direct

0 80
wrist angle

100

200

300

0 100

Direct

200
Example 11 Direct

400

200

300

Example 10

Example 12 Indirect

80 60
Direct Indirect Neutral

60 40 20 0
Direct Indirect Neutral

50
Neutral

40 20

100

200

300

100

200

300

100

200

300

frame number

frame number

frame number

12 Sudden!Time Strong!Weight 10

10 9 8 7
sternum height

Sustained!Time Light!Weight

8
sternum height

6 5 4 3 2 1

1 2 3 4 5 6 7 8 9 101112
motion samples

1 2 3 4 5 6 7 8 9 101112
motion samples

W 1,1 W 2,1 W I,1 X1 X2

1

h1
bias 1 bias

W 1,2 W 2,2 W I,2

1

h2
bias

XI

W 1,M W 2,M W I,M

1 Input Layer

hn
bias

bias

Sigmoid function

hidden Layer

Sigmoid function

Output Layer

200 150
wrist velocity acceleration

600 500 400 300 200 100 100 0 0 100

swivel velocity wrist zero!crossings

10 8 6 4 2 0 0 5
swivel zero!crossings

100 50 0 direct neutral indirect 0 50

velocity

200

10

0.4
wrist pendulum sum (scaled)

120 direct neutral indirect 100 80 60 40 20 0 0 50 100 150

max wrist angle

100 80 60 40 20 0 20 10
swivel zero!crossings

0.3
torsion

0.2 0.1 0

0.5

1.5

40

60

80

curvature

swivel angle sum

swivel angle

100 80
wrist max angle curvature

1.5

8 6 4 2

direct neutral indirect

60 40 20 0 0 50
swivel minmax diff

0.5

100

50
velocity

100

50
max wrist angle

100

Error versus weight updates (BPNN 0)

Error versus weight updates (BPNN 1)

0.35 0.3 0.25

Error

Training error Validation error

0.35 0.3 0.25

Error

Training error Validation error

0.2 0.15 0.1 0.05 0 0 500 1000 1500

0.2 0.15 0.1 0.05 0 0 500 1000 1500

Number of weight updates (in hundreds)

Number of weight updates (in hundreds)

Error versus weight updates (BPNN 2)

Error versus weight updates (BPNN 3)

0.35 0.3 0.25

Error

Training error Validation error

0.35 0.3 0.25

Error

Training error Validation error

0.2 0.15 0.1 0.05 0 0 100 200 300 400

0.2 0.15 0.1 0.05 0 0 500 1000 1500

Number of weight updates (in hundreds)

Number of weight updates (in hundreds)

Camera 1 Image Analysis 3D Estimation Camera 2 Image Analysis

...

Noise filtering

Feature Extraction

key structure Expressive Motion Engine (EMOTE) Style EMOTE parameters

O T1 O1 T2 O2

f1 y
(x1 ,y1 )

f2 image plane

P1
(x,y) 1 1

P 1

(x,y) 2 2

P 2

P2(x2 ,y2)

z Q1 (X1,Y1,Z1) Q2(X2,Y2,Z2)

Q 1(X,Y,Z) 1 1 1 Q 2 (X,Y,Z) 2 2 2

2D Data (Neutral Space)

3D Data (Neutral Space)

400 300
y coordinates

200 0 !200 600 150 200 250 300 350 400 400 200 200
x coordinates

200 100 0 100

400

y coordinates

x coordinates 2D Data (Direct Space)

3D Data (Direct Space)

350 300
y coordinates

200 0 !200 600 200 250 300 350 400

250 200 150 100

x coordinates 2D Data (Indirect Space)

400

200

y coordinates

200
x coordinates

400

3D Data (Indirect Space)

350 300
y coordinates

200 0 !200 600 200 250 300 350 400 400 400

250 200 150 100 150

200

y coordinates

200
x coordinates

x coordinates

2D Data (Neutral Weight)

3D Data (Neutral Weight)

400 350 200 300

y coordinates

100 0 !100 !200

250 200 150

600 100 50 100 400 200 150 200 250

x coordinates

400 200 0 0 !200

x coordinates

300

350

400

y coordinates

2D Data (Strong Weight)

3D Data (Strong Weight)

350 300 250 200 150 100 200 100

y coordinates

0 !100 !200 !300 600 400 200 0 200 250 300

x coordinates

400 200 0
x coordinates

350

400

y coordinates