08/28/2013

INTRODUCTION
CEN 3721 Human-Computer Interaction
P. McDermott-Wells

Source: MIND: Introduction to Cognitive Science 2nd edition, by Paul Thagard,

ISBN 0-262-70109-X

CEN 3721

Human-Computer Interaction
What is HCI?
Study of interaction between people and computers
The intersection of computer science, behavioral science, design,
ergonomics, and other fields of study
Goal of HCI: Improve the interactions between users and

computers
Design systems that minimize barriers between the human

cognitive model of what they want to accomplish, and the

computers understanding of the users task

CEN 3721

Human-Computer Interaction (contd.)

What do we need to study for successful HCI?
Cognitive science: how the mind works; how do we learn; how
much can we easily remember; what is pleasant/unpleasant; what
motivates/demotivates people; perceptions
Ergonomics of sight, hearing, and motion: how do we look at
screens, how do we interact with them
Physics of sight, hearing, motion, color
Cultural differences: shapes, colors, and images with cultural
meanings
Demographic differences: age, gender, physical
abilities/disabilities
Technology: input/output modalities and how that affects design

08/28/2013

CEN 3721

Human-Machine Interaction
Human-Machine Interaction: Study of human factors

looks at the design of objects that humans use or interact

with
Engineering mantra in the past: Form-Fit-Function
Form: shape, size, dimensions, mass, or visual parameters which

uniquely characterize an item

Fit: the ability to physically interface or interconnect with or become

an integral part of another item or assembly

Function: the action(s) that it is designed to perform

Is it still used today?

Examples of poor design are everywhere!
http://www.baddesigns.com/

CEN 3721

Cognitive Science: Studying the Mind

Why study the mind?
Computers can be made more intelligent by reflecting on what
makes people intelligent
How does the mind work?
Cognitive Science: the science that explains how people

accomplish various kinds of thinking

Knowledge in the mind consists of mental

representations
Mental procedures operate on mental representations to produce

thought and action

Different kinds of mental representations foster different kinds of

mental procedures

CEN 3721

Cognitive Science
Central hypothesis of cognitive science: Thinking can

best be understood in terms of representational structures

in the mind and computational procedures that operate on
those structures
CRUM: Computational-Representational Understanding
of the Mind

Programs

Mind

Data structures + algorithms Mental representations +

= running programs
computational procedures =
thinking

08/28/2013

CEN 3721

Mental Representations
Various kinds of mental representations have been

proposed, including:
Rules
Concepts
Images
Analogies

Explanatory pattern:
People have mental representations
People have algorithmic processes that operate on those
representations
The processes, applied to the representations, produce the
behavior

CEN 3721

Approaches to Modeling the Mind

6 main approaches to modeling the mind:
Logic
Rules
Concepts
Analogies
Images
Neural connections
These approaches can be evaluated according to 5

criteria:
Representational power
Computational power
Psychological plausibility
Neurological plausibility
Practical applicability

CEN 3721

Approaches to Modeling the Mind

(contd.)
Computational power includes:
Problem solving: how people reason to accomplish their goals; 3
kinds:
Planning
Decision
Explanation

Learning: how people learn; learning from experience

Language
Ability to comprehend language
Ability to produce utterances
Childrens universal ability to learn language

08/28/2013

CEN 3721

10

Modeling the Mind

6 main approaches to modeling the mind:

Logic
Rules
Concepts
Analogies
Images
Neural connections

CEN 3721

11

Logic
Why study logic?
Many basic ideas about representation and computation have
come from the study of logic
Logic is considered to be central to work on reasoning
Logic has substantial representational power
Formal logic began with the Greek philosopher Aristotle

more than 2000 years ago

He studied patterns of inference called syllogisms

Example:

All students are overworked.

Mary is a student
Therefore, Mary is overworked.

CEN 3721

12

Logic (contd.)
Early theory of computation was developed by logicians

such as Alonzo Church and Alan Turing

In the 1950s, when the study of artificial intelligence

began, many (but not all) researchers took logic to be the

most appropriate tool
Still disagreement about the mutual relevance of logic and
psychology

08/28/2013

CEN 3721

13

Logic (contd.)
Explanatory pattern:
People have mental representations similar to sentences in
predicate logic
People have deductive and inductive procedures that operate on
those sentences
The deductive and inductive procedures, applied to the sentences,
produce the inferences

CEN 3721

14

Watsons Selection Task

You will be shown 4 cards, each of which has a number

on one side and a letter on the other side

You are given this rule to be tested:
If a card has an A on one side, then it has a 4 on the other side
Which cards must be turned over to determine whether

this rule holds?

CEN 3721

15

Watsons Selection Task (contd.)

Reasoning:
Must turn over the A to check whether there is a 4 on the reverse
Modus ponens: If A then 4

Must check the 7: if there is an A on the reverse, it refutes the rule

in question
Modus tollens: If A then 4; 7 means not-4; so not-A is required for the

rule to hold
Turning over the B is irrelevant to the rule
Turning over the 4 is irrelevant to the rule

08/28/2013

CEN 3721

16

Watsons Selection Task (contd.)

People have less difficulty with this problem if given

concrete examples
Suppose the cards have information about whether

individuals are in a bar on one side, and numbers on the

other side indicating their age
Rule to be tested: If a person is in the bar, then he/she is
21 years of age or older

CEN 3721

17

Modeling the Mind

6 main approaches to modeling the mind:
Logic

Rules
Concepts
Analogies
Images
Neural connections

CEN 3721

18

Rules
Rules are if-then structures
Similar to logic statements, but different representational and
computational properties
Can represent general information about the world
Ex. Students are overworked. If x is a student, then x is
overworked.
Can represent information about how to do things in the

world
Ex. If you register early, then you will get the courses you want

Can represent linguistic regularities

Ex. If a sentence has a plural subject, then it has a plural verb

08/28/2013

CEN 3721

19

Rules (contd.)
Rules of inference (such as modus ponens) can be recast

in rule form
Ex. If you have an if-then rule, and the if part is true, then the then

part will be true also

Rules are not as representationally elegant as formal logic

Losing some of the rigor of logic-based systems has provided
increased computational power
Rules often contain actions that represent goals
Ex. If you want to go home and you have bus fare, then you can
take a bus home
These types of rules serve to focus the problem solver on the task
at hand

CEN 3721

20

Rules (contd.)
Problem solving with rules: Cryptarithmetic problems -

puzzles with letters substituted for numbers

DONALD
+GE RALD
---------------ROBERT

How do you begin to solve this puzzle?

Col. 2: O + E = O. If 0 is added to a number, then the number is
unchanged. Therefore, E must be 0.
Col. 4: A + A = E. Since E is 0, then A should be 5. But this
assumes there was no carry-over involved in adding L + L in col. 5.
If there was carry-over from col. 3, then E may not be 0, but 9
Assuming that you do not have a leading 0 is also helpful

CEN 3721

21

Rules (contd.)
Explanatory pattern:
People have mental rules
People have procedures for using these rules to search a space of
possible solutions, and procedures for generating new rules
Procedures for using and forming rules produce the behavior
Rule-based computational systems are widely used in

business today, often called expert systems

Financial services: credit applications, mortgage applications,

process control
Recommender systems (based on interactive questions asked of

the user)
Medical diagnosis systems
Require a subject matter expert to build the rules

08/28/2013

CEN 3721

22

Modeling the Mind

6 main approaches to modeling the mind:
Logic
Rules

Concepts
Analogies
Images
Neural connections

CEN 3721

23

Concepts
Plato studied the role of concepts in knowledge
What is justice? and What is knowledge?
He believed that concepts are purely in the mind
Locke & Hume believed that concepts are learned

through sensory experience

What are the processes by which concepts are learned

from experience and from other concepts?

New terminology came into use in the mid 1970s: frame,

schema, and script to describe new views of the nature

of concepts

CEN 3721

24

Concepts (contd.)
Minsky (1975) argued that thinking should be understood

as a frame application rather than logical deduction

Shank and Abelson (1977) showed that a great deal of

our social knowledge consists of scripts

Ex. Going to a restaurant

Rumelhart (1980) described knowledge as concept-like

structures called schemas that represent what is typical

of a concept
Recently, connectionist models have been proposed to
describe concepts

08/28/2013

CEN 3721

25

Concepts (contd.)
For computation, packaging concepts into a hierarchical

organization is powerful
Inheritance allows inferences about concepts

Association of concepts: what do you think of when you

hear the word desk?

Chairs, studying, lamps?

Loose association is a process of spreading activation

One concept in a system is active, and activation spreads in a
network to other concepts that are linked to it by kind or other
relations
Ex. Intelligence data analysis
Concepts are also used for matching and inference

CEN 3721

26

Concepts (contd.)
Concepts can be innate, or formed from experience, from

rules, or by combining other concepts we already have

Innate: basic physical concepts of objects and behavior; ex. playing

peek-a-boo with babies

Learned from example: child learns to distinguish one type of

animal from other types of animals

Try to describe a dog sufficiently so that it can be distinguished from all

other animals
If it quacks like a duck, it must be a duck defining the concept in

terms of itself

Concept application is a quick and easy way to make a

decision, but does not always account for complex

concerns about actions and goals
Hiring decisions: Do you meet the bosss concept of the right fit?

CEN 3721

27

Concepts (contd.)
In spoken and written language, concepts are

represented by words
Lexicon: a set of words in a dictionary
Grammar: a set of rules for language structure
But learning a language is more than just acquiring

grammatical rules it also involves developing a whole

conceptual system
What is the meaning of a concept, and how does it contribute to the

meaning of a sentence?
What about symbolism of concepts?
Bark up the wrong tree
Beat around the bush
Bury the hatchet

08/28/2013

CEN 3721

28

Concepts (contd.)
There is evidence that subject matter experts are simply

people who are better able to mentally organize and

retrieve massive amounts of data
Explanatory pattern:
People have a set of concepts, organized via slots that establish

kind and part hierarchies, and other associations

People have a set of procedures for concept application, including

spreading activation, matching, and inheritance

The procedures applied to the concepts produce the behavior
Concepts can be translated into rules, but they bundle information

differently than sets of rules

Concept maps help show and manage associations:

http://cmap.ihmc.us/ or The Personal Brain
http://www.thebrain.com/products/personalbrain/apps/education/

CEN 3721

29

Modeling the Mind

6 main approaches to modeling the mind:
Logic
Rules
Concepts

Analogies
Images
Neural connections

CEN 3721

30

Analogies
Analogical thinking: dealing with a new situation by

adapting a similar familiar situation

Homer user analogies in the Iliad
Bible uses many analogies

There are numerous computational models for analogical

reasoning
Also called case-based reasoning
Used in artificial intelligence

Analogy is often used to solve exercises at the end of the

chapter by flipping back and relating them to solved

problems in the main text

10

08/28/2013

CEN 3721

31

Analogies (contd.)
Analogies are not always the best way to approach a new

problem
May not have an analogy that is relevant enough

Cognitive science itself uses analogies: it attempts to

explain how the mind works by modeling it as a computer

Effective teachers often try to help students understand

the unfamiliar by systematically comparing it to the

familiar
Analogies can be a fertile source of creative designs
George de Mestral invented Velcro after observing how burrs stuck

to his dog
Reverse engineering is used to figure out how to make an

analogous product

CEN 3721

32

Analogies (contd.)
Explanatory pattern:
People have verbal and visual representations of situations that
can be used as cases or analogs
People have processes or retrieval, mapping, and adaptation that
operate on those analogs
The analogical processes, applied to the representations of
analogs, produce the behavior
The constraints of similarity, structure, and purpose overcome the
difficult problem of how previous experiences can be found and
used to help with new problems

CEN 3721

33

Modeling the Mind

6 main approaches to modeling the mind:
Logic
Rules
Concepts
Analogies

Images
Neural connections

11

08/28/2013

CEN 3721

34

Images
Mental imagery:
How do you get from your home to FIU?
Human thinking involves pictorial representations that are

different from verbal ones

Images can be non-visual:
Taste and smell sensory images: What does pizza taste like?
Tactile images: Does a growth of beard fell like sandpaper?
Motor images: How do you slam-dunk a basketball?
Emotional images: How did you feel when? Do your friends feel
the same way?

CEN 3721

35

Images (contd.)
Vision: easy for humans, but extracting information from

millions of pixels is complex for computers

Optical illusions and cognitive illusions abound
Necker cube: Is the opening at the top, or in the front?

Mark Newbold's Animated Necker Cube

Optical illusions:

http://www.echalk.co.uk/amusements/opticalillusions/illusions.htm

CEN 3721

36

Images (contd.)
The brain combines inferences about edges, perspective,

colors, and other information into a coherent interpretation

of objects
Note that images do not depend on the object being
present to the eyes
We can store images in memory, retrieve them, and manipulate

them

A picture is worth a thousand words

Pictures provide powerful ways of representing how

things look and how they are spatially arranged

Not all information is naturally represented in pictures
Abstract concepts are not visually representable, such as Justice

is fairness

12

08/28/2013

CEN 3721

37

Images (contd.)
Images are accessible to different kinds of computational

procedures than verbal representations:

Inspect
Find
Zoom
Rotate
Transform

Planning with visual representations:

First construct visual representations of the starting and goal states
Then construct a visual path from the start to the goal
Decision making can be based on imagining the various

outcomes

CEN 3721

38

Images (contd.)
Physical abilities can be improved by running the task

through your mind before performing it

Education: mathematical intuition sometimes depends on

visual and spatial representations

Many strategies for improving memory rely on visual

images
Poetry often attempts to induce mental imagery
Image-based expert systems are still fairly rare, but there

are many novel visualization tools

http://mashable.com/2007/05/15/16-awesome-data-visualization-

tools/

CEN 3721

39

Images (contd.)
Explanation pattern:
People have visual images of situations
People have processes such as scanning and rotation that operate
on those images
The processes for constructing and manipulating images produce
the intelligent behavior
Imagery can aid learning, and some metaphorical aspects of
language may have their roots in imagery

13

08/28/2013

CEN 3721

40

Modeling the Mind

6 main approaches to modeling the mind:
Logic
Rules
Concepts
Analogies
Images

Neural connections

CEN 3721

41

Connections
Physical brain contains neurons which signal each other

through contacts at specialized points called synapses

Human brain has about 100 billion neurons, many of which connect

to thousands of other neurons, forming neural networks

Computational approaches emphasize the importance of

connections
Neural networks
Parallel distributed processing

Many cognitive tasks are understood computationally in

terms of processing that simultaneously satisfies

numerous constraints
Enrolling for the semester: must juggle offerings, schedules,

requirements, pre-requisites, professor preferences

CEN 3721

42

Connections (contd.)
Connectionist networks: consist of units and links
Links can be one-way, with activation flowing from one to

another, or symmetric, with activation flowing both ways

Links can be excitatory one unit raises the activation of

another or inhibitory one unit suppresses the

activation of another
Kosslyn & Koenigs octopi network a distributed
feedforward network
Octopi on the bottom layer (input) detect fish and signal to octopi in

the middle row by squeezing their tentacles

Middle row octopi signal to those in the top row in the same manner
Top row octopi (output) throw up tentacles to inform seagulls of the

presence of fish

14

08/28/2013

CEN 3721

43

Source: http://www.oepf.org/Docs/WET_MIND__A_NEW_COGNITIVE_N.PDF

CEN 3721

44

Connections (contd.)
Neural networks: compute by spreading activation

between units that are linked to each other

Links between units have weights indicating their importance or

contribution to the process

Constraints can be satisfied in parallel by repeatedly passing

activation among all the units, until all units reach stable activation
levels

Human analogies:
Social networking sites
Videos, memes, or new buzzwords going viral

CEN 3721

45

Connections (contd.)
Learning by connectionist networks takes place by either

adding new units, or changing the weights on the links

between units
Learning in feedforward networks occurs via
backpropagation
A set of data with known results is processed
Amount of error between the processed results and known results

are compared
Weights on links are adjusted based on what is observed from the

training data in order to try to achieve the known results

Reprocess with new weights
Continue until the margin of error is within the desired threshold

15

08/28/2013

CEN 3721

46

Connections (contd.)
Backpropagation is relatively slow, requiring hundreds or

thousands of examples to train a simple neural network

People can sometime learn from very few examples

Connectionist models are widely used in intelligent

systems
Training networks to recognize bombs, underwater objects, and

handwriting
Training networks to interpret the results of medical tests and

predict the occurrence of disease

CEN 3721

47

Connections (contd.)
Explanatory pattern:
People have representations that involve simple processing units
linked to each other by excitatory and inhibitory connections
People have processes that spread activation between the units via
their connections, as well as processes for modifying the
connections
Applying spreading activation and learning to the units produces
the behavior

CEN 3721

48

Challenges for Cognitive Science

CRUM model has shortcomings, including:
It ignores the important role of emotions in human thinking
It ignores the importance of consciousness in human thinking
It neglects the contribution of the body to human thought and action
It disregards the significant role of physical environments in human
thinking
The mind is a dynamic system, not a computational system
Human thought is inherently social in ways that CRUM ignores

16