Experimentation

Nicole Ronald

CRICOS Provider: 00111D | TOID: 3059

Experimentation

Swinburne

Todays aim
- Understanding of experimental research design in
IT/IS/CS settings
- A brief refresher of research questions
- Different types of human experiments
- An overview of CS experiments

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

Experimentation

Swinburne

How do experiments differ to surveys?

- Surveys are useful when you need to understand what is
currently happening (or could happen)
- Experiments change something and then measure

(Gray distinguishes between descriptive and analytical

surveys)
SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

Swinburne

Experimentation
Process
Identify issue

Analyse

Accept or reject
hypotheses

Review
literature/theories

Conduct study

Report

Develop
hypotheses

Identify
in/dependent
values

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

Experimentation

Swinburne

Question types
Descriptive
What is happening
Normative
What is happening compared to what should happen
Correlative
What is the relationship/strength between X and Y
Impact
What impact does a change in X have on Y

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

Experimentation

Swinburne

When would experimentation be used?

- Testing algorithms, e.g., a new searching algorithm
- Testing ideas in a simulated world, e.g., giving some
entities different information
- Testing parameters, e.g., for an algorithm, in a simulation

- A lab situation is usually an experiment

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

Experimentation

Swinburne

Variables
- Dependent variable
- Outcome
- Independent variable
- Treatment

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

Experimentation

Swinburne

Experimental design
- Experimental
- Randomise allocation to groups
- Variables can be manipulated/control group
- Quasi-experimental
- Group membership cannot be randomised, or is preexisting
- Variables can be manipulated/control group
- Non-experimental
- Group membership cannot be randomised, or is preexisting
- Variables cannot be manipulated
SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

Swinburne

Experimentation
Experimental group with control

- Two groups, randomised:

- One receives treatment, the other not (control)
- Both are evaluated before and after treatment
Pre-test (t1)

Treatment (t2)

Post-test (t3)

Experimental

Yes

Yes

Yes

Control

Yes

No

Yes

- Example: time-to-completion of IT troubleshooting tickets

- Need to check that times in both groups are similar; also
control for experience, age etc.
- Experimental group receives training, control does not
SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

Swinburne

Experimentation
Four-group design

- Two groups, randomised:

- One receives treatment, the other not (control)
- Only one group measured beforehand
Pre-test (t1)

Treatment (t2)

Post-test (t3)

Experimental

Yes

Yes

Yes

Experimental

No

Yes

Yes

Control

Yes

No

Yes

Control

No

No

Yes

- Avoid influencing outcome by pretesting

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

10

Swinburne

Experimentation
Quasi-experimental with control

- Two groups:
- Could be pre-determined, by class/department
- One receives treatment, the other not (control)
- Both groups measured beforehand, hopefully equal
Pre-test (t1)

Treatment (t2)

Post-test (t3)

Experimental

Yes

Yes

Yes

Control

Yes

No

Yes

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

11

Swinburne

Experimentation
Non-experimental
- One group, observe afterwards

- Either unable to observe beforehand (e.g., unexpected

disaster) or just didnt
Experimental

Treatment (t1)

Post-test (t2)

Yes

Yes

- Example: student evaluations

- Can also use a control group

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

12

Experimentation

Swinburne

Longitudinal and cross-sectional studies

- Studies over time
- Cross-sectional: same measurement over time
- Longitudinal: same subjects and measurements over time

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

13

Experimentation

Swinburne

Validity
- Internal
- Assume only independent variable influences
dependent variable
- Threats: selection, external maturation/events, dropout,
pretesting, sharing info
- External
- Effectiveness of generalising
- Threats: people, places, time

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

14

Experimentation

Swinburne

Algorithmic experiments
- Similar principles:
- Control and experimental algorithms
- Different treatments or scenarios
- Fewer ethical issues (if not accessing personal data)

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

15

Experimentation

Swinburne

Simulation
- Representation of the world
- Can simulate many objects
- Useful when real-world experiments impractical

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

16

Experimentation

Swinburne

Experimental design in IT/IS/CS

- Base case (if using simulation then verify against real
world)
- Altered case

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

17

Experimentation

Swinburne

Replication
- Repeatable
- Get same results with same code/data
- Reproducible
- Get same results with method

- Stochastic vs. deterministic

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

18

Experimentation

Swinburne

Takeaways
1. Experiments require careful design
2. Form research questions carefully
3. Think about validity
4. Think about how your simulations/experiments relate to
real world

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

19

Experimentation

Nicole Ronald
nronald@swin.edu.au

Experimentation

Swinburne

Todays tasks
- Discuss experimental paper
- Design an experiment with human subjects
- Design an experiment using computers/equipment

SCIENCE | TECHNOLOGY | INNOVATION | BUSINESS | DESIGN

21

Experimentation

Nicole Ronald
nronald@swin.edu.au