MKTG 269 and MKTG 669

Special Topics: Experiments for Business Decision Making

Dr. Elea Feit
co-developed with Prof. Eric Bradlow
Logistics (Spring 2012)
Class Meetings: Monday and Wednesday 9:00-10:30, Jon M. Huntsman Hall F60
Course Website: https://wharton.instructure.com/courses/150314
Office Hours: Tuesdays 8-10pm via the chat tool on the course website and by appointment
Course Goals
In the past decade, massive shifts in how companies interact with their customers have suddenly made
field experiments an economically feasible way to learn about a variety of business questions such as
what types of promotions are most effective, what products should be stocked at a store, how e-mail
promotions should be designed, how sales staff should be compensated, etc. Many marketers engaged
in online retailing, direct-marketing, online advertising, media management, etc. are rapidly embracing a
test and learn philosophy and a number of platforms such as Google Website Optimizer, have been
developed to facilitate rigorous field experiments in the online environment. Just as with the quality
revolution in manufacturing during the 1980s and 1990s, the rapid rise of the test and learn philosophy
in marketing has created a huge demand for those who can design, field, and analyze marketing
experiments. Through this course, you will learn about, discuss and practice a wide range of critical skills
for experimentation, from the statistical methods used to design and analyze experiments to the
management and strategy required to execute an experiment and act on the results. Although our cases
and examples will focus on marketing problems, the material covered can be applied in a number of other
domains particularly operations, management and product design.
Course Approach
Creating an effective field experiment, one that truly provides value to the organization, requires equal
measures of creativity and technical skill. Consequently, this course will alternate lectures and homework
exercises on statistical methods with case discussions, readings and guest lecturers that will expose you
to broader managerial and strategy issues such as How do we decide what we should test? and How
do we minimize impact on the operations while still learning something from the test? Although the
course outline is structured around the statistical topics, I have designed the schedule so that each week
will include discussions and exercises that emphasize creativity and intuition. Ultimately, my goal is for
you to develop an understanding of the interplay between creativity and rigorous data analysis in
business decision making.
During the first half of the course, we will cover a number of experimental methods that are widely used in
practice and that most marketers should be able to execute independently. You will develop a high level
of hands-on skill with A/B testing, A/B/C/D testing, multivariate testing, optimal design, constrained
designs, and conjoint design. Towards the end of the course, we will cover several more sophisticated
methods (sequential testing, decision theory) that are typically handled in practice by experts. My goal in
exposing you to these more advanced methods is to help you develop the skills to identify when a
straightforward A/B test is appropriate versus when it might be valuable to engage an experimental
design expert to create a sequential design.
Who should take this course?
For MKTG269: This course is ideal for students with a solid background in statistics that covers
hypothesis testing, ANOVA, and multiple regression (STAT 101, STAT 431, or equivalent). I will motivate

statistical material with concrete examples from marketing, so some experience with marketing (MKTG
101), online testing or marketing research (MKTG 212) may also be helpful, but is not necessary.
Students from engineering, business, math, and science are all encouraged to enroll. If you have any
concerns about whether this course is appropriate for you, please dont hesitate to contact me.
For MKTG 669: This course is ideal for students with a solid background in statistics that covers
hypothesis testing, ANOVA, and multiple regression (STAT 621 or equivalent). I will motivate statistical
material with concrete examples from marketing, so some experience with marketing (MKTG 621/622),
online testing or marketing research (MKTG 756) may also be helpful, but is not necessary. If you have
any concerns about whether this course is appropriate for you, please dont hesitate to contact me.
Required Textbook:
Ledolter and Swersey (2007), Testing 1-2-3: Experimental Design with Applications in Marketing and
Service Operations, Stanford University Press.
Required Software:
JMP (Available at http://www.onthehub.com/jmp/)
Optional Textbooks:
Since we will be using JMP, Goupy and Creighton (2007) Introduction to Design of Experiments with
JMP Examples, SAS Press, provides more comprehensive treatment of the JMP modules than
we can cover in class.
For those who want to know more digital testing, I suggest Eisenberg and Quarto-vonTivadar (2008)
Always Be Testing: The Complete Guide to Google Website Optimizer, Sybex.
The classic textbook on Design of Experiments with a more technical approach than Ledolter and
Swersey is Montgomery (2008) Design and Analysis of Experiments, Wiley.
Readings:
Almquist, Eric and Gordon Wyner (2001) Boost Your Marketing ROI With Experimental Design,
Harvard Business Review, October.
Anderson, Eric and Duncan Simester (2011) A Step-by-Step Guide to Smart Business Experiments:,
Harvard Business Review.
Ariely, Dan (2010), Why Businesses Dont Experiment, Harvard Business Review, April.
(optional) Bronnenberg, Bart J., Jean-Pierre Dube and Carl F. Mela (2010) Do Digital Video
Recorders Influence Sales?, Journal of Marketing Research, 47, 998-1010.
Bryant, Adam (2011) Googles Quest to Build a Better Boss, New York Times.
Chatterjee, Rabikar, Johoshua Eliashber, Hubert Gatignon, and Leonard Lodish (1988) A Practical
Bayesian Approach to Selection of Optimal Market Testing Strategies, Journal of Marketing
Research, 25, 363-375.
Christian, James, Eleanor McDonnel Feit and Mark A. Beltramo (2007) Expert Feature: Conjoint
Analysis at General Motors in Feinberg, Kinnear and Taylor, Modern Marketing Research:
Methods and Cases.
Davenport, Thomas (2009), How to Design Smart Business Experiments, Harvard Business
Review, February.
Eisenberg and Quarto-vonTivadar (2008) Always Be Testing: The Complete Guide to Google
Website Optimizer, Sybex, pp. 19-124.
(optional) Hauser, John and Olivia Toubia (2007), On Managerial Efficiency for Experimental
Designs, Marketing Science, 26(6).

(optional) Grant, Adam M. (2008) The Significance of Task Significance: Job Performance Effects,
Relational Mechanisms and Boundary Conditions, Journal of Applied Psychology, 93, 1, 108124.
Grant, Adam M. and David A. Hofmann (2011) Its Not All About Me: Motivating Hand Hygiene
Among Health Care Professionals by Focusing on Patients, Psychological Science, forthcoming.
(optional) Huber, Joel and Klaus Zwerina (1996), The Importance of Utility Balance in Efficient
Choice Designs, Journal of Marketing Research, 33 (August) 307-317.
JMP (2007), JMP Design of Experiments
Lewis, Randall and David Reiley (2009WP), Does Retail Advertising Work? Measuring the Effects of
Advertising on Sales via a Controlled Experiment on Yahoo!
Monetate 13 Website Testing Wins eBook
Monetate Urban Organic Badging eBook
Thomke, Stefan (2002) Bank of America (A), Harvard Case.
Urban, Glen, John R. Hauser, William J. Quallis, Bruce D. Weinberg, Jonathan D. Bohlmann and
Roberta A. Chicos (1997) Information Acceleration: Validation and Lessons from the Field,
Journal of Marketing Research, 34, 143-153.
Venkatensan, Rajkumar and Paul W. Farris (2008) Advertising Experiments at the Ohio Art
Company, Harvard Case.
Assignments and Grading
Homework and case write-ups (40%)
Homeworks will be assigned approximately once every two weeks. Some assignments will consist of
exercises designed to help you learn the technical mechanics of designing and analyzing experiments.
You will also be required to complete case write-ups or answer essay questions designed to help you
think broadly about how to apply these techniques in the context of a marketing problem and
organization. You may discuss assignments with each other, but each student should write up his or her
submission independently. All assignments will be submitted electronically through the course website.
Mid-term Exam I (20%)
The in-class, pencil-and-paper mid-term exam will cover A/B testing, research design and multivariate
testing and will combine short-answer, computational and essay questions.
Mid-term Exam II (35%)
The second mid-term exam will be similar in format to the first mid-term and will cover all of the material in
the course. It will be held on the last day of classes.
Participation (5%)
Active participation, particularly during case discussions, is expected. Your contributions to the discussion
will be evaluated based on how your questions or comments contribute to the learning of your
classmates. Grandstanding is discouraged.
Disabilities
In compliance with Penn policy and equal access laws, I am available to discuss appropriate academic
accommodations that you may require as a student with a disability. Requests for academic
accommodations need to be made during the first two weeks of the semester, except under unusual
circumstances, to arrange reasonable accommodations. Students must register with Student Disabilities
Services (SDS) for disability verification and for determination of reasonable academic accommodations.
Even if you situation doesnt require a formal accommodations, I encourage you to discuss with me any
special learning needs that you may have, so that I can work with you to make sure that you have the

best possible learning experience. In many cases, I can make simple adjustments that will improve
learning for the entire class. All such discussions will be confidential.
Academic Honesty
Academic honesty is fundamental to our class and the University community and I take issues related to
academic honesty very seriously. Our Code of Academic Integrity is posted at
http://www.upenn.edu/academicintegrity/ and covers cheating, plagiarism, facilitating cheating and other
attempts to gain unfair advantage. I expect that everyone in this course will familiarize themselves with
and follow this code. Suspected violation of the Code in this course will be referred to the Office of
Student Conduct.

Optional Project
For those students who want to explore a particular topic in more depth or who want to try their hand at
fielding an experiment, I am offering the option of doing a project. If you choose to do a project, the
assignments will be weighted as follows: Homework and case write-ups: 30%, Midterm: 15%, Final: 30%,
Project: 20%, Participation: 5%. Final projects will be due on the last day of class.
Projects may involve either fielding an experiment or developing tools for conducting experiments. Below
are some representative ideas for projects:
Use Google Website Optimizer to conduct an A/B test on a real website. Present the results of your
experiment to the website owner. Based on your experience create a video tutorial covering pitfalls
and opportunities in using Website Optimizer.
Conduct an experiment to determine the optimal settings for an eBay auction to sell a particular
product, e.g., USB flash drives. Write up the results of your experiment in the form of a how to guide
for eBay sellers with an appendix describing the details of your experiment. Make sure your guide
addresses the external validity of your findings.
Work with a charitable organization to help them design an experiment on e-mail solicitations for
donations. Present and discuss the results of your experiment to the organizations leadership.
Research the pros and cons of several commercial testing platforms (e.g., Omniture Test & Target)
and write a buyers guide suggesting when one tool would be favored over another.
Create a video tutorial on using JMP custom designs specifically for marketers who are interested in
creating in-store retailing experiments. Cover issues related to constraints, blocking, etc. specifically
in the retail environment.
In grading the projects, I will emphasize the creative application of materials learned in the course and I
expect a careful discussion that demonstrates your mastery of concepts from the class. I also hope that
the output of your project will be useful to some audience, such as your client or your classmates and
when I grade your project, I will consider how well you achieve this goal. Project deliverables may take
any appropriate form, such as a written report, a video tutorial, software, a website, a client presentation
(outside of class), etc.
If you would like to do a project, you should submit a 1-3 page proposal outlining what you plan to do and
what the project deliverables will be. I may respond to your proposal with requested changes. Once we
th
come to an agreement, you will be committed to doing the project. Proposals are due by the 7 week of
classes, but I am happy to accept and discuss proposals earlier in the semester.
In general, projects should be completed individually, but I will consider proposals from pairs of students.
Pairs should submit a single proposal and should explain in their proposal how the partners skills
complement each other and what each student will contribute to the project.

Tentative Course Outline

Introduction (one week)
Experimentation versus data analysis
Statistics review: confidence intervals, hypothesis testing
A/B Testing and Research Design (three weeks)
Designing and analyzing A/B experiments (L&S, Chapter 2)
Research methods (validity, controls, etc.)
Case: Experiments at the Ohio Art Company
A/B/C/D experiments, ANOVA and bocking (L&S, Chapter 3)
Testing in the online environment including demos of Google Website Optimizer and Omniture
Text+Target
Guest lecture: Tools for Online Testing, Nathan Richter (Monetate) + TBD (QVC or URBN)
Field experiments in advertising and management
When should you do an experiment? (An introduction to decision theory)
Multivariate Testing with Two-level Factors (two weeks)
Two-level full factorial experiments (L&S, Chapter 4)
Two-level fractional factorial designs (L&S, Chapter 5)
Cases: Eagle Brands & Magazine Price Test
Guest lecture: A/B and Multivariate Testing in Practice, John Lovett, Web Analytics Demystified
Review & Midterm Exam I (one week)
General Optimal Design (three weeks)
Optional design Information theory and D-optimal designs (L&S, Appendix 8.1, plus additional
readings)
Software tutorial: JMP Custom Designs
Case discussion: Almquist and Wyner
Design of conjoint studies, i.e. multinomial logit studies, bayesian D-optimality
Software tutorial: Choice experiments in JMP
Case discussion: conjoint design for a tablet computer
Predictive and managerial efficiency
Sequential Experiments (one weeks)
More concepts from decision theory
Guest lecture: Sequential Experiments & Multi-Armed Bandits in Practice (Steve Scott, Google)
Experiments in the Organization (two weeks)
Case discussion: Bank of America (A) (building a test-and-learn organization)
Guest lecture: Integrating Experimentation into the Organization (Chris Goldcamp, BarclayCard)
Optional Material (depending on time and interest)
Response surface methods such as Kriging
Guest lecture: Designing tests to evaluate students (Eric Bradlow)