Simulation for Data Science with R

Table of Contents

Simulation for Data Science with R

Credits

About the Author

About the Reviewer

www.PacktPub.com

eBooks, discount offers, and more

Why subscribe?

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Downloading the color images of this book

Errata

Piracy

Questions

1. Introduction

What is simulation and where is it applied?

Why use simulation?

Simulation and big data

Choosing the right simulation technique

Summary

References

2. R and High-Performance Computing

The R statistical environment

Basics in R

Some very basic stuff about R

Installation and updates

Help

The R workspace and the working directory

Data types

Vectors in R

Factors in R

list

data.frame

array

Missing values

Generic functions, methods, and classes

Data manipulation in R

Apply and friends with basic R

Basic data manipulation with the dplyr package

dplyr – creating a local data frame

dplyr – selecting lines

dplyr – order

dplyr – selecting columns

dplyr – uniqueness

dplyr – creating variables

dplyr – grouping and aggregates

dplyr – window functions

Data manipulation with the data.table package

data.table – variable construction

data.table – indexing or subsetting

data.table – keys

data.table – fast subsetting

data.table – calculations in groups

High performance computing

Profiling to detect computationally slow functions in code

Further benchmarking

Parallel computing

Interfaces to C++

Visualizing information

The graphics system in R

The graphics package

Warm-up example – a high-level plot

Control of graphics parameters

The ggplot2 package

References

3. The Discrepancy between Pencil-Driven Theory and Data-Driven Computational Solutions

Machine numbers and rounding problems

Example – the 64-bit representation of numbers

Convergence in the deterministic case

Example – convergence

Condition of problems

Summary

References

4. Simulation of Random Numbers

Real random numbers

Simulating pseudo random numbers

Congruential generators

Linear and multiplicative congruential generators

Lagged Fibonacci generators

More generators

Simulation of non-uniform distributed random variables

The inversion method

The alias method

Estimation of counts in tables with log-linear models

Rejection sampling

Simulating values from a normal distribution

Simulating random numbers from a Beta distribution

Truncated distributions

Metropolis - Hastings algorithm

A few words on Markov chains

The Metropolis sampler

The Gibbs sampler

The two-phase Gibbs sampler

The multiphase Gibbs sampler

Application in linear regression

The diagnosis of MCMC samples

Tests for random numbers

The evaluation of random numbers – an example of a test

Summary

References

5. Monte Carlo Methods for Optimization Problems

Numerical optimization

Gradient ascent/descent

Newton-Raphson methods

Further general-purpose optimization methods

Dealing with stochastic optimization

Simplified procedures (Star Trek, Spaceballs, and Spaceballs princess)

Metropolis-Hastings revisited

Gradient-based stochastic optimization

Summary

References

6. Probability Theory Shown by Simulation

Some basics on probability theory

Probability distributions

Discrete probability distributions

Continuous probability distributions

Winning the lottery

The weak law on large numbers

Emperor penguins and your boss

Limits and convergence of random variables

Convergence of the sample mean – weak law of large numbers

Showing the weak law of large numbers by simulation

The central limit theorem

Properties of estimators

Properties of estimators

Confidence intervals

A note on robust estimators

Summary

References

7. Resampling Methods

The bootstrap

A motivating example with odds ratios

Why the bootstrap works

A closer look at the bootstrap

The plug-in principle

Estimation of standard errors with bootstrapping

An example of a complex estimation using the bootstrap

The parametric bootstrap

Estimating bias with bootstrap

Confidence intervals by bootstrap

The jackknife

Disadvantages of the jackknife

The delete-d jackknife

Jackknife after bootstrap

Cross-validation

The classical linear regression model

The basic concept of cross validation

Classical cross validation – 70/30 method

Leave-one-out cross validation

k-fold cross validation

Summary

References

8. Applications of Resampling Methods and Monte Carlo Tests

The bootstrap in regression analysis

Motivation to use the bootstrap

The most popular but often worst method

Bootstrapping by draws from residuals

Proper variance estimation with missing values

Bootstrapping in time series

Bootstrapping in the case of complex sampling designs

Monte Carlo tests

A motivating example

The permutation test as a special kind of MC test

A Monte Carlo test for multiple groups

Hypothesis testing using a bootstrap

A test for multivariate normality

Size of the test

Power comparisons

Summary

References

9. The EM Algorithm

The basic EM algorithm

Some prerequisites

Formal definition of the EM algorithm

Introductory example for the EM algorithm

The EM algorithm by example of k-means clustering

The EM algorithm for the imputation of missing values

Summary

References

10. Simulation with Complex Data

Different kinds of simulation and software

Simulating data using complex models

A model-based simple example

A model-based example with mixtures

Model-based approach to simulate data

An example of simulating high-dimensional data

Simulating finite populations with cluster or hierarchical structures

Model-based simulation studies

Latent model example continued

A simple example of model-based simulation

A model-based simulation study

Design-based simulation

An example with complex survey data

Simulation of the synthetic population

Estimators of interest

Defining the sampling design

Using stratified sampling

Adding contamination

Performing simulations separately on different domains

Inserting missing values

Summary

References

11. System Dynamics and Agent-Based Models

Agent-based models

Dynamics in love and hate

Dynamic systems in ecological modeling

Summary

References

Index

Simulation for Data Science with R

Simulation for Data Science with R

Copyright © 2016 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: June 2016

Production reference: 1240616

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-78588-116-9

www.packtpub.com

Credits

Author

Matthias Templ

Reviewer

Gerlinde Dinges

Commissioning Editor

Akram Hussain

Acquisition Editor

Vinay Argekar

Content Development Editor

Aishwarya Pandere

Technical Editor

Tanmayee Patil

Copy Editors

Safis Editing

Angad Singh

Project Coordinator

Nidhi Joshi

Proofreader

Safis Editing

Indexer

Monica Ajmera Mehta

Graphics

Jason Monteiro

Production Coordinator

Nilesh Mohite

Cover Work

Nilesh Mohite

About the Author

Matthias Templ is associated professor at the Institute of Statistics and Mathematical Methods in Economics, Vienna University of Technology (Austria). He is additionally employed as a scientist at the methods unit at Statistics Austria, and together with two colleagues, he owns the company called data-analysis OG. His main research interests are in the areas of imputation, statistical disclosure control, visualization, compositional data analysis, computational statistics, robustness teaching in statistics, and multivariate methods. In the last few years, Matthias has published more than 45 papers in well-known indexed scientific journals. He is the author and maintainer of several R packages for official statistics, such as the R package sdcMicro for statistical disclosure control, the VIM package for visualization and imputation of missing values, the simPop package for synthetic population simulation, and the robCompositions package for robust analysis of compositional data. In addition, he is the editor of the Austrian Journal of Statistics that is free of charge and open-access. The probability is high to find him at the top of a mountain in his leisure time.

I'd like to thank my wife, Barbara, for her mental support for the work on this book. In addition, my thanks goes to all the people who provide open source code to the R project.

About the Reviewer

Gerlinde Dinges works as a methodological researcher at the quality management and methods department of Statistics Austria. She studied at the University of Linz and earned a degree in statistics from the University of Vienna. Since 2002, she has been working extensively in the field of model-based estimation for business statistics and published several papers in this field. Gerlinde has broad experience in working with large (administrative) data and has participated in several European research projects. Her main research interest is in the area of explorative data analysis, business statistics, imputation, visualization, and teaching statistics. Together with her colleagues, she developed a computer-assisted teaching system to enhance and support classroom teaching in an interactive way (http://www.statistik.at/tgui).

www.PacktPub.com

eBooks, discount offers, and more

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

https://www2.packtpub.com/books/subscription/packtlib

Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can search, access, and read Packt's entire library of books.

Why subscribe?

Fully searchable across every book published by Packt

Copy and paste, print, and bookmark content

On demand and accessible via a web browser

Preface

"Everybody seems to think I'm lazy

I don't mind, I think they're crazy

Running everywhere at such a speed

Till they find there's no need (There's no need)"

The Beatles in their song I'm only sleeping

The Monte Carlo way and simulation approach are ways to stay lazy and efficient at the same time. Lazy, since a simulation approach is generally much easier to carry out as compared to an analytical approach—there is mostly no need for analytical approaches, and one might be crazy to neglect the whole world of statistical simulation. Efficient, since it costs minimal efforts to get reliable results, and often simulation is the only approach to get results. The simulation approach in data science and statistics is generally a more intuitive approach compared to analytical solutions. It is not hidden behind a wall of mathematics, and using a simulation approach is often the only way to solve complex problems.

Statistical simulation has thus become an essential area in data science and statistics. It can be seen as a data-driven approach to many practical problems in data science and statistics.

In this book, theory is also explained with illustrative examples using the software environment R, for which advanced data processing features are shown in the book.

This book will thus provide a computational and methodological framework for statistical simulation to users with a computational statistics and/or data science background.

More precisely, the aim of this book is to lay into the hands of the readers a book that explains methods, give advice on the usage of the methods, and provide computational tools to solve common problems in statistical simulation and computer-intense methods.

The core issues are on simulating distributions and datasets, Monte Carlo methods for inference statistics, microsimulation and dynamical systems, and presenting solutions using computer-intense approaches. You will see applications in R not only to better understand the methods but also to gain experience when working on real-world data and real-world problems.

The author of the book has tried to make humorous and amusing examples in certain chapters in order to increase interest, staying catchy and memorable. Next to serious text on methods, curious examples on individual mortality and fertility rates of the author of the book are also present as is the system dynamics from the love/hate story of Prince Henry and Chelsy Davy, the Australian guy in the Austrian mountain trying to reach the highest mountain through an optimization problem, or the weak law of winning the lottery are presented as well.

What this book covers

Chapter 1, Introduction, discusses the general aim of simulation experiments in data science and statistics, why and where simulation is used, and the special case of dealing with big data.

Chapter 2, R and High-Performance Computing, consists of comprehensive text on advanced computing, data manipulation, and visualization with R.

Chapter 3, The Discrepancy between Pencil-Driven Theory and Data-Driven Computational Solutions, reports problems on numerical precision, rounding, and convergence in a deterministic setting.

Chapter 4, Simulation of Random Numbers, starts with the simulation of uniform random numbers and transformation methods to obtain other kinds of distributions. It includes a discussion of various types of Markov chain Monte Carlo (MCMC) methods.

Chapter 5, Monte Carlo Methods for Optimization Problems, introduces deterministic and stochastic optimization methods.

Chapter 6, Probability Theory Shown by Simulation, has a strong focus on basic theorems in statistics; for example, the concept of the weak law of large numbers and the central limit theorem are shown by simulation.

Chapter 7, Resampling Methods, is a comprehensive view on the bootstrap, the jackknife and cross-validation.

Chapter 8, Applications of Resampling Methods and Monte Carlo Tests, shows applications in various fields such as regression, imputation, and time series analysis. In addition, Monte Carlo tests and their variants such as permutation tests and bootstrap tests are presented.

Chapter 9, The EM Algorithm, introduces the expectation maximum method to iteratively obtain an optima. Applications in clustering and imputation of missing values are given.

Chapter 10, Simulation with Complex Data, shows how to simulate synthetic data as well as population data that can be used for the comparison of methods in general or also serve as input for agent-based microsimulation models.

Chapter 11, System Dynamics and Agent-Based Models, discusses agent-based microsimulation models and shows basic models in system dynamics to study complex dynamical systems.

What you need for this book

This book heavily depends on the software environment R, version 3.2 or newer (https://cran.r-project.org/). In most chapters, independent and standalone code is written to show methods and execute examples, and no additional packages of R are needed. For a few chapters, additional R packages such as deSolve, cvTools, laeken, VIM, and few others must be installed within R. The packages dplyr and ggplot2 are used throughout the book.

Optionally, the use of a script editor for R, such as RStudio (https://www.rstudio.com/) or Architect + Eclipse (https://www.openanalytics.eu/architect), is recommended.

Who this book is for

This book is for users who are familiar with computational methods and R. If you want to learn about the advanced features of R, along with computer-intense Monte Carlo methods and tools for statistical simulation, and if you prefer data-driven solutions, then this book is for you.

Conventions

In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning.

R code words in text, filenames, file extensions, pathnames, and dummy URLs are shown as follows:

A block of code is set as follows:

[default]

love <- function(t, x, parms){

  with(as.list(c(parms, x)), {

    dPrince_Harry <- a * Chelsy_Davy

    dChelsy_Davy <- -b * Prince_Harry

    res <- c(dPrince_Harry, dChelsy_Davy)

    list(res)

  })

}

Any command-line input or output is written as follows:

dat <- matrix(c(104,11037,189,11034),2,2, byrow=TRUE) ## Loading required package: grid > confint(oddsratio(dat, log=FALSE)) ##    2.5 %    97.5 % ##  0.4324132 0.6998549

Note

Warnings or important notes appear in a box like this.

Tip

Tips and tricks appear like this.

Reader feedback

Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of.

To send us general feedback, simply e-mail <feedback@packtpub.com>, and mention the book's title in the subject of your message.

If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.

Customer support

Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.

Downloading the example code

You can download the example code files for this book from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.

You can download the code files by following these steps:

Log in or register to our website using your e-mail address and password.

Hover the mouse pointer on the SUPPORT tab at the top.

Click on Code Downloads & Errata.

Enter the name of the book in the Search box.

Select the book for which you're looking to download the code files.

Choose from the drop-down menu where you purchased this book from.

Click on Code Download.

You can also download the code files by clicking on the Code Files button on the book's webpage at the Packt Publishing website. This page can be accessed by entering the book's name in the Search box. Please note that you need to be logged in to your Packt account.

Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:

WinRAR / 7-Zip for Windows

Zipeg / iZip / UnRarX for Mac

7-Zip / PeaZip for Linux

The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Simulation-for-Data-Science-with-R. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Downloading the color images of this book

We also provide you with a PDF file that has color images of the screenshots/diagrams used in this book. The color images will help you better understand the changes in the output. You can download this file from https://www.packtpub.com/sites/default/files/downloads/SimulationforDataSciencewithR_ColorImages.pdf.

Errata

Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website or added to any list of existing errata under the Errata section of that title.

To view the previously submitted errata, go to https://www.packtpub.com/books/content/support and enter the name of the book in the search field. The required information will appear under the Errata section.

Piracy

Piracy of copyrighted material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works in any form on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.

Please contact us at <copyright@packtpub.com> with a link to the suspected pirated material.

We appreciate your help in protecting our authors and our ability to bring you valuable content.

Questions

If you have a problem with any aspect of this book, you can contact us at <questions@packtpub.com>, and we will do our best to address the problem.

Chapter 1. Introduction

In the previous century, the Vienna University of Technology in Vienna enrolled a bachelor study called data engineering and statistics. Basically the content was perfectly related to the nowadays commonly-used term data science. Data-oriented lectures in the area of computer science, such as storing and retrieving data, programming, and data security, were in the curriculum, together with applied lectures on statistics, such as multivariate statistics, biostatistics, financial statistics, statistical learning, and official statistics. We had too few students and after a few years the course was canceled. 16 years later, the picture completely changed. New bachelors and masters courses on data science have been developed everywhere in the world over the last few years. Universities have found that they must offer studies on data science, because the industry needs experts on it, but also developments in statistics in recent years have almost exclusively come from an area called computational statistics. Statistics is the original form of computing data, and computational statistics takes this to an extreme where methods and tools are developed in a highly data-dependent manner, using and developing modern computational tools. Computational statistics and data science are closely related. Computational statistics covers a broad swathe of data science, exclusive data management, and data security issues. Computational statistics (and therefore also data science) has become very popular since the eighties, and it is very likely the most influential area of statistics nowadays. In the field of computational statistics, not only is new methodology developed, but it is also implemented in software – nowadays almost exclusively in the old but modern software environment R.

Data science seems like a good term when your work is driven by data with a less strong component on method and algorithm development than computational statistics, but with a lot of pure computer science topics related to storing, retrieving, and handling data sets. It also differs from computational statistics in various aspects. For example, in the area of data visualization also pure process-related visualizations (airflows in an engine, for example) are a topic in data science but not in computational statistics.

Wikipedia defines data science as a field that:

incorporates varying elements and builds on techniques and theories from many fields, including math, statistics, data engineering, pattern recognition and learning, advanced computing, visualization, uncertainty modeling, data warehousing, and high performance computing with the goal of extracting meaning from data and creating data products.

Data science is the management of the entire modeling process, from data collection, storage and managing data, data pre-processing (editing, imputation), data analysis, and modeling, to automatized reporting and presenting the results, all in a reproducible manner. It is thus also an interdisciplinary study to extract meaning from data with statistics, by using a lot of elements in computer science, as well as general subject-matter skills. In that sense, data science is an extension and continuation of statistics. Data scientists use statistics and data-oriented computer science tools to solve the problems they face.

Statistical simulation is an essential area in data science. The core issues of this book are simulating distributions and data sets, Monte Carlo methods for inference statistics, and presenting solutions on computer-intense approaches. This book discusses various areas in statistical simulation, random number simulation, resampling, Monte Carlo methods, statistical theory explained by simulation experiments, agent-based microsimulation, and system dynamics. The aim is to put a book into the hands of readers that explains methods, gives advice on the use of those methods, and provides computational tools to solve common problems in statistical simulation and computer-intense methods.

In this book, the theory is not just explained. The theory is also made understandable with illustrative examples using the R software environment. The reader will get to grips with the R software environment. After getting the background on popular methods in the field, readers will see applications in R to better understand the methods, as well as to gain experience when working on real-world data and real-world problems.

R itself is perfectly suited to carry out simulations. It should be mentioned that the basics of R are not the topic of the book, but advanced data manipulation and advanced visualization tools are shown in R. The reader should therefore not be a complete newbie in R, and if so, should first read a very basic introduction to R.

Readers will get a brief overview of the problems and possibilities of data-driven simulation and resampling methods.

What is simulation and where is it applied?

Statistical simulation is a numerical method for conducting experiments on a computer in order to solve mathematical problems in a data-driven manner.

Each experiment is carried out in two steps:

Drawing a random outcome.

The subsequent application of an estimation function to the drawn data.

Random draws are made by means of simulating random numbers, such as the numbers produced by a chosen random number generator.

Simulation is applied in different ways. It is applied in sampling to gather information about a random object by observing many realizations of it (Kroese et al., 2014).

As computational power keeps increasing, and new methods and algorithms are being developed, opportunities present themselves to not only conduct innovative research, but also to design better social and economic policies and programs through micro-simulation and agent-based modeling, where states change over time according to defined transition probabilities. Another example is the area of system dynamics, which describes the interaction of populations or individuals. Both topics are part of Chapter 11, System Dynamics and Agent-Based Models.

With simulation experiments, one can even show the concepts of probability theory and the basic theorems of statistics. The (weak and strong) law of large numbers can be explained. We may repeatedly do some experiments with related probability mechanisms. The outcomes of these experiments are random – random events have outcomes that are not known with certainty, but in the long run we know the properties. To toss coins is the simplest example. The most fundamental theorem in mathematical statistics, the central limit theorem, can also be shown by simulation. Using simulation experiments, readers will be able to fully understand this important theorem, while the proof with mathematics needs very detailed knowledge of mass and probability theory. More information on simulation to show the basics in statistics is provided in Chapter 6, Probability Theory Shown by Simulation.

Statistical simulation is also used to show the properties of an estimation method regarding different conditions. One example is the question of how an estimator behaves under different kinds of missing values pattern, or how outliers may corrupt the estimator. When samples have been drawn with complex sampling designs from finite populations, the influence of the sampling design on the estimator of interest can be shown with design-based simulations. Both model-based simulation and design-based simulation are shown in Chapter 10, Simulation with Complex Data. How data might be simulated for different kind of problems is also discussed. This covers the high-dimensional data and complex synthetic populations needed for design-based simulations.

Usually, when statisticians talk about simulations, they mean Monte Carlo simulations. The Monte Carlo simulation method uses repeated random sampling to mimic the null hypothesis or simulate data from a model where an estimation function is applied to the simulated data.

The Monte Carlo simulation approach is also essential in Bayesian statistics, where Markov chain Monte Carlo (MCMC) methods are used to sample parameter values from a posterior distribution (see also Kroese et al., 2014). This will be intensively discussed in Chapter 4, Simulation of Random Numbers. Generally, it is crucial to have an excellent random number generator at hand that allows you to simulate uniformly distributed values. Additionally, it is crucial to transform a uniform distribution into a distribution of interest. We can do this with inversion, rejection sampling, or MCMC methods; see also Chapter 4, Simulation of Random Numbers.

The Monte Carlo simulation approach is also central to estimating certain numerical quantities in general, but especially to estimate statistical uncertainty. It turns out (Chapter 6, Probability Theory Shown by Simulation) that almost no mathematics is used to express the statistical uncertainty for any complex estimator. The Monte Carlo simulation method is a data-driven and computational tool. It is the perfect tool for data scientists to make statistical inferences without getting lost in the world of mathematics.

Another application of Monte Carlo simulation is multi-dimensional integrals, which can be solved via Monte Carlo techniques, typically by drawing random numbers