Research Methods for Cyber Security

text.

Part I

Introduction

Outline

Part I. Introduction

Chapter 1 Introduction to Science

Chapter 2 Science and Cyber Security

Chapter 3 Starting Your Research

Part I. Introduction

To raise new questions, new possibilities, to regard old problems from a new angle, requires creative imagination and marks real advance in science.

Albert Einstein

Chapter 1

Introduction to Science

Abstract

This chapter aims to introduce science and the way it has been used to help our understanding of the universe and everything in it, as well as to achieve societal and technological advancement. The philosophy of science, the body of knowledge of science, and the scientific process to discover knowledge will all be discussed. The chapter will provide an overview of the different branches of science, the different forms of scientific research, and the types of methods used. The chapter will discuss empirical evidence provided by scientific research methods and explain the hierarchy of evidence, as well as discussing why the scientific method requires that beliefs and preferences are subordinated to data and information. The continuum of discovery is introduced with a brief historical review of the investigations to understand the planetary motion of the solar system.

Key words

Body of knowledge; continuum of discovery; empirical evidence; research; scientific method; scientific process; skepticism

Science is a powerful tool through which humans have made amazing societal and technological advancement. Science has enabled us to understand our place in the universe, prevent and treat diseases, and even create the Internet. Why then with such a powerful resource at our disposal do we not apply more science practices to cyber security research? If we want cyber security to grow and evolve as a science then it is necessary to start focusing our research on more scientific methods.

In this book, we aim to provide you with and introduction into what it means to execute science in the context of cyber security research by following rigorous and established methods. This book seeks to borrow from the thousands of years of development of the scientific method in other disciplines, and to enhance the conduct of cyber security research as a science in its own right. The intended outcome from using this book is research that is relevant, repeatable, and documented such that colleagues can understand and critique the results and conclusions. The focus of this book is on the practical side of science, the research methods that can be used to perform your research. However, as this may be your first foray into the world of science, it is important to first explain what science is and provide an example of how, over time, it has had a major impact on our knowledge and understanding.

In this chapter we will introduce you to science, defining the various meanings of science, and how science has been cultivated within different fields of research. The continuum of science will be presented to explain where different methods fit in the spectrum. Finally, the concepts of science will be presented through an example. The example will explore the historical progression and improvement of knowledge through science for our, now, well-known solar system.

Chapter Objectives

 Introduce science

 Overview of forms of research and their types of methods

 Describe the continuum of discovery and the hierarchy of evidence

 Explore historical scientific advances in astronomy to contextualize concepts

What is Science

Science is an overloaded term that is used under many different contexts. It is important to understand that science represents three things: a philosophy, a body of knowledge, and a process to discover knowledge. The philosophy of science explores what it means to be an observer of the universe from within it. The body of knowledge of science encapsulates that which we have learned about the universe. Finally, the method of science is a rigorous process to generate evidence for knowledge assimilation from observing the world. While all aspects of science are interesting topics worth in-depth exploration, this book is focused on the practical aspects of science and the methods of collecting knowledge about cyber space and security.

What science is not is engineering. Engineering turns the knowledge gained through science into usable applications and solutions to address challenges or problems. While technically applying science is not science, it is a critical part of cyber security and similar to science, which requires a rigorous process if done right. Therefore, applied research methods are covered in this book.

The process of science has evolved into current forms with the goal of instilling confidence in what we learn from observation. In order to achieve this there are multiple important characteristics of research methods. First, research methods provide a rigorous and methodical approach to study. This ensures that the execution of research is thoroughly conceived and disciplined. Second, research methods provide a process to empirically ground theories and conceptual models. Third, research methods ensure that evidence is driven by logical and reasoned thinking. Finally, research culture is imbued with a healthy level of skepticism to always challenge the approach and results found to instill confidence in accepted knowledge.

Through the process of research science strives for knowledge with two valuable attributes. First, knowledge explains phenomena and what processes affect behavior in systems. Second, it provides an ability to predict future events by projecting from current state and possible stimuli. Armed with this type of knowledge, we are able to effectively engineer technologies that solve societal problems or make some process more efficient. In the case of cyber security, the goal of scientific exploration is to gain the knowledge to make it possible to quantify security and predict what tools and practices will enable us to thwart or stymie cyber attackers.

Types of Science

Science can take many forms. In our quest for knowledge we have explored many different fields of study. Each field represents a core set of questions that to answer them pose unique challenges. Owing to these challenges, each field has developed an approach to scientific research to best generate empirical evidence to validate theories. Out of these approaches have fallen a set of research methods that are used by their research communities to continue to build their knowledge base. The following table provides a brief overview of the different types of researches with example fields of study.

As you can see in Table 1.1, each category of research relies upon different forms of research methods. However, each field uses all forms of research methods at some level. The majority of this book covers these forms of research and specific useful methods for cyber security research. These forms of research include observational, mathematical, experimental methods.

Table 1.1

Description and Examples of Branches of Science

Observational—The phenomenon of interest is embedded in a larger system that is dynamic. The investigator can seek instances where the dynamics are less noisy, but it’s not possible to conduct an experiment free of influences from uncontrolled or uncontrollable variables. Very often, a test bed or microcosm is developed, which is a simplification of the natural environment in order to gain an understanding of basic relationships among variables associated with the phenomenon of interest.

The vast majority of science research engages in some form of observational experiments, using simplifications to gain understanding. One example is biological research using genetically controlled strains of experimental animals to clarify dose responses to chemicals or pharmaceuticals, and tissue cultures to study the basics of cell biology without the complexities of those tissues functioning within a live organism. Another example is physics experiments conducted with accelerators to control the energy and location of subatomic particle collisions in order to be able to collect data on the outcomes of those collisions.

In all of these cases, one can argue that the experimental setup is artificial and that the results may not reflect what actually occurs in the wild. However, collecting data on fortuitous subatomic particle collisions in wild is not practical, and testing chemicals and pharmaceuticals on humans as initial experiments is unethical. Test beds are necessary in order to advance our knowledge.

Mathematical—Unlike experimental and observational research, mathematical research is based upon logic and formal proofs. There is a persistent debate on whether mathematics is actually a science because science relies on evidence, not logic. We will leave that debate to others. What is true is that advances in mathematics are very often the precursors to advances in experimental and observational sciences. Mathematics is necessary for data collection, analysis, and interpretation.

Experimental—The investigator has full control of the phenomenon being observed and the mechanisms for data collection. All of the variables are known and can be either held constant or made to change in order to assess the consequences of those changes on the phenomenon of interest. An example is conducting experiments on a pure substance to determine its state (gas, liquid, solid) at various temperatures and pressures.

The boundary between experimental and observational research is at best a blurry line. Research requires controlled experiments to generate basic understanding and observational experiments to test the relevance of basic understanding to phenomena in the wild. In both cases the conclusions are based on evidence.

One additional form of research we cover is applied research. Applied research leverages concepts and techniques from the other forms of research to study and assess our ability to apply knowledge to solving or addressing a societal problem. Applied research is a core topic in cyber security research because the overall objective of securing a system is applied.

Science is Messy

Short definitions of science typically include the concepts of gaining systematic knowledge of the physical and natural world through data collection and experimentation. Refining the definition becomes problematic because the practice of science is much messier than the conceptual description. Fundamental to science is conducting experiments to investigate relationships and causality for phenomena of interest. An oft-repeated premise in the debate of applying the scientific method to cyber security is that the cyber environment is ever changing and in unpredictable ways. The notion of a repeatable experiment in a cyber environment is sometimes represented as unattainable. Fundamentally, research in cyber security is an observational science in that we can observe what occurs in many ways and at many scales. We are unable to conduct fully controlled experiments at the scales that are meaningful for operational relevance. There is a rich history of operational sciences with exactly this dilemma.

We rely on many observational sciences for understanding the world around us and for anticipating what is about to occur. Examples include atmospheric sciences, ecology, hydrology, agronomy, and cosmology. To further complicate the endeavor, domains such as economics are not only observational but are also profoundly influenced by human judgments, perceptions, and cultural values. In the practice of these fields, it is very often the case that progress occurs through a combination of experiments and observations that span multiple scales. As an example, fundamental to atmospheric science is the understanding of relationships among temperature and pressure to establish a triple-point diagram for the phases of water under different conditions. These diagrams can be developed by conducting controlled experiments in a laboratory. The insights from the experiments and resulting diagrams inform models of atmospheric processes that simply can’t be investigated with controlled experiments. We are unable to generate homogeneous clouds at multikilometer scales to investigate their behavior. Similarly, we can conduct experiments on laboratory animals to gain insight into the efficacy and side effects of new pharmaceuticals, but there is no substitute for human trials to determine the impacts to humans. Variations in genetics, culture, lifestyle choices, and nutrition make it extremely difficult to separate the influence of the pharmaceutical on the response of the individuals from variations in the population.

Fig. 1.1 is a schematic diagram of the progression of observational science and the trade-off between simple experiments that generate results that can be interpreted and repeated versus the realism embodied in those experiments relative to the processes in nature that we want to understand. The path to operational relevance is unknown, but experience indicates that failing to start with simple experiments very often leads to distractions and biases that impede progress to the goal of insight. What makes this difficult is that very often our best efforts result in firmly held beliefs that are simply not true. In the absence of experiments to challenge our assumptions, the apparent progress we’re making is an illusion.

Figure 1.1 Experimental repeatability versus operational relevance.

The science of cyber security shares many attributes with the observational sciences ones listed above. The trade-offs between simple experiments and operational relevance are part of the ongoing discussion in the community on how to increase the scientific rigor of research to better understand cybersystems. The following discussion uses the development of the heliocentric model of the solar system as an example of the evolution of knowledge and fundamental principles in the domain of astronomy where the scale at which we can conduct experiments is vastly different from the scale of the phenomena that we want to understand. The intent is to provide an example that can inform the evolution of the investigations and insights into cybersystems. The path of discovery necessarily starts with empirical descriptions of how the system works. The goal should always be to move toward physical and mathematical models that illuminate physical and systems principles and that generalize across various realizations of cyber systems. The solar system example also emphasizes the impact of the perspective of the observer on what is observed. There are key advances in the progression from a geocentric universe to a heliocentric solar system within a vast universe that occurred only because the investigator was able to analyze the same data from a different perspective.

Hierarchy of Evidence

Not all empirical evidence provided by research methods is equal. Some research methods generate stronger evidence than others, while some develop relative rankings of evidence, called the hierarchy of evidence. This discussion on the Hierarchy of Evidence is not meant to forever cast in stone, better or worse methods of cyber security research. Much discussion was had about even including this topic. However, for a curious reader, it is useful to point out the relative merits and detractions from the various methods of research presented throughout this book, specifically observational methods. The following discussion should hopefully encourage readers to question what sort of research is being conducted or leveraged to build a case for or against any position. The simple question of what sort of study, or experiment, you will use, will help draw out the utility and applicability regardless of the ranking.

One of the commonly cited rankings, at least from the medical community comes from Trisha Greenhalgh who proposes that the research community rank types of research follow this hierarchy:

1. Systematic reviews and meta-analyses of Randomized Control Trials (RCTs) with definitive results.

2. RCTs with definitive results (confidence intervals that do not overlap the threshold clinically significant effect).

3. RCTs with nondefinitive results (a point estimate that suggests a clinically significant effect but with confidence intervals overlapping the threshold for this effect).

4. Cohort studies.

5. Case-control studies.

6. Cross-sectional surveys.

7. Case reports.¹

The hierarchy here places appropriate emphasis on randomized, controlled experimentation. But more valuable than a single experiment is a bevy, a preponderance of multiple rigorous experiments that share similar finds, so-called definitive results. This is the reason that reproducing research is so important in any field, but the lack of reproducible research continues to handicap the field of cyber security. These large analyses of several experiments will lead to foundational understandings in our field.

Of lower rank in the hierarchy are the various types of observational studies. Special emphasis should be made that these forms of research are very much worth doing. They are worth doing in and of themselves, but especially worth doing when other types of research are not able to be conducted (for financial, technical, or ethical reasons). The value is not at all in the decision of what sort of research to conduct, rather, it is in helping the researcher, and the audience better understand how to interpret, build upon, and leverage the results. For example, a case study that concludes with a connection between a behavior and a response would be very different from a large randomized controlled hypothetico-deductive experiment that concludes the same thing. Again it is less about the input and conduct of research and more about how the results are used.

Another hierarchy worth mentioning comes from an article on observational research still from the medical community,² Observational Studies: Cohort and Case-Control Studies, by Song et al. looks at exploration-discussed, evidence-based research (medicine) and describes the levels of evidence.

Levels of Evidence-based Medicine

From Ptolemy to Einstein—Science and the Discovery of the Nature of the Sky

From the dawn of civilization, the night sky has pulled at the human imagination and stimulated the curiosity and creativity of those attempting to navigate land and sea. Along the way, the night sky also became elemental to various belief systems from prosaic forms such as astrology to the tenants of early Christianity that held the Earth as the center of God’s creation based on the Old Testament. The boundaries between the physical and metaphysical pursuits were often blurred or nonexistent.

Early in the development of civilizations the cycles of the Earth (day), Moon (month), and Sun (year) became the basis of timekeeping and calendars. Evidence of the earliest calendars date back to 8000 BC. All of the civilizations in Europe, Asia, and the Americas developed their versions. Calendars continue to be important to the annual repetitions of civil and religious events as well as keeping track of significant cultural histories.

Early calendars were based upon lunar cycles. The average lunar cycle is 29.5 days and 12 lunar cycles generate a year that is 354 days, or 11.25 days short of a solar year. Early civilizations accommodated the offset with intercalary periods to realign the calendar with celestial events, such as the vernal equinox. The adjustments tended to be arbitrary and the accuracy of those calendars was poor. The Egyptians developed the first known solar calendar. The challenge with a solar calendar is having a discrete celestial event to mark the start of the year. Sirius is the brightest star in the sky and is occluded by the Sun for part of the year. The reemergence of Sirius in the eastern sky just before sunrise coincided with the flooding of the Nile River. The heliacal rising of Sirius was used as the starting point of the Egyptian solar year.

Much has been written of astronomic investigations of the solar system and the universe over the course of the 2500 years from the Classical Period to the present. There are contemporary treatises that have survived to the modern day and historical analyses of the trajectory of astronomic discoveries in civilizations located in Asia, Europe, Africa, and the Americas.

The following barely scratches the surface of the richness and complexity of the evolution of our current understanding of the motions of the celestial bodies. Our primary purpose is to learn about the practice of science and the influence of culture through the works of Ptolemy, Copernicus, Galileo, Kepler, Newton, and Einstein.

A Science Continuum of Discovery

As we investigate the development of our knowledge of the solar system, it will be useful to consider the information in the context of a continuum of understanding that helps to recognize the maturation of our insights into how the natural systems function. The paradox is that our ability to assess where we are in the continuum is often thwarted by strongly held beliefs. It is exactly for this reason that the scientific method is essential for advancing our knowledge of the world around us. The scientific method requires that beliefs and preferences are subordinated to data and information.

Fig. 1.2 is a representation of a science continuum of discovery. Many endeavors to understand a particular phenomenon start with a very poor understanding of what actually occurs and what stimulates or causes changes that are interesting to the investigator. This is the left side of the diagram. As individual observations are made, patterns typically emerge. These patterns underpin a conceptual model for how things work. This is an inductive process of taking specific observations to develop a general model.

Figure 1.2 A science continuum of discovery.

Once a conceptual model is developed, it can be challenged for its veracity. The center of the diagram represents the domain of classical experiments conducted within the constraints of the scientific method. This is a deductive process where a general understanding or model is challenged and refined with specific observations and experiments.

Finally, the right side of the diagram is where science becomes useful. As will be demonstrated by the evolution of our knowledge of the solar system and the universe, understanding does not have to be perfect in order to be useful and to have impact.

While the continuum in Fig. 1.2 appears orderly, it is important to recognize that science is a very messy proposition. It is challenging to accurately assess the maturity of understanding of a system and often a single observation can reveal fatal flaws in well-accepted models.

It takes courage and humility to start over at the left end of the continuum. There are also social pressures within scientific domains that impede the very thing everyone wants, which is a more complete understanding of how the world works. The scientific method challenges our desire to be right with evidence of what actually occurs. Cultural belief systems are often so strongly entrenched that challenging them with evidence can put the investigator in peril. Nonetheless, eventually the pursuit of knowledge embodies a recurrence of evidence that cannot be denied.

Did You Know?

The commonly used business lingo paradigm shift actually has its roots in the philosophy of science. In the 1962 book The Structure of Scientific Revolutions physicist and philosopher Thomas Kuhn coined the phrase paradigm shift to attempt to explain the societal influence on science. The premise is that certain conceptual models and theories become entrenched in cultures of research and that there is great social inertia to move away from them. It takes a significant result or amount of evidence to change of perspective, or shift the paradigm, of fields of study from long-held beliefs.

The Ptolemaic Model and Supporting Assumptions

The Earth as the center of the universe, or the geocentric model, emerged early in the Classical Period (roughly 8th-century BC to 6th-century AD) as part of Greek astronomy and philosophy that strongly influenced perceptions of science and philosophy throughout the Mediterranean, southwest Asia, northeast Africa, and Europe. The casual observations available today of the Sun, planets, and stars rotating around the Earth and the apparent stillness of Earth relative to those rotations were the basis of the development of a mathematics that describes and predicts the locations of celestial bodies. Alternative models not centered on Earth were posited but the geocentric model prevailed until it was challenged by Copernicus in the 16th century and eventually gave way to the heliocentric model in the 17th century.

Dig Deeper: Forward and Inverse Problems

A forward problem starts with knowledge of causal factors and calculates the results.

An inverse problem is solved by collecting observations and estimating the causal factors.

The vast majority of questions in science, including the arrangements and motions of planets in the solar system are inverse problems.

The geocentric model of celestial motion is interesting to our understanding of science from several perspectives. The first is that the model is grounded in empirical evidence, or knowledge gained by means of our senses. The ancient astronomers were remarkable in their ability to make measurements of locations of the Sun and planets with the naked eye and to develop mathematical representations of those measurements and movements. Arguably, the most famous and enduring author was Claudius Ptolemy who standardized the geocentric model in a work called Almagest in the 2nd-century AD.

The Ptolemaic model requires several assumptions beyond the Earth being stationary and the center of the universe. Observations of a planet from Earth over time scribes a path that reverses direction. Ptolemy’s resolution of this phenomenon consisted of a larger orbit around the Earth (deferent) and a smaller epicycle, which is a circular path that rotates on the deferent, as shown in Fig. 1.3.

Figure 1.3 Ptolemaic astronomy—large donut shape with a dashed line is the deferent, the small dashed circle is the epicycle. The center is at X. The black dot is the equant point to adjust for being slightly off center and Earth is opposite the equant.

Representing the Ptolemaic conceptual model in equation form provided the means to develop the diagram in Fig. 1.4 that shows the application of the astronomic equations for deferents, epicycles, and equants to the locations of the Sun, Mercury, and Venus relative to the Earth, as printed in the Encyclopedia Britannica in 1771. The dominant features in this diagram are the epicyclic loops. It’s also interesting that the Sun and planets each have their own frequencies of epicycles.

Figure 1.4 Orbits of the Sun, Mercury, and Venus around the Earth, from Encyclopedia Britannica 1771.

Fig. 1.4 highlights the assumption that planetary masses change direction within their complex orbits around the Earth. Galileo’s law of inertia was unknown until 1612. What’s also interesting about this assumption is that there isn’t a proposed mechanism that would cause the Sun or the planets to change direction. The empirical evidence of planets changing direction was sufficient validation.

Another assumption was related to the distances from the Earth to the stars. One line of reasoning in defense of the geocentric model was that if the Earth was in motion then the relative positions of stars in constellations should change due to parallax depending upon the relative location of the Earth. The constellations appear to be constant in the relationships among their stars. The error in this line of evidence was that the Greek astronomers and their descendants assumed the stars to be much closer to Earth than they actually are.

Another assumption in the Ptolemaic model is that the large orbits (deferents) of the planets are circular. This assumption was rooted in a philosophical notion that a circle is in some sense a thing of perfection. It was a matter of belief that God’s creation would necessarily be perfect and therefore the orbits of the planets circular. An additional assumption of the model was that the celestial bodies moved at a constant speed.

Dig Deeper: Confirmation