{http://www.personal.kent.edu/~rmuhamma/OpSystems/os.html} {http://williamstallings.com/Extras/OS-Notes/notes.html} {http://www.linux.org/lessons/beginner/toc.html } {http://beginnerlinuxtutorial.

com}

Operating systems Lecture Notes

Definition: An operating system (OS) is a set of system software
programs in a computer that regulate the ways application software programs use the computer hardware and the ways that users control the computer. Operating systems are found on almost any device that contains a computer with multiple programsfrom cellular phones and video game consoles to supercomputers and web servers. Operating systems are twosided platforms, bringing consumers (the first side) and program developers (the second side) together in a single market. Some popular modern operating systems for personal computers include Microsoft Windows, Mac OS X, and Linux

VIEWS OF AN OPERATING SYSTEM

As a scheduler/allocator: The operating system has resources for which it is in charge. Responsible for handing them out (and later recovering them). Resources include CPU, memory, I/O devices, and disk space.

As a virtual machine:

Operating system provides a "new" machine.

This machine could be the same as the underlying machine. Allows many users to believe they have an entire piece of hardware to themselves. This could implement a different, perhaps more powerful, machine. Or just a different machine entirely. It may be useful to be able to completely simulate another machine with your current hardware. As a multiplexor:

Allows sharing of resources, and provides protection from interference. Provides for a level of cooperation between users.

Desirable Qualities of an operating system

Usability: o Robustness it must accept all valid input without error, and gracefully handle all invalid inputs o Consistency E.g., if "-" means options flags in one place, it means it in another. o Proportionality Simple, cheap and frequent things are easy. Also, expensive and disastrous things are hard. o Forgiving Errors can be recovered from. Reasonable error messages. o Convenient Not necessary to repeat things, or do awkward procedures to accomplish things. Example copying a file took a batch job. o Powerful Has high level facilities. Facilities o Sufficient for intended use. o Complete. Dont leave out part of a facility. o Appropriate. Do not use fixed-width field input from terminal. Costs o Want low cost and efficient services. o Good algorithms. o Low overhead. Cost of doing nothing should be low. E.g., idle time at a terminal. o Low maintenance cost. System should not require constant attention. Adaptability o Tailored to the environment. Support necessary activities. Do not impose unnecessary restrictions. What are the things people do most -- make them easy. o Changeable over time. Adapt as needs and resources change. E.g., expanding memory and new devices, or new user population. o Extendible-Extensible Adding new facilities and features - which look like the old ones.

Objectives of Operating Systems

Modern Operating systems generally have following three major goals. Operating systems generally accomplish these goals by running processes in low privilege and providing service calls that invoke the operating system kernel in high-privilege state. To hide details of hardware by creating abstraction An abstraction is software that hides lower level details and provides

a set of higher-level functions. An operating system transforms the physical world of devices, instructions, memory, and time into virtual world that is the result of abstractions built by the operating system. There are several reasons for abstraction. First, the code needed to control peripheral devices is not standardized. Operating systems provide subroutines called device drivers that perform operations on behalf of programs for example, input/output operations. Second, the operating system introduces new functions as it abstracts the hardware. For instance, operating system introduces the file abstraction so that programs do not have to deal with disks. Third, the operating system transforms the computer hardware into multiple virtual computers, each belonging to a different program. Each program that is running is called a process. Each process views the hardware through the lens of abstraction. Fourth, the operating system can enforce security through abstraction. To allocate resources to processes (Manage resources) An operating system controls how processes (the active agents) may access resources (passive entities).

Provide a pleasant and effective user interface The user interacts with the operating systems through the user interface and usually interested in the look and feel of the operating system. The most important components of the user interface are the command interpreter, the file system, on-line help, and application integration. The recent trend has been toward increasingly integrated graphical user interfaces that encompass the activities of multiple processes on networks of computers.

Operating Systems Structure System Components

Even though, not all systems have the same structure many modern operating systems share the same goal of supporting the following types of system components.

Process Management The operating system manages many kinds of activities ranging from user programs to system programs like printer spooler, name servers, file server etc. Each of these activities is encapsulated in a process. A process includes the complete execution context (code, data, PC, registers, OS resources in use etc.). It is important to note that a process is not a program. A process is only ONE instant of a program in execution. There are many processes that can be running the same program.

The five major activities of an operating system in regard to process management are

Creation and deletion of user and system processes. Suspension and resumption of processes. A mechanism for process synchronization. A mechanism for process communication. A mechanism for deadlock handling.

Main-Memory Management Primary-Memory or Main-Memory is a large array of words or bytes. Each word or byte has its own address. Main-memory provides storage that can be accessed directly by the CPU. That is to say for a program to be executed, it must be in the main memory. The major activities of an operating in regard to memory-management are: Keep track of which part of memory are currently being used and by whom. Decide which process are loaded into memory when memory space becomes available. Allocate and deallocate memory space as needed.

File Management A file is a collection of related information defined by its creator. Computers can store files on the disk (secondary storage), which provide long term storage. Some examples of storage media are magnetic tape, magnetic disk and optical disk. Each of these media has its own properties like speed, capacity, data transfer rate and access methods. file systems are normally organized into directories to ease their use. These directories may contain files and other directions. The five main major activities of an operating system in regard to file management are 1. 2. 3. 4. 5. The creation and deletion of files. The creation and deletion of directions. The support of primitives for manipulating files and directions. The mapping of files onto secondary storage. The back up of files on stable storage media.

I/O System Management I/O subsystem hides the peculiarities of specific hardware devices from the user. Only the device driver knows the peculiarities of the specific device to whom it is assigned.

Secondary-Storage Management Generally speaking, systems have several levels of storage, including primary storage, secondary storage and cache storage. Instructions and data must be placed in primary storage or cache to be referenced by a running program. Because main memory is too small to accommodate all data and programs, and its data are lost when power is lost, the computer system must provide secondary storage to back up main memory. Secondary storage consists of tapes, disks, and other media designed to hold information that will eventually be accessed in primary storage (primary, secondary, cache) is ordinarily divided into bytes or words consisting of a fixed number of bytes. Each location in storage has an address; the set of all addresses available to a program is called an address space. The three major activities of an operating system in regard to secondary storage management are: 1. Managing the free space available on the secondary-storage device. 2. Allocation of storage space when new files have to be written. 3. Scheduling the requests for memory access.

Networking A distributed system is a collection of processors that do not share memory, peripheral devices, or a clock. The processors communicate with one another through communication lines called network. The communication-network design must consider routing and connection strategies, and the problems of contention and security.

Protection System If a computer system has multiple users and allows the concurrent execution of multiple processes, then the various processes must be protected from one another's activities. Protection refers to mechanism for controlling the access of programs, processes, or users to the resources defined by a computer system.

Command Interpreter System A command interpreter is an interface of the operating system with the user. The user gives commands which are executed by the operating system (usually by turning them into system calls). The main function of a command interpreter is to get and execute the next user specified command. Command-Interpreter is usually not part of the kernel, since multiple command interpreters (shell, in UNIX/LINUX terminology) may be supported by an operating system,

and they do not really need to run in kernel mode. There are two main advantages to separating the command interpreter from the kernel. 1. If we want to change the way the command interpreter looks, i.e., I want to change the interface of command interpreter, I am able to do that if the command interpreter is separate from the kernel. I cannot change the code of the kernel so I cannot modify the interface. 2. If the command interpreter is a part of the kernel it is possible for a malicious process to gain access to certain part of the kernel that it showed not have to avoid this ugly scenario it is advantageous to have the command interpreter separate from kernel.

Operating Systems Services

Following are the five services provided by an operating system to the convenience of the users. Program Execution The purpose of a computer system is to allow the user to execute programs. So the operating system provides an environment where the user can conveniently run programs. The user does not have to worry about the memory allocation or multitasking or anything. These things are taken care of by the operating system. Running a program involves the allocating and deallocating memory, CPU scheduling in case of multiprocess. These functions cannot be given to the user-level programs. So user-level programs cannot help the user to run programs independently without the help from operating systems.

I/O Operations Each program requires an input and produces output. This involves the use of I/O. The operating systems hides the user the details of underlying hardware for the I/O. All the user sees is that the I/O has been performed without any details. So the operating system by providing I/O makes it convenient for the users to run programs. For efficiency and protection users cannot control I/O, so this service cannot be provided by user-level programs. File System Manipulation The output of a program may need to be written into new files or input taken from some files. The operating system provides this service. The user does not have to worry about secondary storage management. User gives a command for reading or writing to a file and sees the task

being accomplished. Thus operating systems make it easier for user programs to accomplish their task. This service involves secondary storage management. The speed of I/O that depends on secondary storage management is critical to the speed of many programs and hence it is best relegated to the operating systems to manage it than giving individual users the control of it. Communications There are instances where processes need to communicate with each other to exchange information. It may be between processes running on the same computer or running on the different computers. By providing this service the operating system relieves the user of the worry of passing messages between processes. In case where the messages need to be passed to processes on the other computers through a network it can be done by the user programs. The user program may be customized to the specifics of the hardware through which the message transits and provides the service interface to the operating system. Error Detection An error in one part of the system may cause malfunctioning of the complete system. To avoid such a situation the operating system constantly monitors the system for detecting the errors. This relieves the user of the worry of errors propagating to various part of the system and causing malfunctioning. This service cannot be allowed to be handled by user programs because it involves monitoring and in some cases altering the memory area or deallocation of memory for a faulty process, or may be relinquishing the CPU of a process that goes into an infinite loop. These tasks are too critical to be handed over to the user programs. A user program if given these privileges can interfere with the correct (normal) operation of the operating systems.

System Calls and System Programs

System calls provide an interface between the process and the operating system. System calls allow user-level processes to request some services from the operating system which the process itself is not allowed to do. In handling the trap, the operating system will enter in the kernel mode, where it has access to privileged instructions, and can perform the desired service on behalf of user-level process. It is because of the critical nature of operations that the operating system itself does them every time they are needed. For example, an I/O process involves a system call telling the operating system to read or write particular area and this request is satisfied by the operating system. System programs provide basic functioning to users so that they do not need to write their own environment for program development (editors, compilers) and program execution (shells). In some sense, they are bundles of useful system calls.

Layered Approach Design

In this case the system is easier to debug and modify, because changes affect only limited portions of the code, and programmer does not have to know the details of the other layers.

Information is also kept only where it is needed and is accessible only in certain ways, so bugs affecting that data are limited to a specific module or layer.

Mechanisms and Policies

The policies specify what is to be done while the mechanism specifies how it is to be done. For instance, the timer construct for ensuring CPU protection is mechanism. On the other hand, the decision of how long the timer is set for a particular user is a policy decision. The separation of mechanism and policy is important to provide flexibility to a system. If the interface between mechanism and policy is well defined, the change of policy may affect only a few parameters. On the other hand, if interface between these two is vague or not well defined, it might involve much deeper change to the system. Once the policy has been decided it gives the programmer the choice of using his/her own implementation. Also, the underlying implementation may be changed for a more efficient one without much trouble if the mechanism and policy are well defined. Specifically, separating these two provides flexibility in a variety of ways. First, the same mechanism can be used to implement a variety of policies, so changing the policy might not require the development of a new mechanism, but just a change in parameters for that mechanism from a library of mechanisms. Second, the mechanism can be changed for example, to increase its efficiency or to move to a new platform, without changing the overall policy.

Process
Definition The term process has been given many definitions for instance

A program in Execution. An asynchronous activity. The 'animated sprit' of a procedure in execution. The entity to which processors are assigned. The 'dispatchable' unit.

and many more definitions have given. As we can see from above that there is no universally agreed upon definition, but the definition "Program in Execution" seem to be most frequently used. And this is a concept are will use in the present study of operating systems. What is the relation between process and program? It is same beast with different name or when this beast is sleeping (not executing) it is called program and when it is executing becomes process. Well, to be very precise. Process is not the same as program. In the following discussion we point out some of the difference between process and program. Process is not the same as program. A process is more than a program code. A process is an 'active' entity as opposed to program which is considered to be a 'passive' entity. As we all

know that a program is an algorithm expressed in some suitable notation, (e.g., programming language). Being passive, a program is only a part of process. A Process includes: Current value of Program Counter (PC) Contents of the processors registers Value of the variables The process stacks (SP) which typically contains temporary data such as subroutine parameter, return address, and temporary variables. A data section that contains global variables.

A process is the unit of work in a system. In Process model, all software on the computer is organized into a number of sequential processes. A process includes PC, registers, and variables. Conceptually, each process has its own virtual CPU. In reality, the CPU switches back and forth among processes. (The rapid switching back and forth is called multiprogramming).

Process State
The process state consist of everything necessary to resume the process execution if it is somehow put aside temporarily. The process state consists of at least following:

Code for the program. Program's static data. Program's dynamic data. Program's procedure call stack. Contents of general purpose registers. Contents of program counter (PC) Contents of program status word (PSW). Operating Systems resource in use.

A process goes through a series of discrete process states. New State: The process being created. Running State: A process is said to be running if it has the CPU, that is, process actually using the CPU at that particular instant. Blocked (or waiting) State: A process is said to be blocked if it is waiting for some event to happen such as an I/O completion before it can proceed. Note that, a process is unable to run until some external event happens. Ready State: A process is said to be ready if it can use a CPU if one were available. A ready state process is runable but temporarily stopped running to let another process run. Terminated state: The process has finished execution.

Logically, the 'Running' and 'Ready' states are similar. In both cases the process is willing to run, only in the case of 'Ready' state, there is temporarily no CPU available for it. The 'Blocked' state is different from the 'Running' and 'Ready' states in that the process cannot run, even if the CPU is available.

Process Operations
Process Creation
In general-purpose systems, some way is needed to create processes as needed during operation. There are four principal events that led to processes creation. System initialization. Execution of a process Creation System calls by a running process. A user request to create a new process. Initialization of a batch job.

Foreground processes interact with users. Background processes, are those that stay in background, sleeping but suddenly springing to life to handle activity such as email, webpage, printing, and so on. Background processes are called daemons. This call creates an exact clone of the calling process. A process may create a new process by some create process such as 'fork', the creating process is called parent process and the created one is called the child processes. Only one parent is needed to create a child process. This creation of process (processes) yields a hierarchical structure of processes, each child has only one parent but each parent may have many children. After the fork, the two processes, the parent and the child, have the same memory image, the same environment strings and the same open files. After a process is created, both the parent and child have their own distinct address space. If either process changes a word in its address space, the change is not visible to the other process. Following are some reasons for creation of a process

User logs on. User starts a program. Operating systems creates process to provide service, e.g., to manage printer. Some program starts another process, e.g., Netscape calls xv to display a picture.

Process Termination
A process terminates when it finishes executing its last statement. Its resources are returned to the system, it is purged from any system lists or tables, and its process control block (PCB) is erased i.e., the PCB's memory space is returned to a free memory pool. The new process terminates the existing process, usually due to following reasons: Normal Exist done their job.

Most processes terminates because they have

Error Exist When process discovers a fatal error. For example, a user tries to compile a program that does not exist. Fatal Error An error caused by process due to a bug in program for example, executing an illegal instruction, referring nonexisting memory or dividing by zero. Killed by another Process A process executes a system call telling the Operating Systems to terminate some other process.

Process State Transitions

Following are six(6) possible transitions among above mentioned five (5) states FIGURE Transition 1 occurs when process discovers that it cannot continue. If running process initiates an I/O operation before its allotted time expires, the running process voluntarily relinquishes the CPU.

This state transition is: Block (process-name): Running Block. Transition 2 occurs when the scheduler decides that the running process has run long enough and it is time to let another process have CPU time.

This state transition is: Time-Run-Out (process-name): Running Ready. Transition 3 occurs when all other processes have had their share and it is time for the first process to run again

this state transition is: Dispatch (process-name): Ready Running. Transition 4 occurs when the external event for which a process was waiting (such as arrival of input) happens.

This state transition is: Wakeup (process-name): Blocked Ready.

Transition 5 occurs when the process is created.

This state transition is:

Admitted (process-name): New Ready.

Transition 6 occurs when the process has finished execution.

This state transition is: Exit (process-name): Running Terminated.

Process Control Block

A process in an operating system is represented by a data structure known as a process control block (PCB) or process descriptor. The PCB contains important information about the specific process including The current state of the process i.e., whether it is ready, running, waiting, or whatever. Unique identification of the process in order to track "which is which" information. A pointer to parent process. Similarly, a pointer to child process (if it exists). The priority of process (a part of CPU scheduling information). Pointers to locate memory of processes. A register save area. The processor it is running on.

The PCB is a certain store that allows the operating systems to locate key information about a process. Thus, the PCB is the data structure that defines a process to the operating systems.

Threads

Threads Processes Vs Threads Why Threads? User-Level Threads Kernel-Level Threads Advantages of Threads over Multiple Processes Disadvantages of Threads over Multiprocesses Application that Benefits from Threads Application that cannot benefit from Threads Resources used in Thread creation and Process Creation Context Switch Major Steps of Context Switching Action of Kernel to Context switch among threads Action of kernel to Context switch among processes

Threads Despite the fact that a thread must execute in process, the process and its associated threads are different concepts. Processes are used to group resources together and threads are the entities

scheduled for execution on the CPU. A thread is a single sequence stream within a process. Because threads have some of the properties of processes, they are sometimes called lightweight processes. In a process, threads allow multiple executions of streams. In many respect, threads are a popular way to improve application through parallelism. The CPU switches rapidly back and forth among the threads giving illusion that the threads are running in parallel. Like a traditional process i.e., process with one thread, a thread can be in any of several states (Running, Blocked, Ready or terminated). Each thread has its own stack. Since threads will generally call different procedures and thus a different execution history. This is why a thread needs its own stack. An operating system that has thread facility, the basic unit of CPU utilization is a thread. A thread has or consists of a program counter (PC), a register set, and a stack space. Threads are not independent of one other like processes as a result threads shares with other threads their code section, data section, OS resources also known as task, such as open files and signals. Processes Vs Threads As we mentioned earlier that in many respect threads operate in the same way as that of processes. Some of the similarities and differences are: Similarities Like processes threads share CPU and only one thread is active (running) at a time. Like processes, threads within a process execute sequentially. Like processes, threads can create children. And like process, if one thread is blocked, another thread can run.

Differences

Unlike processes, threads are not independent of one another. Unlike processes, all threads can access every address in the

task. Unlike processes, threads are design to assist one other. Note that processes might or might not assist one another because processes may originate from different users. Why Threads? Following are some reasons why we use threads in designing operating systems. 1. A process with multiple threads makes a great server for example printer server. 2. Because threads can share common data, they do not need to use interprocess communication. 3. Because of the very nature, threads can take advantage of multiprocessors. Threads are cheap in the sense that

1. They only need a stack and storage for registers therefore, threads are cheap to create. 2. Threads use very little resources of an operating system in which they are working. That is, threads do not need new address space, global data, program code or operating system resources. 3. Context switching is fast when working with threads. The reason is that we only have to save and/or restore PC, SP and registers. The biggest drawback is that there is no protection between threads. User-Level Threads User-level threads implement in user-level libraries, rather than via systems calls, so thread switching does not need to call the operating system and to cause interrupt to the kernel. In fact, the kernel knows nothing about user-level threads and manages them as if they were singlethreaded processes. Advantages: The most obvious advantage of this technique is that a user-level threads package can be implemented on an Operating System that does not support threads. Some other advantages are User-level threads do not require modification to operating systems. Simple Representation: Each thread is represented simply by a PC, registers, stack and a small control block, all stored in the user process address space. Simple Management: This simply means that creating a thread, switching between threads and synchronization between threads can all be done without intervention of the kernel. Fast and Efficient: Thread switching is not much more expensive than a procedure call.

Disadvantages: There is a lack of coordination between threads and operating system kernel. Therefore, a process as a whole gets one time slice irrespective of whether the process has one thread or 1000 threads within. It is up to each thread to relinquish control to other threads. User-level threads require non-blocking systems call i.e., a multithreaded kernel. Otherwise, entire process will blocked in the kernel, even if there are threads that can be run left in the processes. For example, if one thread causes a page fault, the process blocks.

Kernel-Level Threads In this method, the kernel knows about and manages the threads. No runtime system is needed in this case. Instead of thread table in each process, the kernel has a thread table that keeps track of all threads in the system. In addition, the kernel also maintains the traditional process table to keep track of processes. Operating Systems kernel provides system calls to create and manage threads. Advantages: Because the kernel has full knowledge of all threads, the Scheduler may decide to give more time to a process having large number of threads than process having small number of threads. Kernel-level threads are especially good for applications that frequently block.

Disadvantages: The kernel-level threads are slow and inefficient. For instance, threads operations are hundreds of times slower than that of userlevel threads. Since kernel must manage and schedule threads as well as processes. It requires a full thread control block (TCB) for each thread to maintain information about threads. As a result there is significant overhead and increased in kernel complexity.

Advantages of Threads over Multiple Processes Context Switching Threads are very inexpensive to create and destroy, and they are inexpensive to represent. For example, they require space to store, the PC, the SP, and the general-purpose registers, but they do not require space to share memory information, Information about open files of I/O devices in use, etc. With so little context, it is much faster to switch between threads. In other words, it is relatively easier for a context switch using threads. Sharing Treads allow the sharing of a lot resources that cannot be shared in process, for example, sharing code section, data section, Operating System resources like open file etc.

Disadvantages of Threads over Multiprocesses Blocking The major disadvantage if that if the kernel is single threaded, a system call of one thread will block the whole process and CPU may be idle during the blocking period. Security Since there is, an extensive sharing among threads there is a potential problem of security. It is quite possible that one thread over writes the stack of another thread (or damaged shared data) although it is very unlikely since threads are meant to cooperate on a single task.

Application that Benefits from Threads A proxy server satisfying the requests for a number of computers on a LAN would be benefited by a multi-threaded process. In general, any program that has to do more than one task at a time could benefit from multitasking. For example, a program that reads input, process it, and outputs could have three threads, one for each task. Application that cannot Benefit from Threads Any sequential process that cannot be divided into parallel task will not benefit from thread, as they would block until the previous one completes. For example, a program that displays the time of the day would not benefit from multiple threads. Resources used in Thread Creation and Process Creation When a new thread is created it shares its code section, data section and operating system resources like open files with other threads. But it is allocated its own stack, register set and a program counter. The creation of a new process differs from that of a thread mainly in the fact that all the shared resources of a thread are needed explicitly for each process. So though two processes may be running the same piece of code they need to have their own copy of the code in the main memory to be able to run. Two processes also do not share other resources with each other. This makes the creation of a new process very costly compared to that of a new thread. Context Switch To give each process on a multiprogrammed machine a fair share of the CPU, a hardware clock generates interrupts periodically. This allows the operating system to schedule all processes in main memory (using scheduling algorithm) to run on the CPU at equal intervals. Each time a clock interrupt occurs, the interrupt handler checks how much time the current running process has used. If it has used up its entire time slice, then the CPU scheduling algorithm (in kernel) picks a different process to run. Each switch of the CPU from one process to another is called a context switch. Major Steps of Context Switching The values of the CPU registers are saved in the process table of the process that was running just before the clock interrupt occurred. The registers are loaded from the process picked by the CPU scheduler to run next.

In a multiprogrammed uniprocessor computing system, context switches occur frequently enough that all processes appear to be running concurrently. If a process has more than one thread, the Operating System can use the context switching technique to schedule the threads so they appear to execute in parallel. This is the case if threads are implemented at the kernel level. Threads can also be implemented entirely at the user level in run-time libraries. Since in this case no thread scheduling is provided by the Operating System, it is the responsibility of

the programmer to yield the CPU frequently enough in each thread so all threads in the process can make progress. Action of Kernel to Context Switch among Threads The threads share a lot of resources with other peer threads belonging to the same process. So a context switch among threads for the same process is easy. It involves switch of register set, the program counter and the stack. It is relatively easy for the kernel to accomplished this task. Action of kernel to Context Switch Among Processes Context switches among processes are expensive. Before a process can be switched its process control block (PCB) must be saved by the operating system. The PCB consists of the following information:

The process state. The program counter, PC. The values of the different registers. The CPU scheduling information for the process. Memory management information regarding the process. Possible accounting information for this process. I/O status information of the process.

When the PCB of the currently executing process is saved the operating system loads the PCB of the next process that has to be run on CPU. This is a heavy task and it takes a lot of time.

CPU/Process Scheduling
The assignment of physical processors to processes allows processors to accomplish work. The problem of determining when processors should be assigned and to which processes is called processor scheduling or CPU scheduling. When more than one process is runable, the operating system must decide which one first. The part of the operating system concerned with this decision is called the scheduler, and the algorithm it uses is called the scheduling algorithm. Goals of Scheduling (objectives) In this section we try to answer following question: What the scheduler try to achieve? Many objectives must be considered in the design of a scheduling discipline. In particular, a scheduler should consider fairness, efficiency, response time, turnaround time, throughput, etc., Some of these goals depends on the system one is using for example batch system, interactive system or real-time system, etc. but there are also some goals that are desirable in all systems. General Goals Fairness Fairness is important under all circumstances. A scheduler makes sure that each process

gets its fair share of the CPU and no process can suffer indefinite postponement. Note that giving equivalent or equal time is not fair. Think of safety control and payroll at a nuclear plant. Policy Enforcement The scheduler has to make sure that system's policy is enforced. For example, if the local policy is safety then the safety control processes must be able to run whenever they want to, even if it means delay in payroll processes. Efficiency Scheduler should keep the system (or in particular CPU) busy cent percent of the time when possible. If the CPU and all the Input/Output devices can be kept running all the time, more work gets done per second than if some components are idle. Response Time A scheduler should minimize the response time for interactive user. Turnaround A scheduler should minimize the time batch users must wait for an output. Throughput A scheduler should maximize the number of jobs processed per unit time. A little thought will show that some of these goals are contradictory. It can be shown that any scheduling algorithm that favors some class of jobs hurts another class of jobs. The amount of CPU time available is finite, after all. Preemptive Vs Non-preemptive Scheduling The Scheduling algorithms can be divided into two categories with respect to how they deal with clock interrupts. Non-preemptive Scheduling A scheduling discipline is non-preemptive if, once a process has been given the CPU, the CPU cannot be taken away from that process. Following are some characteristics of non-preemptive scheduling 1. In non-preemptive systems, short jobs are made to wait by longer jobs but the overall treatment of all processes is fair. 2. In non-preemptive system, response times are more predictable because incoming high priority jobs can not displace waiting jobs. 3. In non-preemptive scheduling, a scheduler executes jobs in the following two situations. a. When a process switches from running state to the waiting state. b. When a process terminates.

Preemptive Scheduling A scheduling discipline is preemptive if, once a process has been given the CPU, the CPU can take it away. The strategy of allowing processes that are logically runable to be temporarily suspended is called Preemptive Scheduling and it is contrast to the "run to completion" method.

Scheduling Algorithms
CPU Scheduling deals with the problem of deciding which of the processes in the ready queue is to be allocated the CPU. Following are some scheduling algorithms we will study

FCFS Scheduling. Round Robin Scheduling. SJF Scheduling. SRT Scheduling. Priority Scheduling. Multilevel Queue Scheduling. Multilevel Feedback Queue Scheduling.

First-Come-First-Served (FCFS) Scheduling

Other names of this algorithm are:

First-In-First-Out (FIFO) Run-to-Completion Run-Until-Done

Perhaps, First-Come-First-Served algorithm is the simplest scheduling algorithm. Processes are dispatched according to their arrival time on the ready queue. Being a non-preemptive discipline, once a process has a CPU, it runs to completion. The FCFS scheduling is fair in the formal sense or human sense of fairness but it is unfair in the sense that long jobs make short jobs wait and unimportant jobs make important jobs wait. FCFS is more predictable than most of other schemes since it offers time. FCFS scheme is not useful in scheduling interactive users because it cannot guarantee good response time. The code for FCFS scheduling is simple to write and understand. One of the major drawbacks of this scheme is that the average time is often quite long. The First-Come-First-Served algorithm is rarely used as a master scheme in modern operating systems but it is often embedded within other schemes

Round Robin Scheduling

One of the oldest, simplest, fairest and most widely used algorithm is round robin (RR).

In the round robin scheduling, processes are dispatched in a FIFO manner but are given a limited amount of CPU time called a time-slice or a quantum. If a process does not complete before its CPU-time expires, the CPU is preempted and given to the next process waiting in a queue. The preempted process is then placed at the back of the ready list. Round Robin Scheduling is preemptive (at the end of time-slice) therefore it is effective in time-sharing environments in which the system needs to guarantee reasonable response times for interactive users. The only interesting issue with round robin scheme is the length of the quantum. Setting the quantum too short causes too many context switches and lower the CPU efficiency. On the other hand, setting the quantum too long may cause poor response time and approximates FCFS. In any event, the average waiting time under round robin scheduling is often quite long.

Shortest-Job-First (SJF) Scheduling

Other name of this algorithm is Shortest-Process-Next (SPN). Shortest-Job-First (SJF) is a non-preemptive discipline in which waiting job (or process) with the smallest estimated run-time-to-completion is run next. In other words, when CPU is available, it is assigned to the process that has smallest next CPU burst. The SJF scheduling is especially appropriate for batch jobs for which the run times are known in advance. Since the SJF scheduling algorithm gives the minimum average time for a given set of processes, it is probably optimal. The SJF algorithm favors short jobs (or processors) at the expense of longer ones. The obvious problem with SJF scheme is that it requires precise knowledge of how long a job or process will run, and this information is not usually available. The best SJF algorithm can do is to rely on user estimates of run times. In the production environment where the same jobs run regularly, it may be possible to provide reasonable estimate of run time, based on the past performance of the process. But in the development environment users rarely know how their program will execute. Like FCFS, SJF is non preemptive therefore, it is not useful in timesharing environment in which reasonable response time must be guaranteed.

Shortest-Remaining-Time (SRT) Scheduling

The SRT is the preemptive counterpart of SJF and useful in time-sharing environment.

In SRT scheduling, the process with the smallest estimated runtime to completion is run next, including new arrivals. In SJF scheme, once a job begins executing, it run to completion. In SRT scheme, a running process may be preempted by a new arrival process with shortest estimated run-time. The algorithm SRT has higher overhead than its counterpart SJF. The SRT must keep track of the elapsed time of the running process and must handle occasional preemptions. In this scheme, arrival of small processes will run almost immediately. However, longer jobs have even longer mean waiting time.

Priority Scheduling
The basic idea is straightforward: each process is assigned a priority, and the first priority is allowed to run. Equal-Priority processes are scheduled in FCFS order. The shortest-Job-First (SJF) algorithm is a special case of general priority scheduling algorithm. An SJF algorithm is simply a priority algorithm where the priority is the inverse of the (predicted) next CPU burst. That is, the longer the CPU burst, the lower the priority and vice versa. Priority can be defined either internally or externally. Internally defined priorities use some measurable quantities or qualities to compute priority of a process. Examples of Internal priorities are Time limits. Memory requirements. File requirements, for example, number of open files. CPU Vs I/O requirements.

Externally defined priorities are set by criteria that are external to operating system such as

The importance of process. Type or amount of funds being paid for computer use. The department sponsoring the work. Politics.

Priority scheduling can be either preemptive or non preemptive A preemptive priority algorithm will preemptive the CPU if the priority of the newly arrival process is higher than the priority of the currently running process.

A non-preemptive priority algorithm will simply put the new process at the head of the ready queue.

A major problem with priority scheduling is indefinite blocking or starvation. A solution to the problem of indefinite blockage of the low-priority process is aging. Aging is a technique of gradually increasing the priority of processes that wait in the system for a long period of time.

Multilevel Queue Scheduling

A multilevel queue scheduling algorithm partitions the ready queue in several separate queues, in a multilevel queue, scheduling processes are permanently assigned to one queue. The processes are permanently assigned to one another, based on some property of the process, such as

Memory size Process priority Process type

Algorithm chooses the process from the occupied queue that has the highest priority, and runs that process either

Preemptive or Non-preemptively

Each queue has its own scheduling algorithm or policy. Possibility I If each queue has absolute priority over lower-priority queues then no process in the queue could run unless the queue for the highest-priority processes were all empty. Possibility II If there is a time slice between the queues then each queue gets a certain amount of CPU times, which it can then schedule among the processes in its queue. For instance;

80% of the CPU time to foreground queue using RR. 20% of the CPU time to background queue using FCFS.

Since processes do not move between queues so, this policy has the advantage of low scheduling overhead, but it is inflexible.

Multilevel Feedback Queue Scheduling

Multilevel feedback queue-scheduling algorithm allows a process to move between queues. It uses many ready queues and associates a different priority with each queue.The Algorithm chooses the process with highest priority from the occupied queue and runs that process either preemptively or non-preemptively. If the process uses too much CPU time it will moved to a lower-priority queue. Similarly, a process that wait too long in the lower-priority queue may be

moved to a higher-priority queue may be moved to a highest-priority queue. Note that this form of aging prevents starvation.

Interprocess Communication
Since processes frequently need to communicate with other processes, there is a need for a well-structured communication, without causing interrupts, among processes. Race Conditions In operating systems, processes that are working together share some common storage (main memory, file etc.) that each process can read and write. When two or more processes are reading or writing some shared data and the final result depends on who runs precisely and when, these are called race conditions. Concurrently executing threads that share data need to synchronize their operations and processing in order to avoid race condition on shared data. Only one customer thread at a time should be allowed to examine and update the shared variable. Race conditions are also possible in Operating Systems. If the ready queue is implemented as a linked list and if the ready queue is being manipulated during the handling of an interrupt, then interrupts must be disabled to prevent another interrupt before the first one completes. If interrupts are not disabled then the linked list could become corrupt.

Critical Section
How to avoid race conditions?

The key to preventing trouble involving shared storage is find some way to prohibit more than one process from reading and writing the shared data simultaneously. That part of the program where the shared memory is accessed is called the Critical Section. To avoid race conditions and flawed results, one must identify codes in Critical Sections in each thread. The characteristic properties of the code that form a Critical Section are Codes that reference one or more variables in a read-updatewrite fashion while any of those variables is possibly being altered by another thread. Codes that alter one or more variables that are possibly being referenced in read-update-write fashion by another thread. Codes use a data structure while any part of it is possibly being altered by another thread. Codes alter any part of a data structure while it is possibly in use by another thread.

Here, the important point is that when one process is executing shared modifiable data in its critical section, no other process is to be allowed to execute in its critical section. Thus, the execution of critical sections by the processes is mutually exclusive in time.

Mutual Exclusion
A way of making sure that if one process is using a shared modifiable data, the other processes will be excluded from doing the same thing. Formally, while one process executes the shared variable, all other processes desiring to do so at the same moment should be kept waiting; when that process has finished executing the shared variable, one of the processes waiting to do so should be allowed to proceed. In this fashion, each process executing the shared data (variables) excludes all others from doing so simultaneously. This is called Mutual Exclusion. Note that mutual exclusion needs to be enforced only when processes access shared modifiable data - when processes are performing operations that do not conflict with one another they should be allowed to proceed concurrently.

Mutual Exclusion Conditions If we could arrange matters such that no two processes were ever in their critical sections simultaneously, we could avoid race conditions. We need four conditions to hold to have a good solution for the critical section problem (mutual exclusion). No two processes may be at the same moment inside their critical sections. No assumptions are made about relative speeds of processes or number of CPUs. No process outside its critical section should block other processes. No process should wait arbitrary long to enter its critical section.

Proposals for Achieving Mutual Exclusion

The mutual exclusion problem is to devise a pre-protocol (or entry protocol) and a postprotocol (or exist protocol) to keep two or more threads from being in their critical sections at the same time.

Problem
When one process is updating shared modifiable data in its critical section, no other process should be allowed to enter in its critical section.

Proposal 1 -Disabling Interrupts (Hardware Solution)

Each process disables all interrupts just after entering in its critical section and re-enables all interrupts just before leaving critical section. With interrupts turned off, the CPU cannot not be switched to other processes. Hence, no other process will enter its critical and mutual exclusion achieved. Conclusion Disabling interrupts is sometimes a useful technique within the kernel of an operating system, but it is not appropriate as a general mutual exclusion mechanism for user processes. The reason is that it is unwise to give user processes the power to turn off interrupts.

Proposal 2 - Lock Variable (Software Solution)

In this solution, we consider a single, shared, (lock) variable, initially 0. When a process wants to enter in its critical section, it first tests the lock. If lock is 0, the process first sets it to 1 and then enters the critical section. If the lock is already 1, the process just waits until (lock) variable becomes 0. Thus, a 0 means that no process in its critical section, and 1 means hold your horses - some process is in its critical section. Conclusion The flaw in this proposal can be best explained by example. Suppose process A sees that the lock is 0. Before it can set the lock to 1 another process B is scheduled, runs, and sets the lock to 1. When the process A runs again, it will also set the lock to 1, and two processes will be in their critical section simultaneously.

Proposal 3 - Strict Alteration

In this proposed solution, the integer variable 'turn' keeps track of whose turn is to enter the critical section. Initially, process A inspects turn, finds it to be 0, and enters in its critical section. Process B also finds it to be 0 and sits in a loop continually testing 'turn' to see when it becomes 1.Continuously testing a variable waiting for some value to appear is called the BusyWaiting. Conclusion Taking turns is not a good idea when one of the processes is much slower than the other. Suppose process 0 finishes its critical section quickly, so both processes are now in their noncritical section. This situation violates above mentioned condition 3. Using Systems calls 'sleep' and 'wakeup' Basically, what above mentioned solution do is this: when a process wants to enter in its critical section, it checks to see if the entry is allowed. If it is not, the process goes into tight loop and waits (i.e., start busy waiting) until it is allowed to enter. This approach waste CPU-time. Now look at some interprocess communication primitives in the pair of sleep-wakeup.

Sleep

It is a system call that causes the caller to block, that is, be suspended until some other process wakes it up.
o

Wakeup
o

It is a system call that wakes up the process.

Both 'sleep' and 'wakeup' system calls have one parameter that represents a memory address used to match up 'sleeps' and wakeups. The Bounded Buffer Producers and Consumers The bounded buffer producers and consumers assumes that there is a fixed buffer size i.e., a finite number of slots are available. Statement To suspend the producers when the buffer is full, to suspend the consumers when the buffer is empty, and to make sure that only one process at a time manipulates a buffer so there are no race conditions or lost updates. As an example how sleep-wakeup system calls are used, consider the producer-consumer problem also known as bounded buffer problem. Two processes share a common, fixed-size (bounded) buffer. The producer puts information into the buffer and the consumer takes information out. Trouble arises when 1. The producer wants to put new data in the buffer, but buffer is already full. Solution: Producer goes to sleep and to be awakened when the consumer has removed data. 2. The consumer wants to remove data fromthe buffer but buffer is already empty. Solution: Consumer goes to sleep until the producer puts some data in buffer and wakes consumer up. Conclusion This approach also leads to same race conditions we have seen in earlier approaches. Race condition can occur due to the fact that access to 'count' is unconstrained. The essence of the problem is that a wakeup call, sent to a process that is not sleeping, is lost.

Semaphores

Definition A semaphore is a protected variable whose value can be accessed and altered only by the operations P and V and initialization operation called 'Semaphoiinitislize'. Binary Semaphores can assume only the value 0 or the value 1 counting semaphores also called general semaphores can assume only nonnegative values. The P (or wait or sleep or down) operation on semaphores S, written as P(S) or wait (S), operates as follows:

P(S): IF S > 0
THEN S := S - 1 ELSE (wait on S) The V (or signal or wakeup or up) operation on semaphore S, written as V(S) or signal (S), operates as follows:

V(S): IF (one or more process are waiting on S)

THEN (let one of these processes proceed) ELSE S := S +1 Operations P and V are done as single, indivisible, atomic action. It is guaranteed that once a semaphore operation has stared, no other process can access the semaphore until operation has completed. Mutual exclusion on the semaphore, S, is enforced within P(S) and V(S). If several processes attempt a P(S) simultaneously, only one process will be allowed to proceed. The other processes will be kept waiting, but the implementation of P and V guarantees that processes will not suffer indefinite postponement. Semaphores solve the lost-wakeup problem. Producer-Consumer Problem Using Semaphores The Solution to producer-consumer problem uses three semaphores, namely, full, empty and mutex. The semaphore 'full' is used for counting the number of slots in the buffer that are full. The 'empty' for counting the number of slots that are empty and semaphore 'mutex' to make sure that the producer and consumer do not access modifiable shared section of the buffer simultaneously. Initialization Set full buffer slots to 0. i.e., semaphore Full = 0. Set empty buffer slots to N. i.e., semaphore empty = N. For control access to critical section set mutex to 1. i.e., semaphore mutex = 1.

Producer ( ) WHILE (true) produce-Item ( ); P (empty); P (mutex); enter-Item ( ) V (mutex) V (full); Consumer ( ) WHILE (true) P (full) P (mutex); remove-Item ( ); V (mutex); V (empty); consume-Item (Item)

Deadlocks
A set of process is in a deadlock state if each process in the set is waiting for an event that can be caused by only another process in the set. In other words, each member of the set of deadlock processes is waiting for a resource that can be released only by a deadlock process. None of the processes can run, none of them can release any resources, and none of them can be awakened. It is important to note that the number of processes and the number and kinds of resources possessed and requested are unimportant. The resources may be either physical or logical. Examples of physical resources are Printers, Tape Drivers, Memory Space, and CPU Cycles. Examples of logical resources are Files, Semaphores, and Monitors. The simplest example of deadlock is where process 1 has been allocated non-shareable resources A, say, a tap drive, and process 2 has been allocated non-sharable resource B, say, a printer. Now, if it turns out that process 1 needs resource B (printer) to proceed and process 2 needs resource A (the tape drive) to proceed and these are the only two processes in the system, each is blocking the other and all useful work in the system stops. This situation is termed deadlock. The system is in deadlock state because each process holds a resource being requested by the other process neither process is willing to release the resource it holds. Preemptable and Non-preemptable Resources Resources come in two flavors: preemptable and nonpreemptable. A preemptable resource is one that can be taken away from the process with no ill effects. Memory is an example of a preemptable resource. On the other hand, a nonpreemptable resource is one that cannot be taken away from process (without causing ill effect). For example, CD resources are not preemptable at an arbitrary moment.

Reallocating resources can resolve deadlocks that involve preemptable resources. Deadlocks that involve nonpreemptable resources are difficult to deal with.

Necessary and Sufficient Deadlock Conditions

Coffman (1971) identified four (4) conditions that must hold simultaneously for there to be a deadlock. 1. Mutual Exclusion Condition The resources involved are non-shareable. Explanation: At least one resource (thread) must be held in a nonshareable mode, that is, only one process at a time claims exclusive control of the resource. If another process requests that resource, the requesting process must be delayed until the resource has been released. 2. Hold and Wait Condition Requesting process hold resources they already have, while waiting for requested resources. Explanation: There must exist a process that is holding a resource already allocated to it while waiting for additional resource that are currently being held by other processes. 3. No-Preemptive Condition Resources already allocated to a process cannot be preempted. Explanation: Resources cannot be removed from the processes until they are used to completion or released voluntarily by the process holding it. 4. Circular Wait Condition The processes in the system form a circular list or chain where each process in the list is waiting for a resource held by the next process in the list. As an example, consider the traffic deadlock in the following figure

Consider each section of the street as a resource. Mutual exclusion condition applies, since only one vehicle can be on a section of the street at a time.
1.

Hold-and-wait condition applies, since each vehicle is occupying a section of the street, and waiting to move on to the next section of the street. 3. No-preemptive condition applies, since a section of the street that is occupied by a vehicle cannot be taken away from it. 4. Circular wait condition applies, since each vehicle is waiting on the next vehicle to move. That is, each vehicle in the traffic is waiting for a section of street held by the next vehicle in the traffic.
2.

The simple rule to avoid traffic deadlock is that a vehicle should only enter an intersection if it is assured that it will not have to stop inside the intersection. It is not possible to have a deadlock involving only one single process. The deadlock involves a circular hold-and-wait condition between two or more processes, so one process cannot hold a resource, yet it will be waiting for another resource that it is holding. In addition, deadlock is not possible between two threads in a process, because it is the process that holds resources, not the thread that is, each thread has access to the resources held by the process.

Dealing with Deadlock Problem

In general, there are four strategies of dealing with deadlock problem: 1. The Ostrich Approach Just ignore the deadlock problem altogether. 2. Deadlock Detection and Recovery Detect deadlock and, when it occurs, take steps to recover. 3. Deadlock Avoidance Avoid deadlock by careful resource scheduling. 4. Deadlock Prevention Prevent deadlock by resource scheduling so as to negate at least one of the four conditions.

Now we consider each strategy in order of decreasing severity.

Deadlock Prevention
Havender in his pioneering work showed that since all four of the conditions are necessary for deadlock to occur, it follows that deadlock might be prevented by denying any one of the conditions.

Elimination of Mutual Exclusion Condition The mutual exclusion condition must hold for non-sharable resources. That is, several processes cannot simultaneously share a single resource. This condition is difficult to eliminate because some resources, such as the tap drive and printer, are inherently nonshareable. Note that shareable resources like read-only-file do not require mutually exclusive access and thus cannot be involved in deadlock. Elimination of Hold and Wait Condition There are two possibilities for elimination of the second condition. The first alternative is that a process request be granted all of the resources it needs at once, prior to execution. The second alternative is to disallow a process from requesting resources whenever it has previously allocated resources. This strategy requires that all of the resources a process will need must be requested at once. The system must grant resources on all or none basis. If the complete set of resources needed by a process is not currently available, then the process must wait until the complete set is available. While the process waits, however, it may not hold any resources. Thus the wait for condition is denied and deadlocks simply cannot occur. This strategy can lead to serious waste of resources. For example, a program requiring ten tap drives must request and receive all ten derives before it begins executing. If the program needs only one tap drive to begin execution and then does not need the remaining tap drives for several hours. Then substantial computer resources (9 tape drives) will sit idle for several hours. This strategy can cause indefinite postponement (starvation). Since not all the required resources may become available at once. Elimination of Non-preemption Condition The non-preemption condition can be alleviated by forcing a process waiting for a resource that cannot immediately be allocated to relinquish all of its currently held resources, so that other processes may use them to finish. Suppose a system does allow processes to hold resources while requesting additional resources. Consider what happens when a request cannot be satisfied. A process holds resources a second process may need in order to proceed while second process may hold the resources needed by the first process. This is a deadlock. This strategy requires that when a process that is holding some resources is denied a request for additional resources. The process must release its held resources and, if necessary, request them again together with additional resources. Implementation of this strategy denies the no-preemptive condition effectively. High Cost When a process releases resources the process may lose all its work to that point. One serious consequence of this strategy is the possibility of indefinite postponement (starvation). A process might be held off indefinitely as it repeatedly requests and releases the same resources.

Elimination of Circular Wait Condition The last condition, the circular wait, can be denied by imposing a total ordering on all of the resource types and then forcing all processes to request the resources in order (increasing or decreasing). This strategy impose a total ordering of all resources types, and requires that each process requests resources in a numerical order (increasing or decreasing) of enumeration. With this rule, the resource allocation graph can never have a cycle. For example, provide a global numbering of all the resources, as shown

1 2 3 4 5

Card reader Printer Plotter Tape drive Card punch Now the rule is this: processes can request resources whenever they want to, but all requests must be made in numerical order. A process may request first printer and then a tape drive (order: 2, 4), but it may not request first a plotter and then a printer (order: 3, 2). the problem with this strategy is that it may be impossible to find an ordering that satisfies everyone.

Deadlock Avoidance
This approach to the deadlock problem anticipates deadlock before it actually occurs. This approach employs an algorithm to access the possibility that deadlock could occur and acts accordingly. This method differs from deadlock prevention, which guarantees that deadlock cannot occur by denying one of the necessary conditions of deadlock. If the necessary conditions for a deadlock are in place, it is still possible to avoid deadlock by being careful when resources are allocated. Perhaps the most famous deadlock avoidance algorithm is the Bankers algorithm, so named because the process is analogous to that used by a banker in deciding if a loan can be safely made. Bankers Algorithm In this analogy Custome processes rs Units resources, say, tape drive Banker Operating System Custome Use Ma rs d x

A B C D

0 0 0 0
Fig. 1

6 5 4 7

Available Units = 10

In the above figure, we see four customers each of whom has been granted a number of credit units. The banker reserved only 10 units rather than 22 units to service them. At a certain moment, the situation becomes Custome rs A B C D Use d 1 1 2 4
Fig. 2

Ma x 6 Available Units = 5 2 4 7

Safe State The key to a state being safe is that there is at least one way for all users to finish. In other analogy, the state of figure 2 is safe because with 2 units left, the banker can delay any request except C's, thus letting C finish and release all four resources. With four units in hand, the banker can let either D or B have the necessary units and so on. Unsafe State Consider what would happen if a request from B for one more unit were granted in above figure 2. We would have following situation Custome rs A B C D Use d 1 2 2 4 Fig. 3 This is an unsafe state. If all the customers namely A, B, C, and D asked for their maximum loans, then banker could not satisfy any of them and we would have a deadlock. Ma x 6 Available Units = 5 1 4 7

Important Note: It is important to note that an unsafe state does not imply the existence or even the eventual existence of a deadlock. What an unsafe state does imply is simply that some unfortunate sequence of events might lead to a deadlock. The Banker's algorithm is thus to consider each request as it occurs, and see if granting it leads to a safe state. If it does, the request is granted, otherwise, it is postponed until later. Haberman [1969] has shown that executing the algorithm has complexity proportional to N2 where N is the number of processes and since the algorithm is executed each time a resource request occurs, the overhead is significant.

Deadlock Detection
Deadlock detection is the process of actually determining that a deadlock exists and identifying the processes and resources involved in the deadlock. The basic idea is to check allocation against resource availability for all possible allocation sequences to determine if the system is in deadlocked state. Of course, the deadlock detection algorithm is only half of this strategy. Once a deadlock is detected, there is supposed to be a way to recover, several alternatives exists: Temporarily prevent resources from deadlocked processes. Back off a process to some check point allowing preemption of a needed resource and restarting the process at the checkpoint later. Successively kill processes until the system is deadlock free.

These methods are expensive in the sense that each iteration calls the detection algorithm until the system proves to be deadlock free. Another potential problem is starvation; same process are killed repeatedly.

Introduction to Memory Management

Point of memory management algorithms - support sharing of main memory. We will focus on having multiple processes sharing the same physical memory. Key issues: o Protection. Must allow one process to protect its memory from access by other processes. o Naming. How do processes identify shared pieces of memory. o Transparency. How transparent is sharing. Does user program have to manage anything explicitly? o Efficiency. Any memory management strategy should not impose too much of a performance burden. Why share memory between processes? Because want to multiprogram the processor. To time share system, to overlap computation and I/O. So, must provide for multiple processes to be resident in physical memory at the same time. Processes must share the physical memory. Historical Development.

For first computers, loaded one program onto machine and it executed to completion. No sharing required. OS was just a subroutine library, and there was no protection. What addresses does program generate? o Desire to increase processor utilization in the face of long I/O delays drove the adoptation of multiprogramming. So, one process runs until it does I/O, then OS lets another process run. How do processes share memory? Alternatives: Load both processes into memory, then switch between them under OS control. Must relocate program when load it. Big Problem: Protection. A bug in one process can kill the other process. MS-DOS, MS-Windows use this strategy. Copy entire memory of process to disk when it does I/O, then copy back when it restarts. No need to relocate when load. Obvious performance problems. Early version of Unix did this. Do access checking on each memory reference. Give each program a piece of memory that it can access, and on every memory reference check that it stays within its address space. Typical mechanism: base and bounds registers. Where is check done? Answer: in hardware for speed. When OS runs process, loads the base and bounds registers for that process. Cray-1 did this. Note: there is now a translation process. Program generates virtual addresses that get translated into physical addresses. But, no longer have a protection problem: one process cannot access another's memory, because it is outside its address space. If it tries to access it, the hardware will generate an exception. End up with a model where physical memory of machine is dynamically allocated to processes as they enter and exit the system. Variety of allocation strategies: best fit, first fit, etc. All suffer from external fragmentation. In worst case, may have enough memory free to load a process, but can't use it because it is fragmented into little pieces. What if cannot find a space big enough to run a process? Either because of fragmentation or because physical memory is too small to hold all address spaces. Can compact and relocate processes (easy with base and bounds hardware, not so easy for direct physical address machines). Or, can swap a process out to disk then restore when space becomes available. In both cases incur copying overhead. When move process within memory, must copy between memory locations. When move to disk, must copy back and forth to disk. One way to avoid external fragmentation: allocate physical memory to processes in fixed size chunks called page frames. Present abstraction to application of a single linear address space. Inside machine, break address space of application up into fixed size
o

chunks called pages. Pages and page frames are same size. Store pages in page frames. When process generates an address, dynamically translate to the physical page frame which holds data for that page. So, a virtual address now consists of two pieces: a page number and an offset within that page. Page sizes are typically powers of 2; this simplifies extraction of page numbers and offsets. To access a piece of data at a given address, system automatically does the following: o Extracts page number. o Extracts offset. o Translate page number to physical page frame id. o Accesses data at offset in physical page frame. How does system perform translation? Simplest solution: use a page table. Page table is a linear array indexed by virtual page number that gives the physical page frame that contains that page. What is lookup process? o Extract page number. o Extract offset. o Check that page number is within address space of process. o Look up page number in page table. o Add offset to resulting physical page number o Access memory location. With paging, still have protection. One process cannot access a piece of physical memory unless its page table points to that physical page. So, if the page tables of two processes point to different physical pages, the processes cannot access each other's physical memory. Fixed size allocation of physical memory in page frames dramatically simplifies allocation algorithm. OS can just keep track of free and used pages and allocate free pages when a process needs memory. There is no fragmentation of physical memory into smaller and smaller allocatable chunks. But, are still pieces of memory that are unused. What happens if a program's address space does not end on a page boundary? Rest of page goes unused. This kind of memory loss is called internal fragmentation.

Introduction to Paging
Basic idea: allocate physical memory to processes in fixed size chunks called page frames. Present abstraction to application of a single linear address space. Inside machine, break address space of application up into fixed size chunks called pages. Pages and page frames are same size. Store pages in page frames. When process generates an address, dynamically translate to the physical page frame which holds data for that page.

So, a virtual address now consists of two pieces: a page number and an offset within that page. Page sizes are typically powers of 2; this simplifies extraction of page numbers and offsets. To access a piece of data at a given address, system automatically does the following: o Extracts page number. o Extracts offset. o Translate page number to physical page frame id. o Accesses data at offset in physical page frame. How does system perform translation? Simplest solution: use a page table. Page table is a linear array indexed by virtual page number that gives the physical page frame that contains that page. What is lookup process? o Extract page number. o Extract offset. o Check that page number is within address space of process. o Look up page number in page table. o Add offset to resulting physical page number o Access memory location.

Problem: for each memory access that processor generates, must now generate two physical memory accesses. Speed up the lookup problem with a cache. Store most recent page lookup values in TLB. TLB design options: fully associative, direct mapped, set associative, etc. Can make direct mapped larger for a given amount of circuit space. How does lookup work now? o Extract page number. o Extract offset. o Look up page number in TLB. o If there, add offset to physical page number and access memory location. o Otherwise, trap to OS. OS performs check, looks up physical page number, and loads translation into TLB. Restarts the instruction. Like any cache, TLB can work well, or it can work poorly. What is a good and bad case for a direct mapped TLB? What about fully associative TLBs, or set associative TLB? Fixed size allocation of physical memory in page frames dramatically simplifies allocation algorithm. OS can just keep track of free and used pages and allocate free pages when a process needs memory. There is no fragmentation of physical memory into smaller and smaller allocatable chunks. But, are still pieces of memory that are unused. What happens if a program's address space does not end on a page boundary? Rest of page goes unused. Book calls this internal fragmentation.

How do processes share memory? The OS makes their page tables point to the same physical page frames. Useful for fast interprocess communication mechanisms. This is very nice because it allows transparent sharing at speed. What about protection? There are a variety of protections: o Preventing one process from reading or writing another process' memory. o Preventing one process from reading another process' memory. o Preventing a process from reading or writing some of its own memory. o Preventing a process from reading some of its own memory.

How is this protection integrated into the above scheme? Preventing a process from reading or writing memory: OS refuses to establish a mapping from virtual address space to physical page frame containing the protected memory. When program attempts to access this memory, OS will typically generate a fault. If user process catches the fault, can take action to fix things up. Preventing a process from writing memory, but allowing a process to read memory. OS sets a write protect bit in the TLB entry. If process attempts to write the memory, OS generates a fault. But, reads go through just fine. Virtual Memory Introduction. When a segmented system needed more memory, it swapped segments out to disk and then swapped them back in again when necessary. Page based systems can do something similar on a page basis. Basic idea: when OS needs to a physical page frame to store a page, and there are none free, it can select one page and store it out to disk. It can then use the newly free page frame for the new page. Some pragmatic considerations: o In practice, it makes sense to keep a few free page frames. When number of free pages drops below this threshold, choose a page and store it out. This way, can overlap I/O required to store out a page with computation that uses the newly allocated page frame. o In practice the page frame size usually equals the disk block size. Why? o Do you need to allocate disk space for a virtual page before you swap it out? (Not if always keep one page frame free) Why did BSD do this? At some point OS must refuse to allocate a process more memory because has no swap space. When can this happen? (malloc, stack extension, new process creation).

When process tries to access paged out memory, OS must run off to the disk, find a free page frame, then read page back off of disk into the page frame and restart process. What is advantage of virtual memory/paging? o Can run programs whose virtual address space is larger than physical memory. In effect, one process shares physical memory with itself. o Can also flexibly share machine between processes whose total address space sizes exceed the physical memory size. o Supports a wide range of user-level stuff - See Li and Appel paper. Disadvantages of VM/paging: extra resource consumption. o Memory overhead for storing page tables. In extreme cases, page table may take up a significant portion of virtual memory. One Solution: page the page table. Others: go to a more complicated data structure for storing virtual to physical translations. o Translation overhead.

Issues in Paging and Virtual Memory

Page Table Structure. Where to store page tables, issues in page table design. In a real machine, page tables stored in physical memory. Several issues arise: o How much memory does the page table take up? o How to manage the page table memory. Contiguous allocation? Blocked allocation? What about paging the page table? On TLB misses, OS must access page tables. Issue: how the page table design affects the TLB miss penalty. Real operating systems provide the abstraction of sparse address spaces. Issue: how well does a particular page table design support sparse address spaces? Linear Page Table. o OS now faces variable-sized allocation problem for the page table storage. o Page table may occupy significant amount of physical memory. Page table for 32-bit address space with 4K byte pages has 232 / 212 = 220 entries. If each entry is 32 bits, need 4M bytes of memory to store page table. o Does not support sparse address spaces well - too many wasted entries. o TLB miss handler is very simple - just index the page table. Two-Level Page Table. Page Table itself is broken up into pages. An outer page table indexes pages of page table. Assuming 4K byte pages and 32 byte page table entries, each page can hold 212 / 4 =

210 entries. There are 10 bits left in virtual address for index of outer page table. Virtual address now has 10 bit outer page table index, 10 bit inner page table offset and 12 bit page offset. Page table lookup for two level page table. o Find physical page containing outer page table for process. o Extract top 10 bits of address. o Index outer page table using extracted bits to get physical page number of page table page. o Extract next 10 bits of address. o Index page table page using extracted bits to get physical page number of accessed page. o Extract final 12 bits of address - the page offset. o Index accessed physical page using 12 bit page offset. Evaluation of two level scheme. o Eliminates variable-sized allocation problem for page tables. Have one page for outer page table, and rest of page table is allocated in page-size chunks. o Have internal fragmentation - both for last page table page and for outer page table page. o If page table takes up too much memory, can page the pages of the page table. Question: is there anything that OS MUST keep in physical memory? o Supports sparse address spaces - can eliminate page table pages if corresponding parts of address space are not valid. o Increases TLB miss time. Have to perform two table lookups instead of one. Three level scheme. Like two level scheme, only with one more level. May become scheme of choice for machines with 64 bit address space. On such machines the outer page table can become much larger than one page. SPARC uses three level page tables. Primary job of page table: doing TLB reload. So why map entire address space? Instead, just maintain mappings for pages resident in physical memory. Inverted Page Table. Has one entry for each physical page frame specifying process that owns the page and the virtual address of page frame. On a TLB miss, search inverted page table data structure to find physical page frame for virtual address of process generating the access. Speed up the lookup by: o Hashing o Associative table of recently accessed entries. IBM machines (RS/6000, RT, System 38) and HP Spectrum machines use this scheme. What if accessed page is not in memory? Must look up the disk location in a data structure that looks much like a standard page

table. Since this data structure should not be accessed very often, it can be paged. All of these schemes have advantages and disadvantages. Which one should the hardware implement? Answer: hardware designer does not have to decide! Most modern machines handle TLB misses in software, so the OS can use whatever page table scheme it wants. Hardware only ``knows'' about the TLB. Sharing code and data. If two page table entries in different processes point to same physical page, the processes share the memory. If one process writes the data, other process will see the changes. Is a very efficient way to communicate. Can also share code. For example, only one copy of editor or compiler code can be kept in memory, and all editor or compiler processes can execute that one copy of the code. Helps memory utilization. Concept of reentrant code. Reentrant code cannot modify itself and must make sure that it has a separate copy of per-process global variables. All of your Nachos kernel code should be reentrant. Complications with sharing code: virtually indexed caches and inverted page tables. Virtual Memory. Basic idea: main memory used as a cache for backing store. Usual solution: demand paging. A page can be resident either on disk or in main memory. First extension for demand paging: the valid bit. Each page table or TLB entry has a valid bit. If the valid bit is set, the page is in physical memory. In a simple system, page is on disk if valid bit is not set. The operating system manages the transfer of pages to and from the backing store. It manages the valid bit and the page table setup. What does OS do on a page fault? o Trap to OS. o Save user registers and process state. o Determine that exeception was page fault. o Check that reference was legal and find page on disk. o Find a free page frame. o Issue read from disk to free page frame. Queue up for disk. Program disk controller to read page. Wait for seek and latency. Transfer page into memory. o As soon as program controller, allocate CPU to another process. Must schedule the CPU, restore process state. o Take disk transfer completed interrupt. o Save user registers and process state. o Determine that interrupt was a disk interrupt. o Find process and update page tables. o Reschedule CPU. o Restore process state and resume execution.

How well do systems perform under demand paging? Compute the effective access time. p is proportion of memory accesses that generate a page fault (0 <= p <= 1). What is effective acess time? If data in memory, between 10 and 200 nanoseconds, depending on if it is cached or not. Call it 100 nanoseconds for purposes of argument. Retrieving page from disk may take 25 milliseconds latency of 8 milliseconds, seek of 15 milliseconds and transfer of 1 millisecond. Add on OS time for page fault handling, and get 25 milliseconds. Effective access time = (1-p) * 100 + p * 25*106. If we want overall effective access time to be 110, less than one memory reference out of 2.5 * 106 can fault. In the future, difference between local accesses and faulting accesses will only get worse. In practice people simply do not run computations with a lot of paging - they just buy more memory or run smaller computations. Where to swap. Swap space - a part of disk dedicated to paging. Can usually make swap space accesses go faster than normal file accesses because avoid the overheads associated with a normal file system. On the other hand, using the file system immediately makes the paging system work with any device that the file system is implemented on. So, can page remotely on a diskless workstation using file system. May not always use backing store for all of process's data. o Executable code. Can just use executable file on disk as backing store. (Problem: recompilation). o Unreferenced pages in uninitialized data segment. Just zero the page on first access - no need to access backing store. Called zero on demand paging. To get a free page, may need to write a page out to backing store. When write a page out, need to clear the valid bit for the corresponding page table entry. A core map helps this process along. A core map records, for each physical page frame, which process and virtual page occupy that page frame. Core maps will be useful for other operations. When invalidate a page, must also clear the TLB to avoid having a stale entry cached. Page replacement algorithms. Which page to swap out? Two considerations: o A page that will not be accessed for a long time. o A clean page that does not have to be written back to the backing store. Hardware provides two bits to help the OS develop a reasonable page replacement policy. o Use bit. Set every time page accessed. o Dirty bit. Set every time page written.

Hardware with software-managed TLBs only set the bits in the TLB. So, TLB fault handlers must keep TLB entries coherent with page table entries when eject TLB entries. There is another way to synthesize these bits in software that makes TLB reload faster. How to evaluate a given policy? Consider how well it works with strings of page references. So, the string 1, 2, 3, 4, 1, 2, 5, 1, 2, 3, 4, 5 represents a sequence of references to pages 1, 2, 3, 4, 1, 2, etc. FIFO page replacement. When need to replace a page, choose the first page brought in. So, if we have three physical page frames, here is what happens for the above sequence: o Pages 1, 2, 3 brought in to memory. o Page 1 ejected, page 4 in - 2, 3, 4 in memory. o Page 2 ejected, page 1 in - 3, 4, 1 in memory. o Page 3 ejected, page 2 in - 4, 1, 2 in memory. o Page 4 ejected, page 5 in - 1, 2, 5 in memory. o Pages 1 and 2 accessed in memory. o Page 1 ejected, page 3 in - 2, 5, 3 in memory. o Page 2 ejected, page 4 in - 5, 3, 2 in memory. o Page 5 accessed in memory.

9 page faults total. What is disadvantage of FIFO? May eject a heavily used page. Belady's anomaly - adding more physical memory may actually make paging algorithm behave worse! Consider above example with four physical page frames. o Pages 1, 2, 3, 4 brought in to memory. o Pages 1 and 2 accessed in memory. o Page 1 ejected, page 5 in - 2, 3, 4, 5 in memory. o Page 2 ejected, page 1 in - 3, 4, 5, 1 in memory. o Page 3 ejected, page 2 in - 4, 5, 1, 2 in memory. o Page 4 ejected, page 3 in - 5, 1, 2, 3 in memory. o Page 5 ejected, page 4 in - 1, 2, 3, 4 in memory. o Page 1 ejected, page 5 in - 2, 3, 4, 5 in memory.

10 page faults total. with strings of page references. So, the string 1, 2, 3, 4, 1, 2, 5, 1, 2, 3, 4, 5 represents a sequence of references to pages 1, 2, LRU - eject least recently used page. Consider above example with four physical page frames. o Pages 1, 2, 3, 4 brought in to memory. o Pages 1 and 2 accessed in memory - 3, 4, 1 2 in memory. o Page 3 ejected, page 5 in - 4, 1, 2, 5 in memory. o Pages 1 and 2 accessed in memory. o Page 4 ejected, page 3 in - 5, 1, 2, 3 in memory. o Page 5 ejected, page 4 in - 1, 2, 3, 4 in memory.

Page 1 ejected, page 5 in - 2, 3, 4, 5 in memory.

8 page faults total.

How to implement LRU? Two strategies: o Build a clock and mark each page with the time every time it is accessed. o Move page to front of list every time it is accessed.

Both strategies are totally impractical on a modern computer overhead is too large. So, implement an approximation to LRU. One version is equivalent to LRU with a clock that ticks very slowly. So, many pages may be marked with the same time. Have a low-resolution LRU. Implement using use bits. Periodically, OS goes through all pages in physical memory and shifts the use bit into a history register for the page. It then clears the use bit. When read in a new page, clear page's history register. FIFO with second chance. Basically, use a FIFO page ordering. Keep a FIFO queue of pages to be paged out. But, if page at front of FIFO list has its use bit on when it is due to be paged out, clear the use bit and put page at end of FIFO queue. Can enhance FIFO with second chance to take into account four levels of page replacement desirability: o use = 0, dirty = 0: Best page to replace. o use = 0, dirty = 1: Next best - has not been recently used. o use = 1, dirty = 0: Next best - don't have to write out. o use = 1, dirty = 1: Worst.

Go through the FIFO list several times. Each time, look for next highest level. Stop when find first suitable page. Most paging algorithms try to free up pages in advance. So, don't have to write ejected page out when the fault actually happens. Keep a list of modified pages, and write pages out to disk whenever paging device is idle. Then clear the dirty bits. Increases probability that page will be clean when it is ejected, so don't have to write page out. Keep a pool of free frames, but remember which virtual pages they contain. If get a reference for one of these virtual pages, retrieve from the free frame pool. VAX/VMS uses this with a FIFO replacement policy - use bit didn't work on early versions of VAX! What if hardware does not implement use or dirty bits. Can the OS? Yes, if hardware has a valid and readonly bit.

Hardware traps if valid bit is not set in a page table or TLB entry. Hardware also traps if readonly bit is set and reference is a write. To implement a use bit, OS clears valid bit every time it clears use bit. OS keeps track of true state of page in different data structure. When the first reference to the page traps, OS figures out that page is really resident, and sets use and valid bits. To implement dirty bit, OS sets readonly bit on clean pages. OS keeps track of true state of page in different data structure. When the first write traps, OS sets dirty bit and clears readonly bit. Many systems use this scheme even if TLB has dirty and use bits - with this scheme, don't have to rewrite page table entries when eject an entry from TLB. Concept of a working set. Each process has a set of pages that it frequently accesses. For example, if the process is doing a grid relaxation algorithm, the working set would include the pages storing the grd and the pages storing the grid relaxation code. Working set may change over time. If the process finishes the relaxing one grid and starts relaxing another, the pages for the old grid drop out of the working set and the pages for the new grid come into the working set. Discussion so far focussed on running program. Two complications: loading a program and extending address space. Invariant: must reserve space in backing store for all pages of a running process. If don't, may get in a situation when need to eject a page, but backing store is full. When load a process, must reserve space in backing store. Note: do not have to actually write pages to backing store - can just load into memory. Makes startup time faster. What needs to be initialized? Only init data segment, in principle. o Can use zero on demand pages for uninit data segment. o Can use executable file to hold code.

Makes sense to preload much of code segment to make startup go faster. Of course, must be prepared to page even during startup what if init data segment does not fit in available physical memory? Must also allocate more backing store when extend address space via something like malloc. What is involved to allocate backing store pages? Just manipulate data structure in kernel memory that maintains state for backing store page allocation. Thrashing. A system thrashes if physical memory is too small to hold the working sets of all the processes running. The upshot of thrashing is that pages always spend their time waiting for the backing store to fetch their pages. Thrashing is a discrete phenonmenon. Usually, there is a phase transition from no thrashing to thrashing.

Typical way thrashing came about in early batch systems. Scheduler brought in new process whenever CPU utilization goes down. Eventually the size of the working sets become larger than physical memory, and processes start to page. The CPU utilization drops even further, so more processes come in, and the system starts to thrash. Throughput drops like crazy. Eliminating thrashing. Must drop degree of multiprogramming. So, swap all of a process out to backing store and suspend process. Come up with a Page Fault Frequency based solution. Basic idea: a process should have an ideal page fault frequency. If it faults too often, need to give it more physical page frames. If it faults too infrequently, it has too many physical page frames and need to take some from it and give to other process. When all processes fault too frequently, choose one to swap out. Another swapping algorithm - keep a free list threshold. When processes demand free pages at a high rate, free list will be consumed faster than pages can be swapped out to fill it. When amount of free memory goes above threshold, system starts swapping out processes. They start coming back in when free memory drops back below threshold. Page size. How big are current page sizes? 4K bytes or 8K bytes. Why aren't they bigger? o Tradition and existing code. o Increased fragmentation.

Why aren't they smaller? Smaller page size has more page faults for same working set size. o More TLB entries required. o Larger page tables required.
o

Page sizes have increased as memory has become cheaper and page faults more relatively expensive. Will probably increase in the future.

Pinning pages. Some pages cannot or should not be paged out. o There is some data that OS must always have resident to operate correctly. o Some memory accessed very frequently - disk buffers. o Sometimes DMA into memory - DMA device usually works with physical addresses, so must lock corresponding page into memory until DMA finishes.

Threats to Computer Security

Source: An Introduction to Computer Security: The NIST Handbook National Institute of Standards and Technology

Technology Administration U.S. Department of Commerce Special Publication 800-12

Computer systems are vulnerable to many threats that can inflict various types of damage resulting in significant losses. This damage can range from errors harming database integrity to fires destroying entire computer centers. Losses can stem, for example, from the actions of supposedly trusted employees defrauding a system, from outside hackers, or from careless data entry clerks. Precision in estimating computer security-related losses is not possible because many losses are never discovered, and others are "swept under the carpet" to avoid unfavorable publicity. The effects of various threats varies considerably: some affect the confidentiality or integrity of data while others affect the availability of a system. 1. Errors and Omissions Errors and omissions are an important threat to data and system integrity. These errors are caused not only by data entry clerks processing hundreds of transactions per day, but also by all types of users who create and edit data. Many programs, especially those designed by users for personal computers, lack quality control measures. However, even the most sophisticated programs cannot detect all types of input errors or omissions. A sound awareness and training program can help an organization reduce the number and severity of errors and omissions. Users, data entry clerks, system operators, and programmers frequently make errors that contribute directly or indirectly to security problems. In some cases, the error is the threat, such as a data entry error or a programming error that crashes a system. In other cases, the errors create vulnerabilities. Errors can occur during all phases of the systems life cycle. 2. Fraud and Theft Computer systems can be exploited for both fraud and theft both by "automating" traditional methods of fraud and by using new methods. For example, individuals may use a computer to skim small amounts of money from a large number of financial accounts, assuming that small discrepancies may not be investigated. Financial systems are not the only ones at risk. Systems that control access to any resource are targets (e.g., time and attendance systems, inventory systems, school grading systems, and long-distance telephone systems). Computer fraud and theft can be committed by insiders or outsiders. Insiders (i.e., authorized users of a system) are responsible for the majority of fraud. Since insiders have both access to and familiarity with the victim computer system (including what resources it controls and its flaws), authorized system users are in a better position to commit crimes. Insiders can be both general users (such as clerks) or technical staff members. An organization's former employees, with their knowledge of an organization's operations, may also pose a threat, particularly if their access is not terminated promptly. 3. Employee Sabotage Employees are most familiar with their employer's computers and applications, including knowing what actions might cause the most damage, mischief, or sabotage. The downsizing of organizations in both the public and private sectors has created a group of individuals with organizational knowledge, who may retain potential system access (e.g., if system accounts are

not deleted in a timely manner). The number of incidents of employee sabotage is believed to be much smaller than the instances of theft, but the cost of such incidents can be quite high. Common examples of computer-related employee sabotage include: destroying hardware or facilities, planting logic bombs that destroy programs or data, entering data incorrectly, "crashing" systems, deleting data, holding data hostage, and changing data. 4. Loss of Physical and Infrastructure Support The loss of supporting infrastructure includes power failures (outages, spikes, and brownouts), loss of communications, water outages and leaks, sewer problems, lack of transportation services, fire, flood, civil unrest, and strikes. 5. Malicious Hackers The term malicious hackers, sometimes called crackers, refers to those who break into computers without authorization. They can include both outsiders and insiders. Much of the rise of hacker activity is often attributed to increases in connectivity in both government and industry. One 1992 study of a particular Internet site (i.e., one computer system) found that hackers attempted to break in at least once every other day. The hacker threat should be considered in terms of past and potential future damage. Although current losses due to hacker attacks are significantly smaller than losses due to insider theft and sabotage, the hacker problem is widespread and serious. 6. Industrial Espionage Industrial espionage is the act of gathering proprietary data from private companies or the government for the purpose of aiding another company(ies). Industrial espionage can be perpetrated either by companies seeking to improve their competitive advantage or by governments seeking to aid their domestic industries. Foreign industrial espionage carried out by a government is often referred to as economic espionage. Since information is processed and stored on computer systems, computer security can help protect against such threats; it can do little, however, to reduce the threat of authorized employees selling that information.

7. Malicious Code Malicious code refers to viruses, worms, Trojan horses, logic bombs, and other "uninvited" software. Sometimes mistakenly associated only with personal computers, malicious code can attack other platforms. Actual costs attributed to the presence of malicious code have resulted primarily from system outages and staff time involved in repairing the systems. Nonetheless, these costs can be significant. Malicious Software: A Few Key Terms Virus: A code segment that replicates by attaching copies of itself to existing executables. The new copy of the virus is executed when a user executes the new host program. The virus may include an additional "payload" that triggers when specific conditions are met. For example, some viruses display a text string on a particular date. There are many types of viruses, including variants, overwriting, resident, stealth, and polymorphic. Trojan Horse: A program that performs a desired task, but that also includes unexpected (and undesirable) functions. Consider as an example an editing program for a multiuser system. This program could be modified to randomly delete one of the users' files each time they perform a useful function (editing), but the deletions are unexpected and definitely undesired! Worm: A self-replicating program that is self-contained and does not require a host program. The program creates a copy of itself and causes it to execute; no user intervention is required. Worms commonly use network services to propagate to other host systems. 8. Threats to Personal Privacy The accumulation of vast amounts of electronic information about individuals by governments, credit bureaus, and private companies, combined with the ability of computers to monitor, process, and aggregate large amounts of information about individuals have created a threat to individual privacy. The possibility that all of this information and technology may be able to be linked together has arisen as a specter of the modern information age. The most common Linux shell is named "Bash"

An A-Z Index of the Bash command line for Linux.

adduser Add a user to the system

addgroup Add a group to the system alias apropos Create an alias Search Help manual pages (man -k)

apt-get Search for and install software packages (Debian/Ubuntu) aptitude Search for and install software packages (Debian/Ubuntu)

aspell awk b

Spell Checker Find and Replace text, database sort/validate/index

basename Strip directory and suffix from filenames bash bc bg break builtin bzip2 c cal case cat cd cfdisk chgrp chmod chown chroot Display a calendar Conditionally perform a command Display the contents of a file Change Directory Partition table manipulator for Linux Change group ownership Change access permissions Change file owner and group Run a command with a different root directory GNU Bourne-Again SHell Arbitrary precision calculator language Send to background Exit from a loop Run a shell builtin Compress or decompress named file(s)

chkconfig System services (runlevel) cksum clear cmp comm Print CRC checksum and byte counts Clear terminal screen Compare two files Compare two sorted files line by line

command

Run a command - ignoring shell functions

continue Resume the next iteration of a loop cp cron crontab csplit cut d date dc dd records Display or change the date & time Desk Calculator Convert and copy a file, write disk headers, boot Copy one or more files to another location Daemon to execute scheduled commands Schedule a command to run at a later time Split a file into context-determined pieces Divide a file into several parts

ddrescue Data recovery tool declare df diff diff3 dig dir Declare variables and give them attributes Display free disk space Display the differences between two files Show differences among three files DNS lookup Briefly list directory contents

dircolors Colour setup for `ls' dirname dirs dmesg du e echo Display message on screen Convert a full pathname to just a path Display list of remembered directories Print kernel & driver messages Estimate file space usage

egrep Search file(s) for lines that match an extended expression eject enable env ethtool eval exec exit expect terminal expand export expr f false Do nothing, unsuccessfully Eject removable media Enable and disable builtin shell commands Environment variables Ethernet card settings Evaluate several commands/arguments Execute a command Exit the shell Automate arbitrary applications accessed over a Convert tabs to spaces Set an environment variable Evaluate expressions

fdformat Low-level format a floppy disk fdisk fg fgrep file find fmt fold for format Partition table manipulator for Linux Send job to foreground Search file(s) for lines that match a fixed string Determine file type Search for files that meet a desired criteria Reformat paragraph text Wrap text to fit a specified width. Expand words, and execute commands Format disks or tapes

free fsck ftp

Display memory usage File system consistency check and repair File Transfer Protocol

function Define Function Macros fuser g gawk getopts grep groups gzip h hash head help history Remember the full pathname of a name argument Output the first part of file(s) Display help for a built-in command Command History Find and Replace text within file(s) Parse positional parameters Search file(s) for lines that match a given pattern Print group names a user is in Compress or decompress named file(s) Identify/kill the process that is accessing a file

hostname Print or set system name i id if Print user and group id's Conditionally perform a command

ifconfig Configure a network interface ifdown ifup import file install Stop a network interface Start a network interface up Capture an X server screen and save the image to Copy files and set attributes

j jobs join k kill killall l less let ln local locate logname logout look lpc lpr lprint lprintd lprintq lprm ls lsof m make Recompile a group of programs Display output one screen at a time Perform arithmetic on shell variables Make links between files Create variables Find files Print current login name Exit a login shell Display lines beginning with a given string Line printer control program Off line print Print a file Abort a print job List the print queue Remove jobs from the print queue List information about file(s) List open files Stop a process from running Kill processes by name List active jobs Join lines on a common field

man mkdir mkfifo mkisofs mknod more mount mtools mtr mv mmv n netstat nice nl nohup

Help manual Create new folder(s) Make FIFOs (named pipes) Create an hybrid ISO9660/JOLIET/HFS filesystem Make block or character special files Display output one screen at a time Mount a file system Manipulate MS-DOS files Network diagnostics (traceroute/ping) Move or rename files or directories Mass Move and rename (files)

Networking information Set the priority of a command or job Number lines and write files Run a command immune to hangups Send desktop notifications

notify-send

nslookup Query Internet name servers interactively o open op p passwd paste pathchk Modify a user password Merge lines of files Check file name portability Open a file in its default application Operator access

ping pkill popd pr

Test a network connection Stop processes from running Restore the previous value of the current directory Prepare files for printing

printcap Printer capability database printenv Print environment variables printf ps pushd pwd q quota Display disk usage and limits Format and print data Process status Save and then change the current directory Print Working Directory

quotacheck Scan a file system for disk usage quotactl Set disk quotas r ram rcp read ram disk device Copy files between two machines Read a line from standard input

readarray Read from stdin into an array variable readonly Mark variables/functions as readonly reboot rename renice remsync return Reboot the system Rename files Alter priority of running processes Synchronize remote files via email Exit a shell function

rev rm rmdir rsync s screen scp sdiff sed select seq set sftp shift shopt

Reverse lines of a file Remove files Remove folder(s) Remote file copy (Synchronize file trees)

Multiplex terminal, run remote shells via ssh Secure copy (remote file copy) Merge two files interactively Stream Editor Accept keyboard input Print numeric sequences Manipulate shell variables and functions Secure File Transfer Program Shift positional parameters Shell Options

shutdown Shutdown or restart linux sleep slocate sort source split ssh strace su sudo Delay for a specified time Find files Sort text files Run commands from a file `.' Split a file into fixed-size pieces Secure Shell client (remote login program) Trace system calls and signals Substitute user identity Execute a command as another user

sum suspend symlink sync t tail tar tee test time times touch top

Print a checksum for a file Suspend execution of this shell Make a new name for a file Synchronize data on disk with memory

Output the last part of files Tape ARchiver Redirect output to multiple files Evaluate a conditional expression Measure Program running time User and system times Change file timestamps List processes running on the system

traceroute Trace Route to Host trap tr true tsort tty type u ulimit umask umount unalias Limit user resources Users file creation mask Unmount a device Remove an alias Run a command when a signal is set(bourne) Translate, squeeze, and/or delete characters Do nothing, successfully Topological sort Print filename of terminal on stdin Describe a command

uname

Print system information

unexpand Convert spaces to tabs uniq units unset unshar until useradd usermod users Uniquify files Convert units from one scale to another Remove variable or function names Unpack shell archive scripts Execute commands (until error) Create new user account Modify user account List users currently logged in

uuencode Encode a binary file uudecode Decode a file created by uuencode v v vdir vi vmstat w watch wc Execute/display a program periodically Print byte, word, and line counts Verbosely list directory contents (`ls -l -b') Verbosely list directory contents (`ls -l -b') Text Editor Report virtual memory statistics

whereis Search the user's $path, man pages and source files for a program which while who whoami Search the user's $path for a program file Execute commands Print all usernames currently logged in Print the current user id and name (`id -un')

Wget write x xargs list(s)

Retrieve web pages or files via HTTP, HTTPS or FTP Send a message to another user

Execute utility, passing constructed argument

xdg-open Open a file or URL in the user's preferred application. yes . ### Print a string until interrupted Run a command script in the current shell Comment / Remark

Commands marked are bash built-ins, these are available under all shells.

bash syntax
Parameters Variables Arrays Permissions Redirection Pipes for another Looping constructs: if-then-else Conditionally perform a command for until while Expand words, and execute commands Execute commands (until error) Execute commands Shell parameters Environment variables Array Variables Allow or Deny actions Spooling to and from files Redirect the output from one command as input

The break and continue builtins may be used to control loop execution. Evaluating expressions: Using brackets to Group and expand expressions

Conditional Execution Chain commands to run one after another (AND/OR) Conditional expressions Arithmetic expressions Shell expansion and Pattern matching (Wildcards) Escape Chars, delimiters and Quotes Command Substitution Working with the bash Shell: Run a bash shell script vi editor Here docs Keyboard Job Control Prompt ### A one page reference to the vi editor Here documents Cursor control - Cut & paste Suspend and resume a process Prompt variable Comment / Remark

BashBangSplat Pronunciation guide for unix Note that wherever a `;' appears in the description of a command's syntax, it may be replaced with one or more newlines.