Precedence Constrained Task Scheduling in LITMUSRT

A Project Report Submitted in Partial Fulllment of Requirements for the Degree of

Bachelor of Technology

by Ashish Prasad(P2008CS1004) Digvijay Singh(P2008CS1110) Shashank Sharma(P2008CS1031) Supervisor Dr. Nitin Auluck

Department of Computer Science & Engineering Indian Institute of Technology Ropar Rupnagar 140001, India April 2012

Abstract

Massively multi-core processors are rapidly gaining market share with major chip vendors oering an ever-increasing number of cores per processor. From a programming perspective, the sequential programming model does not scale very well for such multi-core systems. The LITMUSRT patch is a (soft) real-time extension of the Linux kernel with a focus on multiprocessor real-time scheduling and synchronization. The Linux kernel is modied to support the sporadic task model and modular scheduler plugins. Clustered, partitioned, and global scheduling are included, and semi-partitioned scheduling is supported as well. The primary purpose of the LITMUSRT project is to provide a useful experimental platform for applied real-time systems research. However LITMUS RT does not cater to dependent real time task models (precedent graphs). The aim of our B.Tech project is to schedule the parallel real time dependent task model in Linux which will be an extension to the present LITMUSRT model. In this project, we have implemented a data structure of precedent constraints in our task model (Directed Acyclic Graph) which we use to transfer the precedence constraints from the user space to kernel space via syscall (which we added).In the kernel side the populated data structure is used for proper scheduling of real time constraint task via Global Synchronized Earliest Deadline First Algorithm (GSN-EDF).

Acknowledgements

It is our privilege to express our sincerest regards to our project coordinator, Mr.Nitin Auluck for his valuable inputs and able guidance. We would also like to express our sincere thanks to Mr Jagpreet Singh for his encouragement, whole-hearted cooperation and constructive criticism throughout the project

Honor Code
We certify that we have properly cited any material taken from other sources and have obtained permission for any copyrighted material included in this report. We take full responsibility for any code submitted as part of this project and the contents of this report.

Ashish Prasad(P2008CS1004)

Digvijay Singh(P2008CS1110)

Shashank Sharma(P2008CS1031)

iii

Certicate

It is certied that the B. Tech. project Title of the Project has been done by Author-1(Entry-No), Author-2 (Entry-No) under my supervision. This report has been submitted towards partial fulllment of B. Tech. project requirements.

Supervisors Name Project Supervisor Department of Computer Science & Engineering Indian Institute of Technology Ropar Rupnagar-140001

Contents
Nomenclature 1 Introduction 2 What is LITMUSRT 2.1 2.2 2.3 2.4 2.5 About . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Non-Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Release Version used by Current us:LITMUSRT Version 2011.1 . . . . . . . . . . Version:LITMUSRT Version 2012.1 . . . . . . . . . . . . . . . . vii 1 2 2 2 3 3 3 4 6 7 7 9 9 10 11 12 12 12 12 13 13

3 GSN-EDF 3.1 3.2 3.3 GSN-EDF Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Theorem1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Theorem2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4 TASK MODEL 4.1 4.2 Directed Acyclic Graph . . . . . . . . . . . . . . . . . . . . . . . . . . . Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5 DATA STRUCTURE 5.1 5.2 5.3 5.4 Level-1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Level-2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Level-3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . List of Schedulable Task . . . . . . . . . . . . . . . . . . . . . . . . . . .

6 User Space 6.1 Rt Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

CONTENTS

6.2 6.3 6.4

Set Constraint Thread . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conditions Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Deletion of Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

13 15 15 16 18 20 21 21 22 23 24

7 SYSTEM CALLS 7.1 7.2 7.3 7.4 Syscall: long sys init dep subtask (pid t subtask pid) . . . . . . . . . . . Syscall: long sys exit dep task (pid t main task pid) . . . . . . . . . . .

Syscall: long sys set main task pid(pid t subtask pid, pid t main task pid) 16 Syscall:long sys add parent to subtask(pid t parent pid, pid t subtask pid) 19

8 KERNEL SPACE 8.1 8.2 Dependent Schedulable Subtasks List . . . . . . . . . . . . . . . . . . . . Release Wait Queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

9 Conclusions Appendix A

vi

List of Figures
3.1 3.2 4.1 5.1 6.1 7.1 7.2 7.3 7.4 8.1 8.2 8.3 WHY-GSN-EDF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . GSN-EDF ADVANTAGE . . . . . . . . . . . . . . . . . . . . . . . . . . TASK MODEL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . DATA STRUCTURE . . . . . . . . . . . . . . . . . . . . . . . . . . . . USER SPACE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 8 9 11 14 17 18 19 20 21 22 22 24

SYSCALL-1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . SYSCALL-2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . SYSCALL-3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . SYSCALL-4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . OBTAINING THE SCHEDULABLE SUBTASK LIST . . . . . . . . . . ADD TO SUBTASK WAIT LIST . . . . . . . . . . . . . . . . . . . . . . JOB EXIT AND TRANSFER OF SCHEDULABLE SUBTASK FROM WAIT TO READY QUEUE . . . . . . . . . . . . . . . . . . . . . . . .

TASK GRAPH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

vii

Chapter 1

Introduction
LITMUSRT currently supports scheduling of independent real time tasks in linux via dierent scheduler plugins.The task model which we have used to express precedence constraints is in the form of Directed Acyclic Graph .We have modeled our data structure in the form of a multilevel linked list storing parent child relationship corresponding to each task in the kernel. To begin with , user species the tasks and the precedence constraints among them in the user space. To communicate this information between kernel and user space we have added 4 new syscall . The rst one is used to initialize these tasks in the data structure while the rest are used to transfer constraint information to the kernel and the subsequent deletion of task from the data structure. In the kernel side we use the populated data structure to obtain the list of schedulable task satisfying the given constraints. Details of each process will be specied later.

Chapter 2

What is LITMUSRT
LITMUSRT stands for Linux Testbed for Multiprocessor Scheduling in Real Time Systems.

2.1

About

The LITMUSRT patch is a (soft) real-time extension of the Linux kernel with a focus on multiprocessor real-time scheduling and synchronization. The Linux kernel is modied to support the sporadic task model and modular scheduler plugins. Clustered, partitioned, and global scheduling are included, and semi-partitioned scheduling is supported as well.

2.2

Goals

The primary purpose of the LITMUSRT project is to provide a useful experimental platform for applied real-time systems research. To that end, LITMUSRT provides abstractions and interfaces within the kernel that simplify the prototyping of multiprocessor real-time scheduling and synchronization algorithms (compared to modifying a vanilla Linux kernel). As a secondary goal, LITMUSRT serves as a proof of concept, showing how algorithms such as Pfair schedulers can be implemented on current hardware. Finally, we hope that parts of LITMUSRT and the lessons learned may nd value as blueprints/sources of inspiration for other implementation eorts (both commercial and open source).

2.3

Non-Goals
Furthermore,

LITMUSRT is a research prototype that is reasonably stable and tested, but there are currently no plans to turn it into a production-quality system. LITMUSRT s API is not stable, that is, interfaces and implementations may change

without warning between releases. POSIX-compliance is not a goal; the LITMUSRT API oers alternate system call interfaces. While we aim to follow Linux-coding standards, LITMUSRT is not targeted at being merged into mainline Linux. Rather, we hope that some of the ideas protoyped in LITMUSRT may eventually nd adoption in Linux or other kernels.

2.4

Release Version used by us:LITMUSRT Version 2011.1

Based on Linux 2.6.36. Released in January 2011. Major changes (since LITMUS-RT 2010.2): Rebased LITMUSRT from Linux 2.6.34 to Linux 2.6.36. Added support for the ARM architecture (tested on a PB11MPCore baseboard with a four-core ARM11 MPCore CPU). Feather-Trace devices are now allocated dynamically and are properly registered with sysfs. This avoids bugs due to major device number collisions and removes the need for manual device node creation (on a system with standard udev rules). Improved debug tracing output and made trace buer size congurable. Various bug xes concerning C-EDF cluster size changes. The cluster size can now be congured with the le /proc/litmus/plugins/C-EDF/cluster. Various KCong cleanups and improvements. Dropped SCons as the build system for liblitmus and reverted to makeles. Added scope and TAG le generation to liblitmus. st trace can now be controlled with signals (part of ft tools).

2.5

Current Version:LITMUSRT Version 2012.1

The current release of LITMUSRT is 2012.1. It consists of our Linux kernel modications in the form of a patch against Linux 3.0 and liblitmus, the user-space API for realtime tasks, as well as ft tools, a collection of tools used for tracing with Feather-Trace (which is part of the LITMUSRT patch). Major changes (since LITMUS-RT 2011.1): Rebased LITMUSRT from Linux 2.6.36 to Linux 3.0. Added cache-anity aware migrations to GSN-EDF and C-EDF. PFAIR and C-EDF now support the release-master feature. Feather-Trace can now measure interrupt interference. PFAIR now supports sporadic task releases. PFAIR now implements early-releasing of subtasks.

Chapter 3

GSN-EDF
The kernel contains the following real-time scheduling policy implementations: PFAIR, an implementation of the PD2 algorithm, PSN-EDF, a partitioned EDF (P-EDF) implementation with support for the real-time locking protocols, GSN-EDF, a global EDF (G-EDF) implementation with support for real-time locking protocols, C-EDF (Clustered EDF), a hybrid of G-EDF and P-EDF, and Linux, a placeholder policy that disables all real-time functionality added by the LITMUSRT patch. Only one policy can be active at any time. Initially (i.e., during and after boot), the Linux policy is active. You can use the tool showsched (part of liblitmus) to display the name of the currently active policy. Here we will talk about GSN-EDF,because our implementation is based on this algorithm.Figure 3.1 shows the situation before GSNEDF is implemented and Figure 3.2 shows the situation after GSN-EDF is implemented.

Figure 3.1: WHY-GSN-EDF

3.1

GSN-EDF Algorithm

The standard G-EDF scheduling algorithm assumes that jobs are preemptable at all times. In this section, we present a variant of G-EDF that guarantees that a job Tij is only blocked by another non-preemptable job when Tij is either released or resumed, and that such blocking durations are reasonably constrained. For globally-scheduled systems, we say that a Tij is non-preemptively blocked at time t i Tij is one of the m highest-priority runnable jobs and it is not scheduled at t because a lower-priority non-preemptable job is scheduled instead. Before continuing, consider the following na ve modication to the G-EDF algorithm that allows jobs to have non-preemptable sections: At time t, if there are q nonpreemptable pending jobs, then these jobs are scheduled at t. If there are k additional preemptable jobs at t, then the min(k, m q) highest-priority such jobs are also scheduled at t. The problem with the above algorithm is that it is possible for a job Tij to be non-preemptively blocked whenever other jobs are released or resumed. For example, consider the schedule depicted in Fig. 3.1. Even though T1 is always among the two highest-priority runnable jobs (while it is pending), it becomes non-preemptively blocked whenever a higher-priority job arrives, because the lowest-priority scheduled jobs, T2 and T4 , are non-preemptable. As a result, T1 is non-preemptively blocked for three time units, even though the maximum amount of time that any job is nonpreemptable is 2.5 time units. In order to avoid this behavior, we introduce the G-EDF algorithm for suspendable and non-preemptable jobs (GSN-EDF). Under GSN-EDF, a runnable job is either linked to a processor or unlinked. A job Tij is linked at time t i G-EDF would schedule Tij on a processor at t (under the assumption that all jobs are fully preemptable). Thus, at any time t, at which there are at least m runnable jobs, the m highest-priority runnable jobs are linked to processors. Intuitively, if a job Tij is linked to but not scheduled on a processor, then Tij is non-preemptively blocked. Additionally, if a job Ta is scheduled on a processor but is unlinked, then Ta is not one of the m-highest priority runnable tasks and is only scheduled at time t because it is non-preemptable. As an example, consider the two-processor system in Fig. 3.2,which depicts the same system as in Fig. 3.1 except that it is scheduled by GSN-EDF. When T3 is released at time 1, it becomes linked to Processor 2 since its previously linked job, T2 ,had the lowest priority of any linked job. However, since T2 is non-preemptable at time 1, T3 is not scheduled until T2 becomes preemptable, at time 2.5. Thus, over the range [1,

2.5),T2 is non-preemptively blocked. Notice that, in Fig. 3.2, the only time a job is non-preemptively blocked is when it is released, and that the amount of time a job is non-preemptively blocked is upper-bounded by the maximum duration of time any job can be non-preemptable. Comparatively, a job in the na ve modication of G-EDF, considered earlier, can be non-preemptively blocked whenever other jobs are released (e.g., T1 , in Fig. 3.1,is non-preemptively blocked over the time ranges [1, 2.5) and [4, 5.5)) and a job can incur non-preemptive blocking larger than the maximum duration of time a job is non-preemptable. Although it is not depicted, under GSN-EDF, a job may be non-preemptively blocked when it is resumed for a duration of time that is upper-bounded by the longest time any job is non-preemptable. We now state two theorems concerning GSN-EDF that can be used to bound nonpreemptive blocking times. The proofs of these theorems are straightforward, and have been omitted due to space constraints.

3.2

Theorem1

Under GSN-EDF, if a pending job Tij is linked but not scheduled at time t, then it has been linked but not scheduled continuously over the interval [tR , t), where tR denotes the last time when Tij was resumed or released.

3.3

Theorem2

Let Tij be the job linked to Processor k at time t and assume Tij is not scheduled at time t. Let tD be the maximal amount of time a job Ta , where Y(Ta ) Y(Tij ), that is scheduled at time t on Processor k executes non-preemptively.Under GSN-EDF, Tij is either scheduled by time t + tD or becomes unlinked by time t + tD.

Figure 3.2: GSN-EDF ADVANTAGE

Chapter 4

TASK MODEL
Our task model is based on Directed Acyclic Graphs.

4.1

Directed Acyclic Graph

A Directed acyclic graph (DAG), is a directed graph with no directed cycles. That is, it is formed by a collection of vertices and directed edges, each edge connecting one vertex to another, such that there is no way to start at some vertex v and follow a sequence of edges that eventually loops back to v again. Each node represents a task in our task model and parent child relationship is shown via a directed edge between the nodes.

Figure 4.1: TASK MODEL

4.2

Constraints

Each directed edge in the task model represents a constraintthe execution of task represented by the tail node should be completed before the execution of the task represented by head node can start.

10

Chapter 5

DATA STRUCTURE
Our Data Structure basically consists of three levels of linked lists

Figure 5.1: DATA STRUCTURE

11

5.1

Level-1

Each node of the linked list present in this level stores the information about the Task, as well as a pointer to the list containing the subtask of this task. In addition to this a pointer is maintained to the reference subtask list, which is used to reproduce this entire data structure for the next iteration. Whenever we delete a subtask node from the subtask list, we insert a copy of that subtask node in this reference subtask list. Hence when the subtask list becomes empty, the reference subtask list now becomes the original subtask list. A swap of pointers leads us to our initial data structure before beginning our next iteration.

5.2

Level-2

Each node of the linked list at this level stores the following information: 1.The subtask This contains the task structure information of the subtask of the corresponding task. 2.Pointer to the parent list of this subtask The parent list contains a list of all the subtasks whose execution should be completed prior to the execution of this subtask. 3.pointer to the Reference subtask list This subtask list is used to keep our data structure intact at the end of each iteration. Whenever we delete a subtask from the parent list, we insert a copy of that subtask in this reference subtask list. Hence when the parent list becomes empty, the reference parent list now becomes the original parent list. A swap of pointers leads us to our initial data structure before beginning our next iteration.

5.3

Level-3

At this level we have a linked list of parent subtasks corresponding to each subtask.

5.4

List of Schedulable Task

The list of schedulable tasks consists of all those tasks whose parent list is empty, hence it can be now scheduled as the precedence constraint is satised.

12

Chapter 6

User Space
6.1 Rt Threads

In the user space, we create multiple threads. Each thread via the inbuilt system call (getid()), obtains its PID and this PID is used by our syscall to initialize a subtask in the data structure.

6.2

Set Constraint Thread

Apart from the above multiple threads, we created a dummy thread which executes after all the threads are blocked. This thread is used to specify the constraints among dierent threads using their PIDs (already obtained).

13

Figure 6.1: USER SPACE

14

6.3

Conditions Variables

Dummy thread waits on a lock until all the threads are blocked. To get the information about the number of threads blocked we maintain a global variable count (set to the number of threads),and put a lock on it. Each thread after getting its PID decrements this global counter and blocks itself. The Dummy thread gets blocked on the lock till the time the value of the global count variable becomes zero. After this, the thread is activated and it is used to specify the constraints via our syscall.This syscall helps in creation of the parent list of each subtask in our data structure in kernel side.

6.4

Deletion of Threads

Once the task gets completed, the corresponding subtasks are removed from the data structure at kernel side via our dened syscall.

15

Chapter 7

SYSTEM CALLS
To provide an interface between the kernel and the user side, we have implemented 4 new syscalls.They are as follows:

7.1

Syscall: long sys set main task pid(pid t subtask pid, pid t main task pid)

In the structure of each subtask, we have added a new variable main task pid to store the value of the task it belongs to. This syscall,after receiving the PID of subtask and main task from the user side nds the subtask with the subtask PID and sets its Main Task Pid to main task pid on the kernel side. To ensure atomic access to the data structure in the kernel side, we have used a task list lock so the sisal prior to changing the PID variable of the subtask, acquires the read lock and releases it after making the changes.

16

Figure 7.1: SYSCALL-1

17

7.2

Syscall: long sys init dep subtask (pid t subtask pid)

After a thread acquired PID in the user space, this syscall is used to insert the corresponding subtask in the data structure. After acquiring the read lock, as in the previous syscall, it rst searches for the subtask with the corresponding PID and then calls the function addSubtaskToDepTaskList() to insert subtask in the data structure. After this is done, the lock is released.

Figure 7.2: SYSCALL-2

18

7.3

Syscall:long sys add parent to subtask(pid t parent pid, pid t subtask pid)

This is used to transport the constraint information to the kernel side. The parent task with the given parent id is inserted in the parent list of the subtask with the corresponding subtask pid.As done in the above syscalls readlock is acquired and released before and after the execution of the function-addParentToSubtaskInDepTaskList(parent, subtask).

Figure 7.3: SYSCALL-3

19

7.4

Syscall: long sys exit dep task (pid t main task pid)

This syscall is used to delete the corresponding subtask from the data structure in the kernel side.

Figure 7.4: SYSCALL-4

20

Chapter 8

KERNEL SPACE
In the kernel side, we implemented a linked list to keep a record of all the real time tasks in the release queue. From our data structure we obtain a schedulable list of tasks (whose parent list is empty).When a task to be scheduled is to be moved from the release to the ready queue, its occurrence is checked in the schedulable task list .If it occurs then it passed to the ready requeue for scheduling. Else it remains in the release queue. When a task is executed, we regenerate a schedulable task list and each task in this list is searched in our linked list (keeping a record of all real time tasks in the release queue).If it occurs in this list, it means it is eligible for execution. Hence it is passed to the ready queue.

8.1

Dependent Schedulable Subtasks List

Figure 8.1: OBTAINING THE SCHEDULABLE SUBTASK LIST

21

8.2

Release Wait Queue

Figure 8.2: ADD TO SUBTASK WAIT LIST

Figure 8.3: JOB EXIT AND TRANSFER OF SCHEDULABLE SUBTASK FROM WAIT TO READY QUEUE

22

Chapter 9

Conclusions
Through appropriate changes in the liblitmus (user side), litmus (kernel) and new system calls, we accomplished our goal of scheduling real time dependent tasks. As a part of our project, we went through the code of litmus and understood the ow of the multithreaded Linux kernel scheduling. Understanding the execution ow in the kernel was a very demanding job. There was very little (if any) documentation on litmus when we started the work. We rst familiarized ourselves with the concepts of Linux scheduler and then looked at the scheduling design of litmus. Once, we had an idea of the litmus scheduler, we went ahead to attack the problem of dependent task scheduling. We had a rst hand experience in working in Linux kernel and got an opportunity to design and implement the changes in litmus scheduler. We utilized all our expertise of operating system concepts to deal with the challenges of working in Linux kernel. We came across shared resources, race conditions and thread blocking and made use of condition variables and locks to overcome them. Applying the concepts that we studied as a part of our Operating System curriculum was a delightful and enriching experience. We introduced our changes without breaking the existing ow of litmus. Our changes allowed the earlier scheduling of independent tasks to be scheduled in tandem with the dependent task scheduling using the GSN-EDF scheduler plugin. We enabled users to dene the various tasks and specify the constraints using liblitmus API and leave it to scheduler to schedule the tasks in the expected manner. Though, there are some active issues like kernel panic and extension of dependent task scheduling to periodic dependent task scheduling - we have already provided API, but not tested it - that need to be resolved in our implementation, we hope that the work will be undertaken by the upcoming batches and a working patch of our scheduler can be submitted to the LitmusRT people.

23

Appendix A
This is the constraint graph

Figure 1: TASK GRAPH

24

Here is the output le obtained after scheduling of our dependent task which shows the constraint order followed by dierent subtasks. Main thread PID: 2484 RT Thread 0 active. Thread id: 0 PID: 2486 RT Thread 1 active. Thread id: 1 PID: 2487 RT Thread 2 active. Thread id: 2 PID: 2488 RT Thread 3 active. Thread id: 3 PID: 2489 RT Thread 5 active. RT Thread 4 active. Thread id: 5 PID: 2491 Thread id: 4 PID: 2490 RT Thread 6 active. Thread id: 6 PID: 2492 RT Thread 7 active. RT Thread 8 active. Thread id: 8 PID: 2494 Thread id: 7 PID: 2493 RT Thread 9 active. Thread id: 9 PID: 2495 Executing job: (3, 2489). Executing job: (5, 2491). Executing job: (4, 2490). Executing job: (2, 2488). Executing job: (1, 2487). Executing job: (8, 2494). Executing job: (6, 2492). Executing job: (7, 2493). Executing job: (9, 2495). Executing job: (0, 2486).

25

References
Bjorn B. Brandenburg Aaron Block, Hennadiy Leontyev and James H. Anderson. A exible real-time locking protocol for multiprocessors. Xiao Qin & Hong Jiang. Dynamic, reliability-driven scheduling of parallel real-time jobs in heterogeneous systems. Marcocesati and Daniel P. Bovet. Understanding the Linux Kernel. OReilly Media.

26