PRIORITY INVERSION PROBLEM

RAVINDRA V. JOSHI Click to edit Master subtitle style

8/5/12

MARS PATH FINDER

8/5/12

Description of the Problem occurred

then

Path Finder was getting reset every now and System appeared to be unstable Crucial data was getting lost as system was

getting reset
No clue what was the problem, as there was

no test report suggested this behavior!!!

How to analyze? How to debug?? How to

correct???

8/5/12

Analysis Of the Problem

Information bus High Priority Task Data Gathering Task Low Priority Task Information Bus Task was getting reset

continuously Problem

Finally got zeroed-on to Priority Inversion

8/5/12

PROCESSES, TASKS, AND THREADS

An instance of Program in Execution For simplicitys sake let us use the term, tasks
PROCESSE S TASKS

8/5/12

TASKS AND RESOURCES

A typical system contains a set of tasks and

set of resources

8/5/12

SCHEDULING ART OF ASSIGNING RESOURCES TO FIFO TASKS

Shortest Job First Priority based scheduling Round Robin Pre-emptive and Non-PreEmptive ( Co-

operative )
Longest Waiting First

8/5/12

Priority Based Pre-Emptive Scheduling

8/5/12

Priority Based Pre-Emptive Scheduling

8/5/12

Priority Based Pre-Emptive Scheduling

8/5/12

Priority Based Pre-Emptive Scheduling

8/5/12

Locking a Resource

8/5/12

DESCRIPTION OF PRIORITY INVERSION

Consider 3 Tasks with Priority J<Q<K and a

resource JOKER which is required by J and K but not by Q

8/5/12

DESCRIPTION OF PRIORITY INVERSION

Currently J is scheduled and it locks JOKER

8/5/12

DESCRIPTION OF PRIORITY INVERSION

does not require JOKER

Now Q becomes Ready and Pre Empts J as it

8/5/12

DESCRIPTION OF PRIORITY INVERSION

is waiting. This is PRIORITY INVERSION.

Q got scheduled though K with higher priority

8/5/12

Solution
Priority Inheritance = Current Highest Priority Priority Ceiling = Statically scheduled

8/5/12

Debugging and Correcting the problem

by collected Traces and Logs
At last, the problem got reproduced when

Problem was tried for reproduction on ground

everybody gave up and went home except one engineer who finally saw the reproduction
It turned out that VxWorks( the operating

system inside Path Finder ) had implemented a solution ( with Priority Inheritance ) but that function was called with parameter false
A short patch overwriting the parameter was
8/5/12 beamed

to Mars and Program was Corrected

LESSONS LEARNT
Never leave distant machine in Un-

debuggable method. Some format of Remote debugging MUST be present

Exercise the options conservatively. When

cost of using or not using Priority Inheritance is not extraordinarily large, use Priority Inheritance

Do not give up hope

8/5/12

Unsung Heroes
But who had found that such problem can

occur? It was quiet three Academicians from Carnegie Mellon University

Lui Sha John Lehoczky and Raj Rajkumar

Standing Applause

8/5/12

Questions??

THANK YOU
8/5/12

WATCHDOG The Ultimate Savior

preferred

Basic Principle is Never to hang If required, Automatically Resetting is If it finds, that the system is HUNG, it will

reset the system

8/5/12