Real time Linux programming Mark Veltzer mailto:mark@veltzer.

net

Definitions

What is real time?

From wikipedia: real-time computing (RTC), or reactive computing, is the study of hardware and software systems that are subject to a "real-time constraint" e.g. operational deadlines from event to system response. Real-time programs must guarantee response within strict time constraints.[1] Often real-time response times are understood to be in the order of milliseconds and sometimes microseconds. In contrast, a nonreal-time system is one that cannot guarantee a response time in any situation, even if a fast response is the usual result.

Real time hardware

A hardware may guarantee real time performance by utilizing a hardware clock which is a crystal which delivers a pulse at a constant rate. The engineers will make sure that the hardware concludes it's business before the clock strikes again. What you get is a guarantee for how long it takes you to do a certain operation.

Terms

Hard real time means a piece of software that must meet time requirements and never miss even a single dead line. Soft real time is the same but for a system that will continue to perform even if the deadlines are missed. High performance is a piece of software that has to utilize the hardware to the maximum but where each operation is not constrained. A piece of software may fall into more than one category (think chess playing software).

Designing real time software

There are methodologies designed specifically for the design of real time software such as MASCOT, HOOD, AADL, Ravenscar, profile and others. These methodologies deal with how you design your system: decide on numbers of threads or process, the roles of each process, the priorities of the threads and more. They also deal with how to debug a multi-threaded piece of software that must meat the strict deadlines. We will not discuss them further.

Real time and programming language

Real time systems may be written in any language (including Java and Python contrary to public opinion). Because care is needed to disable nondeterministic aspects of the programming language (e.g. garbage collector) then programmers prefer more deterministic languages like C and C++. No, C++ is not less deterministic than C.

Traditional real time operating systems

In traditional real time operating systems the programmers write threads which are part of the operating system kernel (operate in the same space). These systems are kept simple by design so as not to be non deterministic. These systems are fast and deterministic. They are also very hard to debug once the software grows because of the single space.

Where latency/non determinism come from?

Hardware. Examples can be self monitoring interrupts, Caches. Operating system. Examples can be operating system threads, swap, long interrupt handlers, page faults, laziness. Programming language. Examples can be malloc/free(3), garbage collectors. The actual software. Examples can be if statements, algorithm bugs, locking.

The art of real time programming

All this means that to create a real time system one must master all levels of the system: from the hardware, through the operating system, through the programming language to the software. That is also why hard real time programming is hard.

Linux and real time

Linux and real time

Linux was never designed from the beginning to be a real time system and some would claim that UNIX philosophy is contrary to real time. But: Linux has been developing real time capabilities for over a decade. The kernel patch has been written by Ingo Molnar and Thomas Gleixner. It is almost fully integrated into the mainline kernel (3.6). Is expected to be fully integrated in the next year or so. Is usually applied as a patch to the kernel by the people building the real time OS for you. See https://rt.wiki.kernel.org/index.php/Main_Page

What does real time Linux give me?

Interrupts are handled by the kernel in threads with a given priority. The ability to run threads at OS interrupt handler priority or higher. Hard Real time scheduling - separate from the soft scheduling that user space usually uses. Priority inversion. Fine granularity of CPU affinity of user space and operating system threads. New real time clock API for better precision. Less interrupts for real time threads the scheduler does not use regular tick interrupts to check if real time threads need to stop running and be replaced by other threads. Low latency of the kernel (kernel was reviewed and locations where interrupts were disabled were improved) What is the worst time latency? Around 30usec for dedicated systems. More if you put more drivers, less if you use less.

Page faults and their cost

In a Linux system going to the kernel is expensive (1usec region). The system is lazy by default and so uses the MMU to fetch pages it has promised but has not attached to the process. This happens in fetching pages for the executable, data segment, stack, libraries, mmap(2) and malloc(3) (because of mmap(2)). See examples.

Real time programming on Linux

Monitoring page faults

The proc filesystem: /proc/[pid]/stat From the command line: ps -o min_flt Link with libproc.so and use it's API to monitor your own page faults. Use top(1) and configure page faults as one of the fields you see.

Avoiding page faults

Stack: allocate a big buffer on the stack and zero it or call a recursive function a lot. Malloc: zero the memory when you get it. Mmap: zero the memory or pass a flag to mmap. Data segment: zero the memory. For everything including all of the above there is also mlock(2).

Avoiding latency

If you want a really clean CPU all to yourself:

Put your thread on a CPU that does not get hardware interrupts. Do this using the pthread affinity API. On embedded systems hardware interrupts usually go to CPU 0. Use cat /proc/interrupts to make sure. Tune where interrupts go using echo to /proc/irq/ [irq_number]/smp_affinity

Avoiding latency (cont)

On larger systems (Redhat,...) you usually have a /usr/sbin/irqbalance distributing the interrupts evenly between the cores. Either get rid of it (don't run it on boot in /etc/init.d). Or configure it not to send interrupts to the CPU you want to keep clean (use IRQBALANCE_BANNED_CPUS in /etc/default/irqbalance).

Measuring the entire systems latency

Is done using cyclictest(1) which part of the rttests package.

If you don't have it in your system vendors repository then install it from source http://git.kernel.org/cgit/linux/kernel/git/clrkwllms/rt-t It contains not only cyclicttest(1) but other very useful tools.