You can write real time programs using standard Linux as long as you know how to control scheduling. In fact it turns out to be relatively easy and it enables the Raspberry Pi to do things you might not think it capable of. There are also some surprising differences between the one and quad core Pis that make you think again about real time Linux programming. 

 

If you are writing a real time system there are two things that should concern you - how fast the system can act and how poor this response can be in the worst case. 

After learning how to generate accurate and fast pulses we now have the ability to work with I/O down in the microsecond region, but we still have the problem that our program can be interrupted at any time by the operating system.

This means that our outputs and inputs can go drastically wrong. 

For example, if you generate a fast pulse train in the 1 microsecond range using a standard GPIO line and set a logic analyzer to trigger on a long pulse, you will eventually find one or more very long pulses -  typically in the millisecond range. This problem becomes worse the more the CPU is loaded as the operating system switches between tasks to make sure that everything has an opportunity to progress. 

The Problem

If you are familiar with microcontrollers such as the PIC, AMTEL or any dedicated mcu then this idea that there could be something getting between you and the hardware will be new. The majority of simple mcus do nothing but run the program you download. Any talk of an "operating system" generally refers to code that does the downloading or minimal system preparation. When you write a control loop then you can safely assume that the loop will run as you wrote it and without interruption - unless of course you have coded an interrupt handler. 

The point is that in many situations your program is the only program running and you are in complete charge of the processor. 

In the case of running a program on the Raspberry Pi's ARM the situation is very different. Your program is  just one of a number of programs running at any given time. The Raspberry Pi has up to four cores and this means that at most four programs can be running at any given time. The operating system is responsible for starting and stopping programs so that each and every program has a turn to run. 

This is called scheduling and it is a problem if you are trying to write a real time system. 

The problem is that you might write a program that toggles a GPIO line between high and low with a given timing, but whether this timing is honored depends on not just your program but on the operating system as well. You can't even be sure how the operating system will treat your program because it depends in a fairly complex way on what else is running on the system and exactly what the other programs are doing. 

Sometime this is expressed as your program execution being non-deterministic whereas in a simple mcu it is deterministic. This means that if you run the same program twice on on the Raspberry Pi you probably don't get exactly the same result but on an mcu this is a reasonable expectation. 

The whole subject of multi-tasking operating systems and scheduling in particular is a large one and it is usually taught as part of a computer science degree - but generally not as it applies to real time programming. What this means is that there is often a lot of guess work involved in getting programs with real time demands to work properly under general operating systems such as Linux. In fact it is often state that you can't do real time processing under Linux because you cannot even place a bound, an upper limit, on how long your program might be suspended by the OS. This isn't true and real time processing on standard Linux is possible - as long as you are able to live within the constraints. 

As an alternative you could opt to run a specially designed real time OS that does provide guarantees on how quickly a request will be serviced. There are specifically real time versions of Linux that you can install, but since version 2.6 the Linux Kernel has had sufficient real time facilities for many applications so you don't need to move to anything different to the standard Raspbian.

Before we continue it is important to realize that there is no way that a real time operating system can increase the speed of operation of the processor - the maximum speed of operation cannot be improved upon. In the case of the Raspberry Pi this means that you can achieve around the 1 microsecond pulse times if you are careful and no amount of real time programming is going to improve on this. 

What real time provides is higher consistency of that response time. It isn't perfect, however, and after we have used all of the features of real time Linux there will still be small periods of time when your program isn't operating and there is little to be done about this.

RT Scheduling

Every Linux thread is assigned a scheduling policy and a static priority. 

The normal scheduling algorithm, SCHED_OTHER, that Linux uses applies to all threads with static priority zero. If you are not using real time scheduling then all the threads run at priority zero. In place of a static priority each thread is assigned a dynamic priority, which increases each time it is passed over for execution by the scheduler. The scheduler gives the thread with the highest dynamic priority an opportunity to run for one quantum of time or for one time slice. A thread can be suspended before its time slice is up because it has to wait for I/O or because it is blocked in some other way. Any time a thread makes system call it is also a candidate to be suspended in favor of another thread. 

You have only a little control over the computation of the dynamic priority. All you can do is set its initial value using the nice command or setpriority. 

The normal scheduling algorithm doesn't provide much control over what runs. It is "fair" in the sense that all threads get a turn at running, but it isn't possible to set a thread to have a high priority so that it runs in preference to all others. 

To do this we need to look at the real time scheduling options. 

The most important for us is SCHED_FIFO and sometimes the closely related SCHED_RR.

These apply to threads, real time threads, with static priorities 1 to 99(high). 

The first thing to note is that a thread with priority greater than zero will always run in preference to a thread with priority zero and so all real time threads will run before a thread using the normal scheduling algorithm. 

What happens in FIFO is that the system maintains queues of threads that are ready to run at each priority. It then looks for the list with the highest priority with threads ready to run and it starts the thread at the head of the list. 

When a thread is started it is added to the back of its priority queue.

Once a FIFO thread gets to run it can be preempted by a thread with a higher static priority that is ready to run. 

If a FIFO thread is suspended because of a higher priority thread it goes back at the head of the queue. This makes it the next thread to resume. This is the sense in which the schedule is First In First Out FIFO - if a thread is suspended by another thread of higher priority that becomes runnable then it is restarted as soon as that thread that replaced it is suspended or stops running.

Finally if a thread explicitly yields (by calling yield) it goes to the end of its priority queue.

 

Setting Scheduling Priority

This sounds like chaos but if you think about it for a moment and start simply you will see that it provides most of what you are looking for. You are in full control of the Raspberry Pi and so you can determine exactly how many non-zero priority threads there are. By default all of the standard threads are priority zero and scheduled by the normal scheduler.

Now consider what happens if you start a FIFO scheduled thread with priority 1.

It starts and is added to the end of the priority 1 queue. Of course it is the only priority 1 process and so it starts immediately on one of the cores available. If the process never makes a call that causes it to wait for I/O say or become blocked in some other way then it will execute without being interrupted by any other process. 

In principle this should ensure that your process never delivers anything but its fastest response time. 

This is almost but not quite true. 

There are more complex situations you can invent with threads at different priorities according to how important they are but this gets complicated very quickly.

A modification to the SCHED_FIFO scheduler is SCHED_RR - for Round Robin. In this case everything works as for SCHED_FIFO except that each running process is only allowed to run for a single time slice. When the time slice is up the thread at the head of  the priority queue is started and the current thread is added to the end of the queue. You can see that this allows each thread to run for around one time slice in turn - which is a round robin scheduler. 

In most cases for real time programming with the Raspberry Pi the SCHED_FIFO scheduler is what you need and in its simplest form.

The complete set of scheduling commands are: 

• sched_setscheduler Set the scheduling policy and parameters of a specified thread
   
• sched_getscheduler Return the scheduling policy of a specified thread
   
• sched_setparam Set the scheduling parameters of a specified thread
   
• sched_getparam  Fetch the scheduling parameters of a specified thread
   
• sched_get_priority_max Return the maximum priority available in a specified scheduling policy
   
• sched_get_priority_min Return the minimum priority available in a specified scheduling policy
   
• sched_rr_get_interval Fetch the quantum used for threads that are scheduled under the "round-robin" scheduling policy
   
• sched_yield Cause the caller to relinquish the CPU, so that some other thread be executed
   
• sched_setaffinity  Set the CPU affinity of a specified thread
   
• sched_getaffinity  Get the CPU affinity of a specified thread
   
• sched_setattr Set the scheduling policy and parameters of a specified thread; this Linux-specific system call provides a superset of the functionality of sched_setscheduler and sched_setparam
   
• sched_getattr Fetch the scheduling policy and parameters of a specified thread; this Linux-specific system call provides a superset of the functionality of sched_getscheduler and sched_getparam.

The scheduling types supported are:

SCHED_OTHER the standard round-robin time-sharing policy

SCHED_BATCH for "batch" style execution of processes

SCHED_IDLE for running very low priority background jobs

SCHED_FIFO a first-in, first-out policy

SCHED_RR a round-robin policy

where only the final two are real time schedulers. 

Also notice that all of the scheduling function return an error code which you should check to make sure thing have worked. For simplicity the examples that follow ignore this advice. 

How Bad Is The Problem?

The first question we need to answer is how bad the situation is without real time scheduling.

This is not an easy question to answer because it depends on so many factors. Take, for example, a very simple program which toggles a GPIO line as fast as it can:

#include <bcm2835.h>
#include <stdio.h>
#include <sched.h>

int main(int argc, char **argv) {
 if (!bcm2835_init())
   return 1;
 bcm2835_gpio_fsel(RPI_GPIO_P1_07 ,
 BCM2835_GPIO_FSEL_OUTP);
 while (1) {
  bcm2835_gpio_write(RPI_GPIO_P1_07 , HIGH);
  bcm2835_gpio_write(RPI_GPIO_P1_07 , LOW);
 }
 bcm2835_close();
 return 0;
}

As we have already discovered we can generate pulses less than 1 microseconds wide using this method.

The real question is how does the scheduler change this pulse length by interrupting your program?

The Pi Zero Single Core 1GHz

The Pi Zero has a single core ARM processor so it can only do one thing at a time and makes a perfect worst case example. 

If you run the program with a logic analyzer connected and with a freshly booted Pi Zero and no other tasks running then you will see the usual moderately rough pulse train with the regular 0.08 to 0.1 microsecond pulses with .5 microsecond pulses every 1.25ms. 

However if you set the logic analyzer to capture long pulses even if the Pi isn't loaded you will see, after a few seconds, pulses as long as 0.5ms. 

 

 

Even on a lightly loaded Pi Linux will interrupt the running task every few seconds. 

On a Pi Zero running even one demanding task the story is worse. 

 

In this case the task is a simple infinite loop that will use as much of the CPUs time as it is allowed to. The result is a regular 10ms pulse. This is due to the Linux scheduler dividing up the time so that both tasks get a reasonably equal share of the single CPU. If the Pi you are using has more than one core then this behavior will occur when the number of equal priority tasks equals the number of cores. 

Yes, with these conditions the program can produce 10ms pulses mixed in with the 0.1ms pulses.   

Pi Zero Real Time FIFO Scheduling

Now we can try the same test but with FIFO real time scheduling selected.

To do this we need to use the sched_setscheduler function:

sched_setscheduler(pid,SCHED_FIFO,&priority);

where pid is the thread id - if zero then the calling thread is used.

The second parameter sets the type of scheduling used FIFO in this case and the final parameter is a pointer to a structure that specifies the priority. 

Notice you need to include <sched.h> to make use of the scheduling functions.

The modified program is: 

#include <bcm2835.h>
#include <stdio.h>
#include <sched.h>

int main(int argc, char **argv) {
 const struct sched_param priority = {1};
 sched_setscheduler(0, SCHED_FIFO, &priority);

 if (!bcm2835_init())
   return 1;
 bcm2835_gpio_fsel(RPI_GPIO_P1_07 ,
 BCM2835_GPIO_FSEL_OUTP);
 while (1) {
  bcm2835_gpio_write(RPI_GPIO_P1_07 , HIGH);
  bcm2835_gpio_write(RPI_GPIO_P1_07 , LOW);
 }
 bcm2835_close();
 return 0;
}

 

If you run this program you will discover that the result are very different. The first thing you might notice if you have a console open is that the response is very sluggish. The reason is that your program is hoging the CPU.

If you run a logic analyzer then there is a surprise.

The program runs for one second and then it is interrupted for around 50ms.

Clearly there is something with a higher priority is now interrupting your program and probably keeping the console and other essentials alive. 

If you now run a cpu hoging task you will discover that there is no change in the behavior of your program. It runs for a second and then is interrupted for 50ms.  There is no 10ms switching between tasks - your program still gets a full 1 second of uninterrupted cpu time and this is irrespective of the cpu load.

You can also try locking the programs memory to stop the system from paging it if other applications need a lot of memory. In practice this isn't a common occurrence on a device like the Raspberry Pi but if you do need to do it then all you need is a call to mlock in sys/mman.h and to unlock to unlock it. You can lock the current memory or future memory allocations. In most cases it is sufficient to lock current memory unless you are using dynamic memory allocation. 

mlockall(MCL_CURRENT);

In the case of our test program locking memory make no difference as there is plenty of real memory to go around. 

Pi 2  Quad Core 900MHz

The single core Pi Zero has to share its single core even with a high priority FIFO process. The quad core Pi 2 and Pi 3 should behave differently because they have three cores to work with after you monopolize a whole core with your program. 

Lets see how things work with a Pi 2. 

First lets try the program without real time features. With a lightly loaded Pi 2 we have the usual 0.08 to 0.1 microsecond pulses with the usual 0.6 microsecond pulse every 0.125ms and 15 microsecond pulse every 10ms. 

When you add cpu hoging processes then at about four processes irregular 10ms pulses start to appear

The 10ms pulses correspond to periods of time when the program is swapped out and not running. It isn't regular because the four cores are shared between each of the processes. 

If you change the program to use realtime FIFO scheduling then with the same cpu load that produced the 10ms pulses you get a result that looks like a lightly loaded CPU. That is the 10ms pulses vanish and all you have to contend with are the 0.6 microsecond and 15 microsecond interruptions. 

The reason for this behavior is fairly obvious. Now your process can be run by one of the cores while the remaining three are shared between the other non-realtime processes. 

Thus for a Pi 2 or 3 near realtime Linux works better than for a single core Pi Zero or Pi 1. 

For a Pi 2 or 3 you can reasonably expect to respond within 15 microseconds. 

Practical FIFO Scheduling

Adding a simple statement makes your program hog one of the available processor cores and removes the long interruptions that occur when other threads are scheduled to run. 

You might think at this point that the best thing to do is set a maximum priority and use FIFO scheduling as soon as your program is loaded. However hogging a single core on a Raspberry Pi can have some undesirable effects. 

To avoid these it is a good idea to only enable FIFO scheduling when it is absolutely needed or to use the yield command at regular intervals.

For example if you are writing a program that has to decode an incoming pulse stream then using FIFO scheduling for the time it actually does the decoding is the best option. After the decoding is complete return to SCHED_OTHER, 

It is true that no amount of scheduling is going to convert the Raspberry Pi into a real time system. The reason is that it is not real time hardware and not configured to be. There are things the CPU has to do to keep the Pi running and these cannot be disabled. 

The best you can achieve with the Pi Zero is to effectively lock your task into its single core an keep the CPU uninterrupted for a one second period and accept the 50ms interrupt. You can only guarantee to respond to external events in 50ms. Of course you still have the  .5 microsecond interrupts every 1.25ms to take into account but unless you are working with below 1 microsecond timings these are not usually a problem.

For the quad core Pi 2 and 3 you can use one of the cores to keep your program running and only have the  0.6 microsecond pulse every 0.125ms and 15 microsecond pulse every 10ms to worry about. 

So we really do have near real time Linux if not actual real time Linux.

 

 

 

Related Articles

Real Raspberry Pi - Getting Started And Custom NOOBS