Third Real-Time Linux Workshop Work in Progress Reports

ReTiCol (Real Time Control Library) provides a middle-ware API, based on RTAI, for the design and implementation of real time control systems according to the Port Based Object (PBO) architecture. Using ReTiCoL, the programmer can define specific real time processes for managing the different phases of a control application: Input Processes for data acquisition and filtering from sensors, Computing Processes for data computation and Output Processes to drive actuators. These processes have a PBO structure and communicate through one-to-one channels without synchronization. So the typical data flow within a process is:

data reading from the input-port

data computation

data writing to the output port.

The library simplifies the writing of real time applications decoupling the processes and making easier re-using already written processes.

Moreover, it is possible to implement multi modal control applications defining a process (Mode Process) that implements a finite state machine: each state corresponds to a mode of the application. Compiling ReTiCoL with appropriate options, it is possible to debug the control system, monitoring the buffers of the ports and the schedule time of the processes.

The strict modularity of ReTiCoL will permit, in the future, the achievement of a SIMULINK integrated developing environment for the automatic code generation.

The interrupt processing time can dramatically influence the performance of a real-time operating system. In particular, in Linux interrupts are mainly managed in bottom halves that can steal execution time from the currently executing process.

This is a source of unpredictabilities that can completely jeopardize a real-time guarantee enforced by the CPU scheduler. This problem is generally addressed by heavily modifying the kernel, in order to schedule bottom halves together with real-time tasks, so that they can be guaranteed like all the other real-time activities in the system.

In this paper, we propose a different solution, based on a less intrusive approach. The core abstraction upon which our solution is built is the resource reservation abstraction: according to this approach, each process in the system can be guaranteed to receive an amount Q of a specified resource each period T. CPU reservations are widely used in RT/multimedia systems, and have been implemented in a number of RT versions of Linux, such as, for example, Linux/RK.

When an appropriate CPU reservation is attached to a real-time process, the process is theoretically guaranteed not to miss any deadline, but if the machine is subject to a heavy interrupt load (for example, a heavy load coming from the network interfaces), CPU time can be stolen from the the reserved tasks, and they may miss deadlines. In order to avoid this problem, the amount of CPU time Q reserved to a task is increased by a feedback mechanism when the task starts to miss deadlines.

Using this approach, the execution time stolen by interrupt processing is modeled as a variation in the application execution times, and an adaptive scheduling mechanism automatically compensates the effects of this stochastic variation. The proposed approach has been implemented in Linux/RK (without requiring any additional modification to the original RK), and some preliminary experimental results are presented.

Usually reliability of a system can be improved using standard fault-tolerance technologies. For example, hardware components can be replicated and a vote can be taken on their outputs (majority voting). Replication and majority voting are techniques for dealing with random hardware faults. When space redundancy cannot be used for cost or weight constraints, other techniques can be used in order to maintain the properties of correctness and timeliness of a system even in the presence of faults. In such environments, time redundancy becomes an effective method for achieving fault tolerance in a real-time system.

The goal of this work consists of developing a software layer which provides fault-tolerance for hard periodic real-time applications. Among the error recovery models proposed in the literature, we are considering the primary/backup approach, where each fault-tolerant task is characterized by two different implementations of the same algorithm. To enhance system's efficiency and reduce the delays introduced by software redundancy, particular effort has been devoted to the study of a resource reclaiming mechanism, aimed at recycling the allocated resources when they are not used by the tasks. In fact, in primary/backup mechanism, the processor time reserved for the execution of the backup copy can be reclaimed if the primary task executes successfully.

The implementation of this fault-tolerant mechanism has been realized on Linux/RK which is a real-time version of the Linux kernel. Each fault-tolerant task is implemented as two distinct processes (primary and backup copy). Each Primary can tolerate timing faults (deadline missing) as well as data faults. An overrun (excessive execution) in the primary copy, automatically triggers the execution of the backup copy. On the other hand, a suitable acceptance test can discover failures in the data results provided by the primary; such a failure is recovered using a system call which wakes up the backup process. On-line updating of the primary task is also permitted without suspending the whole application. The backup copy can substitute the primary process until a new version of the primary will be ready to start again.

This work in progress report is on an on-going effort toward open control systems, both in terms of hardware as in terms of software. The presented paper gives an overview of different types of control applications and common approaches to implement control systems. Beyond that, the presentation expands to an explanation of the open control paradigm as it differentiates from a proprietary one. An open control architecture for an open controller is proposed in an abstract way. Advantages and requirements of open control systems are highlighted.

As an example of a quite typical real-time project thats going on now a T1/E1 modem will be described that is in the stage of migration from a proprietary OS to an embedded real-time Linux variant, RTLinuxpro. This project is running at Datentechnik AG in Vienna Austria, who is a provider of backbone-infrastructure elements in the telecom area. The device that will be described in detail here is a mpc860 based custom board, developed and produced at Datentechnik AG. The emphasis of this WIP is to show the steps and the pitfalls of such a migration process, hopefully give helpful pointers to projects and software that can support such a migration effort, and at the same time present the basics of a RTLinuxpro based mpc860 device.

Datentechniks K14 device is a T1 capable ISDN access device. Its core functionality consists of four ISDN lines that are bonded by a dedicated DSP to provide the required 1024Kbit transfer to the telecom-provider. On the customer side multiple devices are offered, X21, I430, V35, Ethernet and G703 (up to 2048kbit), allowing for integration into a wide range of existing LAN-infrastructures. Aside from these data-port a operators interface is available either via serial line or via Ethernet, whereby the Ethernet management port is able to be serviced interactive or via SNMP calls (how dare anybody name this SimpleNMP...). The basic challenge for the migration from the proprietary DTOS to a embedded RTLinux system was to pack a full user-land into a system that up to then was simply running a single monolithic application (or monster, which would describe it better). Naturally such a monolithic application will not exhibit such high resource demands as a full-featured GNU/Linux user-land, also the timing problems become more complex in a multitasking multiuser OS than a monolithic solution would show, at least at first glance. So why if GNU/Linux is so much larger and complex would one want to move to this solution in the first place ? A few of the answers are:

GNU/Linux supplies a standard user interface,

can offer a good development environment based on FSMLabs cross-development kit for mpc860,

allows POSXI standard conform application development for user-space and real-time applications,

provide a high level of security built upon well tested server and client applications (ssh/scp etc.),

Wide hardware support and support for many network protocols in the Linux kernel,

Last but definitely not least - the source availability of the entire Linux system.