When it comes to embedded systems running Linux, developers are primarily concerned about two things: porting or making Linux run on the target without errors, and developing applications that can run on Linux to carry out the job the embedded system is designed for.

JTAG debuggers with Linux awareness are rising to the occasion, helping developers to debug embedded systems running Linux. In this article, the author looks at why it is difficult to use traditional methods for Linux debugging, before delving into how JTAG debuggers help sort out bugs in the different stages of Linux booting and Linux application development.

 [stextbox id=”info” caption=”Understanding a Linux-based Embedded System”]

A Linux-based embedded system can be split into three main components:

  • The Linux kernel: The Linux kernel is the core of the Linux operating system (OS), having ultimate authority. It is the first component to start after the system has been initialised by the boot-loader.
  • Kernel modules: Linux kernel modules are dynamically loaded and unloaded as and when needed; once a Linux kernel module is loaded it has the same level of authority as the Linux kernel. Kernel modules are mainly used for device drivers.
  • Application software: Application software cannot access the Linux kernel memory or hardware directly as they run in user-mode on a Linux system, with reduced privileges. So, if some application software needs to access the peripherals or memory, it needs to request the Linux kernel to provide system-level access to it.


Unique challenges posed by Linux

There are several steps involved in the Linux booting process. The boot-loader first copies itself to RAM from Flash and then loads the Linux kernel. Next, the Linux kernel boots up and performs the transition from physical addresses to kernel virtual addresses.

The memory space allocation for particular processes in the physical memory may be fragmented across multiple memory regions, but they appear to be in a continuous memory address space for the process using it. This is done using the virtual addressing, which is maintained by the Memory Management Unit (MMU) of the Linux system.

In Linux, there is a constant switching between the kernel space and user space and also between different processes. This makes tracking of virtual memory impossible and complicates the debugging of these processes or the Linux kernel. These issues cannot be addressed using common methods like adding “print” statements, or agent-based debugging solutions such as the Kernel GNU debugger (KGDB) or the GNU debugger (GDB).

Using KGDB, a developer cannot examine the current state of the system by halting the CPU, especially in a multi-core or multi-processing environment. The breakpoints that we set in KGDB cannot halt the execution of all the processes at once as the Kernel GNU debugger (KGDB) requires a communication port such as Ethernet (also a process) to be working continuously for the KGDB agent present in the target to communicate with the host system. This also means that to start agent-based Linux kernel-debugging, we require a communication channel to be established with a working IP stack and working device driver, which may not be possible when the kernel is not stabilised or when these communication devices themselves need to be debugged.

In some scenarios, the Ethernet or UART port may not be available for agent-based debugging. For example, cell phones simply do not have serial Ethernet interfaces!

Developers debugging in user-mode will require stepping into system calls from user-mode to Linux kernel-mode and back into user-mode. Keeping track of memory mapping and memory allocation during this process is critical. When using the traditional agent-based solution, we need to use both GDB and KGDB to trace system calls into the Linux kernel and kernel modules. The use of multiple agent-based debug tools may complicate the debug process.

These are just some of the problems one could face when trying to solve problems in embedded Linux-based systems using traditional non-JTAG methods. Now, let us look at how things become simpler with Linux-aware JTAG debuggers.