run mode debugging manual linux - lauterbach · run mode debugging manual linux 3 ©1989-2019...

Run Mode Debugging Manual Linux

TRACE32 Online Help

TRACE32 Directory

TRACE32 Index

TRACE32 Documents ......................................................................................................................

OS Awareness Manuals ................................................................................................................

OS Awareness and Run Mode Debugging for Linux ..............................................................

Run Mode Debugging Manual Linux ..................................................................................... 1

History ................................................................................................................................... 3

Debugging Modes for Embedded Linux ............................................................................ 3

Run Mode Debugging with TRACE32 as GDB Front-end 3

Stop Mode Debugging via JTAG 3

Integrated Run & Stop Mode Debugging via JTAG 4

Supported Debugging Modes sorted by Architectures 5

Basic Concepts .................................................................................................................... 6

t32server (only ARM) 6

gdbserver 6

Ethernet as Communication Interface to the gdbserver 8

Operation Theory 8

Switching to Run Mode Debugging 9

DCC as Communication Interface to the t32server 10

The Debug Communications Channel (DCC) 10

Operation Theory 10

Switching to Run Mode Debugging 11

The Space ID for Run Mode Debugging 12

Process Debugging 13

Quick Start Example for ARM ............................................................................................. 15

Quick Start Example for MIPS ............................................................................................. 17

Quick Start Example for PowerPC ...................................................................................... 19

SYStem.Option NoDebugStop 20

Switching between Run & Stop Mode Debugging ............................................................ 21

Commands for Run Mode Debugging ................................................................................ 24

Breakpoint Conventions ...................................................................................................... 25

Debugging of Multi-threaded Applications ........................................................................ 26

Frequently-Asked Questions .............................................................................................. 27

Run Mode Debugging Manual Linux 1 ©1989-2019 Lauterbach GmbH

Run Mode Debugging Manual Linux

Version 06-Nov-2019

History

28-Aug-18 The title of the manual was changed from “RTOS Debugger for <x> - Run Mode” to “Run Mode Debugging Manual <x>”.

Debugging Modes for Embedded Linux

TRACE32 provides 3 modes for debugging embedded Linux: Run Mode Debugging, Stop Mode Debugging and Integrated Run & Stop Mode Debugging.

Run Mode Debugging with TRACE32 as GDB Front-end

Pure software debugger, no TRACE32 hardware required.

The TRACE32 software is licensed by a USB dongle.

Requires a gdbserver/gdbstub running on the target.

The TRACE32 GDB Front-end works in so-called Run Mode Debugging: At a breakpoint only the selected process is stopped, while the kernel and all other processes continue to run.

The TRACE32 GDB Front End is described in the document “TRACE32 as GDB Front-End” (frontend_gdb.pdf).

Stop Mode Debugging via JTAG

Stop Mode Debugging requires a TRACE32 JTAG debugger hardware.

If debugging is performed via the JTAG interface, TRACE32 works in so-called Stop Mode Debugging. At a breakpoint the CPU and thus the whole target system is stopped.


The main advantages of Stop Mode Debugging are:

• Since the only requirement for Stop Mode Debugging is a functioning JTAG interface, debugging can start at the reset vector.

• Since TRACE32 provides both Linux and MMU support, debugging of the kernel and beyond process boundaries is possible.

Stop Mode Debugging is described in the document “OS Awareness Manual Linux” (rtos_linux_stop.pdf).

Integrated Run & Stop Mode Debugging via JTAG

Integrated Run & Stop Mode Debugging requires a TRACE32 JTAG debugger hardware.

If debugging is performed via the JTAG interface, TRACE32 can be configured:

• To allow Stop Mode Debugging via JTAG.

• To allow Run Mode Debugging via the t32server/gdbserver running as debug agent on the target.

• To allow, if required, to switch between the two debugging modes.

TRACE32 communicates with:

• The gdbserver via ethernet for all supported architectures (see table on next page) or

• The t32server via DCC (Debug Communications Channel) for the ARM architecture. Since the ARM-JTAG interface provides a DCC, Integrated Run & Stop Mode Debugging can function with JTAG as the only communication interface.


Supported Debugging Modes sorted by Architectures

Architecture GDB Front-end Stop ModeDebugging

Integrated Run & Stop Mode Debugging

ARM Yes, refer to “TRACE32 as GDB Front-End” (frontend_gdb.pdf).

Yes, refer to “OS Awareness Manual Linux” (rtos_linux_stop.pdf).

Yes, via DCC (except Cortex-M) or Ethernet. For details refer to this manual.

ColdFire — Yes, refer to “OS Awareness Manual Linux” (rtos_linux_stop.pdf).

—

I386 Yes — Yes, via Ethernet

MIPS Yes, refer to “TRACE32 as GDB Front-End” (frontend_gdb.pdf).


Yes, via ethernet. For details refer to this manual.

PowerPC Yes, refer to “TRACE32 as GDB Front-End” (frontend_gdb.pdf).



SH4 Yes Yes, refer to “OS Awareness Manual Linux” (rtos_linux_stop.pdf).


XScale Yes Yes, refer to “OS Awareness Manual Linux” (rtos_linux_stop.pdf).

—


Basic Concepts

For Integrated Run & Stop Mode Debugging, Stop Mode Debugging via the JTAG interface is extended by:

• t32server/gdbserver as debug agent on the target.

• A communication interface between TRACE32 and the debug agent (ethernet or DCC).

t32server (only ARM)

The t32server is a gdbserver extension provided by LAUTERBACH. Compared to the gdbserver the t32server allows debugging over DCC for the ARM architecture. The t32server starts a gdbserver for every process.

• the gdbserver has to be in the /bin directory of the Linux file system.

• the IP address 127.0.0.1 has to be configured, because the t32server communicates with the separate gdbservers via TCP/IP.

In order to provide Integrated Run & Stop Mode Debugging the t32server has to be started as a Linux process on the target via the terminal window e.g.:

gdbserver

You can display the version of your gdbserver by typing in the terminal

If nothing is printed out, then your gdbserver has a version older than 6.2. Using version 6.2 and older is not recommended.

If run mode debugging over Ethernet is used, we recommend to use gdbserver 7.1 or newer.

./t32server ; Communication via DCC

gdbserver --version


If used together with the t32server (communication over DCC), gdbservers newer than 6.5 have to be patched to disable run length encoding:

If the run length encoding is not disabled in the gdbserver, the communication with the t32server will be too slow.

Patching gdbserver newer than 6.5 (DCC communication only)file: gdb-x.x/gdb/gdbserver/remote-utils.c

static inttry_rle (char *buf, int remaining, unsigned char *csum, char **p){ int n;

/* Always output the character. */ *csum += buf[0]; *(*p)++ = buf[0]; return 1; // add this to disable run length encoding /* Don't go past '~'. */ if (remaining > 97) remaining = 97;...


Ethernet as Communication Interface to the gdbserver

TRACE32 communicates with the gdbserver via ethernet for the MIPS and the PowerPC architecture.

Operation Theory

Linux

gdbserver

SOFTWARE

HARDWARE

MIPSCore

JTAG

ethern

et


Switching to Run Mode Debugging

After TRACE32 was started and configured for Stop Mode Debugging switching to Run Mode debugging is performed as follows:

After the communication is configured, debugging can be performed completely via the TRACE32 PowerView GUI.

SYStem.PORT 10.1.2.99:2345

Go.MONitor

SYStem.PORT <ip>:<port> Configure ethernet communication

<ip> is the target IP-Address<port> is the used port number

Go.MONitor Switch to Run Mode Debugging


DCC as Communication Interface to the t32server

TRACE32 communicates with the t32server via DCC for the ARM architecture.

The Debug Communications Channel (DCC)

The JTAG interface of the ARM architecture includes a so-called Debug Communications Channel (DCC). Information exchange via DCC is possible between

• the TRACE32 software running on the host

• and the t32server running as Linux process on the target

On an SMP system, the debugger only communicates with the DCC registers of the first core. Thus, the t32server always run on this core. For details about DCC refer to your ARM Technical Reference Manual.

Operation Theory

Linux

t32server

127.0.0.1:port1gdbserver Process 1

127.0.0.1:port2gdbserver Process 2

127.0.0.1:portngdbserver Process n

SOFTWARE

HARDWARE

ARMCore

JTAG

DC

C


Switching to Run Mode Debugging

After TRACE32 was started and configured for Stop Mode Debugging switching to Run Mode debugging is performed as follows:

After the communication is configured, debugging can be performed completely via the TRACE32 PowerView GUI.

SYStem.MemAccess GdbMON

Go.MONitor

SYStem.MemAccess GdbMON Configure DCC as communcation interface to t32server

Go.MONitor Switch to Run Mode Debugging


The Space ID for Run Mode Debugging

Processes of Linux may reside virtually on the same addresses. To distinguish those addresses, the debugger uses an additional space ID that specifies to which virtual memory space an address refers. In Run Mode Debugging the space ID is equal to the process ID.

The command SYStem.Option MMUspaces ON enables the additional space ID.

A source code listing for the process sieve is displayed as follows:

For details on the space ID for Stop Mode Debugging, refer to “Training Linux Debugging” (training_rtos_linux.pdf).

Space ID (decimal and hex value)

Space ID


Process Debugging

To debug a process proceed as follows:

1. Check if the process is already running.

2. Load the process for debugging.

If the process is not running, the command TASK.RUN can be used to load the process for debugging.

If the process is already running, the command TASK.SELect can be used to attach to it.

TASK.List.tasks List all running processes

TASK.List.tasks

TASK.RUN <process> Load <process> (process not running)

TASK.select <id> Attach to the process(process already running)

; Load process sieve from the Linux file system and prepare it for; debugging

TASK.RUN /bin/sieve

TASK.SELect /bin/sieve


3. Load the symbol and debug information for the process.

Since processes of Linux may reside virtually on the same addresses, the symbol and debug information has to be loaded for the address space of the process by using the <space_id>.

/NOCODE - load only symbol information.

/NoClear - obtain the symbol information loaded for other processes.

/NOREG - avoid any unintended change to the CPU registers.

Example for a PRACTICE script:

Data.LOAD.<file_format> <file> <space_id>:0 /NOCODE /NoClear /NOREG

; Data.LOAD.Elf <file> <space_id>:0 /NOCODE /NoClear /NOREGData.LOAD.Elf sieve.elf 0x91:0 /NOCODE /NoClear /NOREG

; Stop sieve at main and display source listingGo mainData.List

TASK.PROC.SPACEID(<process>) This function returns the <space_id> of a process. This is required for PRACTICE scripts.

LOCAL &spaceid&spaceid=TASK.PROC.SPACEID("sieve")

TASK.RUN /bin/sieve

Data.LOAD.Elf sieve.elf &spaceid:0 /NOCODE /NoClear /NOREG


Quick Start Example for ARM

Integrated Run & Stop Mode Debugging requires that Stop Mode Debugging is working properly before Run Mode Debugging can be activated. The following commands represent a basic TRACE32 setup for Stop Mode Debugging. It is assumed that the target setup and Linux booting is performed by the code in the boot FLASH.

SYStem.CPU ARM920 ; Select the target CPU

SYStem.Option MMUspaces ON ; Extend logical addresses by; space ID

TrOnchip.Set DABORT OFF ; Debug mode is not entered at data; abort exception used by Linux; for page miss!

TrOnchip.Set PABORT OFF ; Debug mode is not entered at; prefetch abort exception used by; Linux for page miss!

TrOnchip.Set UNDEF OFF ; Debug mode is not entered at an; UNDEF instruction may be used by; Linux for FPU detection

SYStem.MemAccess GdbMON ; Configure DCC communication; between TRACE32 and t32server

SYStem.Mode Attach ; Establish the communication; between the debugger and the CPU

Data.LOAD.Elf vmlinux /NOCODE ; Load the kernel symbols

TASK.CONFIG linux ; Enable the TRACE32 linux; awareness

MMU.FORMAT LINUX swapper_pg_dir ; Define an MMU page table format; for Linux

TRANSlation.COMMON 0xC0000000--0xffffffff

; Upper memory pages are valid for; all space IDs

TRANSlation.TableWalk ON ; Configure the debugger to proceed; as follows at an conversion; from virtual to physical address:; ; 1. Look-up in the debugger MMU; table; 2. Walk through the Linux MMU; table, if the look-up in the; debugger MMU fails

TRANSlation.ON ; Switch debugger MMU to ON


The target setup and the preparations for Linux debugging might be more complex for your system. It is strongly recommended to refer to “ARM Debugger” (debugger_arm.pdf) and “OS Awareness Manual Linux” (rtos_linux_stop.pdf) for details.

To configure Run Mode Debugging:

1. Make sure you have a gdbserver in the /bin of the Linux file system on your target.

2. Make sure the loop-back address 127.0.0.1 is configured on your target.

3. Start t32server as Linux process via the terminal window:

4. Then continue to enter the following commands in TRACE32:

./t32server

Go.MONitor ; Switch to Run Mode Debugging


Quick Start Example for MIPS


The target setup and the preparations for Linux debugging might be more complex for your system. It is strongly recommanded to refer to “MIPS Debugger and Trace” (debugger_mips.pdf) and “OS Awareness Manual Linux” (rtos_linux_stop.pdf) for details.

SYStem.CPU MIPS24KE ; Select the target CPU

SYStem.Option Endianness Big ; Select big endian mode



SYStem.PORT 10.1.2.99:2345 ; Configure ethernet communication; between TRACE32 and t32server




TRANSlation.COMMON 0x80000000--0xffffffff


TRANSlation.TableWalk ON ; Configure the debugger to proceed; as follows at an conversion; from virtual to physical address:; ; 1. Look-up in the debugger MMU; table; 2. Walk through the Linux MMU; table, if the look-up in the; debugger MMU fails


SYStem.Option gdbNONSTOP ON ; Enable GDB non-stop mode

SYStem.Option gdbEXTENDED ON ; Enable GDB extended mode



1. Start the gdbserver in multi-process mode:

2. Then continue to enter the following commands in TRACE32

gdbsever --multi:2345



Quick Start Example for PowerPC


The target setup and the preparations for Linux debugging might be more complex for your system. It is strongly recommended to refer to your PowerPC Processor Architecture Manual (e.g. “PPC600 Family Debugger” (debugger_ppc600.pdf) ) and “OS Awareness Manual Linux” (rtos_linux_stop.pdf) for details.

So that ethernet will not be broken in the case of some PowerPCs, please do not stop the CPU after Linux is started. If you would like to debug in stop mode first, please go to run mode with Go.MONitor and back to stop mode with Break.MONitor, so that the ethernet interface could be disabled.

SYStem.CPU MPC8349 ; Select the target CPU



SYStem.PORT 10.1.2.99:2345 ; Configure ethernet communication ; between TRACE32 and t32server




TRANSlation.COMMON 0xC0000000--0xffffffff


TRANSlation.TableWalk ON ; Configure the debugger to proceed; as follows at an conversion; from virtual to physical address:; ; 1. Look-up in the debugger MMU ; table; 2. Walk through the Linux MMU; table, if the look-up in the ; debugger MMU fails


SYStem.Option gdbNONSTOP ON ; Enable GDB non-stop mode

SYStem.Option gdbEXTENDED ON ; Enable GDB extended mode



1. Start the gdbserver in multi-process mode:

2. Then continue to enter the following commands in TRACE32

SYStem.Option NoDebugStop

For PowerPC, software breakpoints and single step are enabled respectively in stop or run mode:

• SYStem.Option NoDebugStop ON: software breakpoints and single step are enabled in run mode.

• SYStem.Option NoDebugStop OFF: software breakpoints and single step are enabled in stop mode.

This option is automatically set/reset by the debugger with every Go.MONitor and Break.MONitor command (except the first Go.MONitor. After the first Go.MONitor command, breakpoints and single step will not work in run mode).

Please note that this option could only be set/reset while the CPU is stopped. If you change this option manually, this could disable breakpoints and single step in you current mode.

gdbsever --multi:2345



Switching between Run & Stop Mode Debugging

The graphic above shows a simple schema of the switching between Run Mode Debugging and Stop Mode Debugging. Not all transitions are covered.

The following commands are used to switch between Run & Stop Mode Debugging:

Go.MONitor If the CPU is stopped, the program execution is started.

Switch to Run Mode Debugging.

In Run Mode Debugging no process is selected for debugging.

Break.MONitor Switch to Stop Mode Debugging and stop the program execution on CPU.

Break.SetMONitor ON Switch to Run Mode Debugging with the next Go.

In Run Mode Debugging no process is selected for debugging.

Break.SetMONitor OFF Switch to Stop Mode Debugging.

If the selected process was running or no process was selected, the CPU stays running in Stop Mode.

If the selected process was stopped, the CPU is stopped in Stop Mode.


If Run Mode Debugging is active, a green M is displayed in the state line of TRACE32.

The following states are possible in Run Mode Debugging:

1. Run Mode Debugging active (green M), no process selected (see TASK.List.tasks).

2. Run Mode Debugging active (green M), selected process (sieve) stopped.


3. Run Mode Debugging active (green M), selected process (sieve) running.


Commands for Run Mode Debugging

If the selected process has been started with TASK.RUN, it will be selected as current process. Otherwise a gdbserver will be started and the selected process will be attached.

Only processes that have been started with a TASK.RUN or that have been attached with TASK.SELect can be killed. After a TASK.KILL, the process and its gdbserver will be killed.

TASK.List.tasks List all running processes

TASK.RUN <process> Load a process for debugging

If the command TASK.RUN is used to load a process for debugging, the process is stopped by the gdbserver at the label _start in ld-linux.

To start process debugging, load the symbol information for the process first and then type Go main.

TASK.select <id> Select a process for debugging

It is only allowed to attach to user process. If you try to attach to a kernel process, an error message will be returned.

TASK.KILL <id> Request GDB agent to end the process

TASK.COPYUP <src> <dest> Copy a file from the target into the host

TASK.COPYDOWN <src> <dest> Copy a file from the host into the target

SYStem.GdbLibPath <path> Set the path to the target libraries on the host


Breakpoint Conventions

For Integrated Run & Stop Mode Debugging please keep the following breakpoint convention:

• Use on-chip breakpoints for Stop Mode Debugging

If an on-chip breakpoint is hit in Run Mode Debugging, the CPU is stopped in Stop Mode debugging (only for ARM).

• Use software breakpoints for Run Mode Debugging

Examples for Stop Mode Debugging:

Examples for Run Mode Debugging:

; Break.Set <space_id>:<address> /Program /OnchipBreak.Set 0x0:0x4578 /Program /Onchip

Break.Set error /Program /Onchip

; Break.Set <space_id>:<address> /Program /SOFTBreak.Set 0x2bc:0x0xd0065789 /Program /SOFT

Break.Set 0x2bc:main /Program /SOFT


Debugging of Multi-threaded Applications

To be able to debug multi-threaded applications, TRACE32 needs the target libraries ld.so.1 (ld-linux.so.2 or ld-linux.so.3 in the case of ARM targets) and libpthread.so.0. The path to this libraries on the host can be set with the TRACE32 command SYStem.GdbLibPat h. For example:

SYStem.GdbLibPath C:\T32\libs


Frequently-Asked Questions

e500 breakpoints

Ref: 0426

Breakpoint are not working properly or stop working completely for the e500 cores

The Linux kernel for e500 cores (PQ3/MPC85XX/QorIQ P10XX/P2020) has to be patched to enable debugging via JTAG. The MSR_KERNEL macro defined in arch/powerpc/include/asm/reg_book.h needs to be changes to include the MSR_DE bit: #define MSR_KERNEL (MSR_ME|MSR_RI|MSR_CE|MSR_DE)

Kernel message "soft lockup"

Ref: 0388

What does the kernel message soft lockup mean?

The kernel detects when there is too much time gone between two timer ticks. This easily happens, if the system is halted with the debugger. Please switch off the soft lockup detection by disabling CONFIG_DETECT_SOFTLOCKUP in the kernel configuration.

Kernel panic when debugger attached

Ref: 0477

The kernel boots fine without debugger, it crashes however when the debugger is attached.

You should check the information printed by the kernel when it panics. Some Linux kernel distributions have security mechanisms that treat a connected debugger via JTAG as a security violation and enter an infinite loop. This behavior can generally be deactivated by editing the kernel configuration or patching the kernel. In the case of some NXP Linux kernels for the i.MX processor family for instance, this security mechanism can be disabled by disabling CRYPTO_DEV_FSL_CAAM_SECVIO in the kernel configuration. In some other cases, the crash could be simply due to debugging only a subset of the cores used by an SMP kernel. In case you attach the debugger to a single core for instance and halt that core, the SMP kernel panics as it detects that single cores are blocked.


Linux power down modes

Ref: 0395

After booting Linux, why does the target die or loose connection to the debugger?

This is often caused by the kernel power management which turns the on-chip debug module off while going to power saving mode. Possible solutions are:

re-compile the kernel with disabled power management.

disable the idle states for all cores from the Linux shell e.g.

echo 1 > /sys/devices/system/cpu/cpu<x>/cpuidle/state<x>/disable

remove the idle-states property from the device tree if available

add a command line parameter that disables power management if sup-ported by the used Linux distrubution. This can be done for instance inNXP Linux distributions for i.MX using the "jtag=on" parameter. In someother distributions the "nohlt" parameter is used. Please refer to the docu-mentation of the kernel command line parameters of your Linux distributionfor more information.

Linux trace in simulator

Ref: 0304

What is needed to revise a Linux trace with a TRACE32 instruction set simulator?

The TRACE32 Linux awareness offers a dialogue window to save important information for a belated trace analysis on the TRACE32 Instruction Set Simulator. You just need to select the menu "Linux" > "Generate RAM Dump". The dialogue is based on the script ramdump.cmm available in the path of the Linux awareness.

MMU.FORMAT LINUX swapper_pg_dir

Ref: 0255

Why does "MMU.FORMAT LINUX swapper_pg_dir" report "invalid combination"?

Space IDs must be switched on for this command to work: "SYStem.Option MMUSPACES ON".

On-chip breakpoint removed by the kernel

Ref: 0478

On-chip breakpoints do not stop the program execution during kernel boot.

On-chip breakpoints can be removed by the kernel during boot:

The Linux kernel resets on Arm the breakpoints as well as the VectorCatch Register when booting if CONFIG_HAVE_HW_BREAKPOINT isenabled in the kernel configuration. See arch/<arm|arm64>/ker-nel/hw_breakpoint.c.

A similar problem can also be seen on Intel x86 when the debug registersare cleared during boot. Please refer to arch/x86/include/asm/debugreg.h.This can be detected by enabling the option TrOnchip.Set GeneralDetectON.


On-demand paging

Ref: 0252

After halting at a function entry point with an on-chip breakpoint, or after stepping into a new function, the List window shows "???" for all assembler lines.

This behavior is probably caused by the on demand paging mechanism of Linux. Please read the "On-Demand Paging" chapter in the Linux Awareness Manual.

PPCSIM Trap

Ref: 0425

The error message "PPCSIM Trap <number>" appears when using the OS awareness

This error message is due to an incompatibility between the TRACE32 software and the loaded OS awareness.
