█ SecBSD: Crash Reports

Minimum information for kernel problems

Familiarize yourself with the general bug reporting procedures first. All of that will apply. When reporting a kernel panic or crash, please remember:

• We need the console output on the screen. Capture it and save it. Serial consoles are best, but if you are on a VGA console you can scroll the console back and take readable pictures with a phone or camera.
• If the kernel panicked we need the traceback. It may be displayed on the screen. If you are at a ddb> prompt, type trace. If you are running SMP, use the mach ddbcpu N command for each of the N processors you have and repeat the trace command for each processor.
• We need the process list. Use the command ps to get that.

Reports without the above information are useless. This is the minimum we need to be able to track down the issue.

Additional information you can send

In some situations more information is desirable. Below are outlined some additional steps you can take in certain situations:

• If your crash appears to involve filesystems. The following additional things would be helpful
  • The output of the ddb> command show uvm
  • The output of the ddb> command show bcstats
  • The output of the mount command from your running machine, so we know what filesystems are mounted and how.
• ... XXX boot crash? XXX
• ... XXX show regs? XXX

Lost the panic message?

Under some circumstances, you may lose the very first message of a panic, stating the reason for the panic.

ddb> show panic
0:      kernel: page fault trap, code=0
ddb>

Note for SMP systems

You should get a trace from each processor as part of your report:

ddb{0}> trace
pool_get(d05e7c20,0,dab19ef8,d0169414,80) at pool_get+0x226
fxp_add_rfabuf(d0a62000,d3c12b00,dab19f10,dab19f10) at fxp_add_rfabuf+0xa5
fxp_intr(d0a62000) at fxp_intr+0x1e7
Xintr_ioapic0() at Xintr_ioapic0+0x6d
--- interrupt ---
idle_loop+0x21:
ddb{0}> machine ddbcpu 1
Stopped at      Debugger+0x4:   leave
ddb{1}> trace
Debugger(d0319e28,d05ff5a0,dab1bee8,d031cc6e,d0a61800) at Debugger+0x4
i386_ipi_db(d0a61800,d05ff5a0,dab1bef8,d01eb997) at i386_ipi_db+0xb
i386_ipi_handler(b0,d05f0058,dab10010,d01d0010,dab10010) at i386_ipi_handler+0x
4a
Xintripi() at Xintripi+0x47
--- interrupt ---
i386_softintlock(0,58,dab10010,dab10010,d01e0010) at i386_softintlock+0x37
Xintrltimer() at Xintrltimer+0x47
--- interrupt ---
idle_loop+0x21:
ddb{1}>

Repeat the machine ddbcpu x followed by trace for each processor in your machine.

How do I gather further information from a kernel crash?

A typical kernel crash on SecBSD might look like this:

kernel: page fault trap, code=0
Stopped at    pf_route+0x263:        mov     0x40(%edi),%edx
ddb>

This crash happened at offset 0x263 in the function pf_route.

The first command to run from the ddb(4) prompt is trace:

ddb> trace
pf_route(e28cb7e4,e28bc978,2,1fad,d0b8b120) at pf_route+0x263
pf_test(2,1f4ad,e28cb7e4,b4c1) at pf_test+0x706
pf_route(e28cbb00,e28bc978,2,d0a65440,d0b8b120) at pf_route+0x207
pf_test(2,d0a65440,e28cbb00,d023c282) at pf_test+0x706
ip_output(d0b6a200,0,0,0,0) at ip_output+0xb67
icmp_send(d0b6a200,0,1,a012) at icmp_send+0x57
icmp_reflect(d0b6a200,0,1,0,3) at icmp_reflect+0x26b
icmp_input(d0b6a200,14,0,0,d0b6a200) at icmp_input+0x42c
ipv4_input(d0b6a200,e289f140,d0a489e0,e289f140) at ipv4_input+0x6eb
ipintr(10,10,e289f140,e289f140,e28cbd38) at ipintr+0x8d
Bad frame pointer: 0xe28cbcac
ddb>

This tells us what function calls lead to the crash.

To find out the particular line of C code that caused the crash, you can do the following:

Find the source file where the crashing function is defined. In this example, that would be pf_route() in /sys/net/pf.c. Use objdump(1) to get the disassembly:

$ cd /sys/arch/$(uname -m)/compile/GENERIC
$ objdump -dlr obj/pf.o >/tmp/pf.dis

In the output, grep for the function name:

$ grep "<pf_route>:" /tmp/pf.dis
00007d88 <pf_route>:

Take this first hex number 7d88 and add the offset 0x263 from the Stopped at line:

$ printf '%x\n' $((0x7d88 + 0x263))
7feb

Scroll down to the line 7feb. The assembler instruction should match the one quoted in the Stopped at line. Then scroll up to the nearest C line number:

$ more /tmp/pf.dis
/sys/net/pf.c:3872
    7fe7:       0f b7 43 02             movzwl 0x2(%ebx),%eax
    7feb:       8b 57 40                mov    0x40(%edi),%edx
    7fee:       39 d0                   cmp    %edx,%eax
    7ff0:       0f 87 92 00 00 00       ja     8088 <pf_route+0x300>

So, it's precisely line 3872 of pf.c that crashes:

$ nl -ba /sys/net/pf.c | sed -n 3872p
  3872		if ((u_int16_t)ip->ip_len <= ifp->if_mtu) {

The kernel that produced the crash output and the object file for objdump must be compiled from the exact same source file, otherwise the offsets won't match.

If you provide both the ddb trace output and the relevant objdump section, that's very helpful.