<html>
<title>
data
</title>
<body BGCOLOR="#FFFFFF" TEXT="#000000" LINK="#0000FF" VLINK="#330088" ALINK="#FF0044">
<center><H1>Multiprocessor Kernel Debugging Using Acid
</H1>
<DL><DD><I>Steven Stallion<br>
stallion@coraid.com<br>
</center></I></DL>
<H4>Introduction
</H4>
<br> <br>
This document describes a method of debugging multiprocessor kernels
using
<A href="/magic/man2html/1/acid"><I>acid</I>(1).
</A>While in-situ debugging is yet to be fully realized, the mechanism
detailed herein provides superior postmortem visibility over previous
approaches.
It is expected that this body of work will result in a debugging
environment supportive of both in-situ and postmortem analysis in the
future.
This approach encourages use of existing tools and culminates in a new
<I>acid</I>
library, which provides a foundation for discovery and analysis of
multiprocessor related defects.
<br> <br>
The reader should not consider this document an introductory text but
merely an exposition of completed work [1].
</center><H4>Mechanism
</H4>
<br> <br>
A panicking kernel will issue a nonmaskable inter-processor interrupt
(IPI) to all processors, excluding the caller.
Upon receiving the trap, each
<TT>Mach</TT>
records a pointer to the current
<TT>Ureg</TT>
structure and the stack pointer to the
<TT>dbgreg</TT>
and
<TT>dbgsp</TT>
members, respectively.
After which, the processor enters a halt state with interrupts
disabled.
The debugging processor will maintain a value of zero for both members
as a hint to use current register state at debug time.
Use of a nonmaskable interrupt yields the ability to record state
regardless of processor context; interrupt processing no longer
interferes with stack generation.
<br> <br>
If
<TT>consdebug</TT>
is non-nil,
<TT>panic</TT>
will branch to a debug routine, typically
<TT>rdb</TT>.
The system is then prepared to accept remote debugging commands,
ostensibly via
<A href="/magic/man2html/4/rdbfs"><I>rdbfs</I>(4).
</A>To simplify setting
<TT>consdebug</TT>
at boot, a new configuration variable, named
<TT>rdb</TT>
was introduced to obviate the need for the
<TT>^T^Td</TT>
control sequence (see
<A href="/magic/man2html/3/cons"><I>cons</I>(3)).
</A><br> <br>
To further simplify remote debugging,
<TT>rdb</TT>
was modified to return if a serial break is received.
<I>Rdbfs</I>
was also updated to send a serial break upon receiving a kill
message, allowing
<I>acid</I>
to terminate the debugging session and transitively reboot the system
using the
<TT>kill</TT>
builtin function.
</center><H4>Using the Library Functions
</H4>
<br> <br>
The
<TT>mach</TT>
library provides a number of functions the user may employ to examine
multiprocessor state.
<TT>Mach</TT>
was developed as a companion to the
<TT>kernel</TT>
library.
As such, both libraries must be defined on the command line when
starting
<I>acid</I>.
Normal
<TT>kernel</TT>
initialization rules apply;
<TT>kinit</TT>
must be called to establish the proper mapping for the kernel prior to
calling functions defined in
<TT>mach</TT>.
<br> <br>
The following example attaches to a remote kernel with
<I>rdbfs</I>
and gathers basic information using the
<TT>mach</TT>
library:
<DL><DT><DD><TT><PRE>
% rdbfs /mnt/consoles/sys
attach /mnt/consoles/sys
% acid -k -l kernel -l mach -r 9pc
9pc:386 plan 9 boot image
/sys/lib/acid/port
/sys/lib/acid/386
/sys/lib/acid/kernel
/sys/lib/acid/mach
acid: kinit()
rc("cd /sys/src/9/pc; mk proc.acid")
include("/sys/src/9/pc/proc.acid")
acid: rc("cd /sys/src/9/pc; mk proc.acid")
8c -FTVw -a -I. ../port/proc.c >proc.acid
acid: include("/sys/src/9/pc/proc.acid")
acid: machno()
0
acid: machs()
0x80017000 0 up 0x00000000
0x8003d000 1 up 0x00000000
</PRE></TT></DL>
</center><H4>Library Functions
</H4>
<br> <br>
The
<TT>mach</TT>
library is located in the directory
<TT>/sys/lib/acid</TT>.
As with other libraries, these functions may be overridden,
personalized, or added to by code defined in
<TT></TT><I>home/lib/acid</I>.
The implementation of these functions can be examined using the
</TT><TT>whatis</TT>
operator and then modified during debugging sessions.
</TT><br> <br>
<DL>
<DT><DT> <DD>
<TT>mach</TT>
prints a one line summary for the given
<I>Mach</I>.
The first printed column is the address of
<I>Mach</I>,
followed by the
<TT>machno</TT>,
and ends with a summary of the currently scheduled process (see
<TT>proc</TT>).
<DT><DT> <DD>
<DL><DT><DD><TT><PRE>
<br>
acid: mach(machp[1])
0x8003d000 1 up 0x00000000
</PRE></TT></DL>
<br>
</dl>
<br> <br>
<DL>
<DT><DT> <DD>
<TT>machgpr</TT>
prints the general purpose registers for the given
<I>Mach</I>.
While
<TT>machgpr</TT>
may be used interactively, this function is typically only called by
<TT>machregs</TT>.
<DT><DT> <DD>
<DL><DT><DD><TT><PRE>
<br>
acid: machgpr(machp[1])
AX 0xc9121c18 BX 0x8025f34c CX 0x000000d8 DX 0x00000000
DI 0x80279b44 SI 0x0005d203 BP 0x80016000
</PRE></TT></DL>
<br>
</dl>
<br> <br>
<DL>
<DT><DT> <DD>
<TT>machlstk</TT>
produces a long format stack trace for the given
<I>Mach</I>,
similar to
<TT>lstk</TT>.
Unlike
<TT>lstk</TT>,
the
<TT>:</TT>
operator should not be used to address variables on processors other
than the debugging
<TT>Mach</TT>
(see
<TT>machno</TT>).
This is a limitation imposed by the current implementation of
<I>acid</I>
and
<I>libmach</I>.
<DT><DT> <DD>
<DL><DT><DD><TT><PRE>
<br>
acid: machlstk(machp[1])
runproc()+0x53 proc.c:531
start=0xc9121c18
p=0x80279948
rq=0x8025f34c
i=0x5d203
sched()+0x165 proc.c:164
p=0x80279948
schedinit()+0x90 proc.c:107
squidboy(apic=0x8026b144)+0x96 mp.c:421
0x80003091 ?file?:0
</PRE></TT></DL>
<br>
</dl>
<br> <br>
<DL>
<DT><DT> <DD>
<TT>machno</TT>
prints the number of the debugging
<TT>Mach</TT>.
<DT><DT> <DD>
<DL><DT><DD><TT><PRE>
<br>
acid: machno()
0
</PRE></TT></DL>
<br>
</dl>
<br> <br>
<DL>
<DT><DT> <DD>
<TT>machregs</TT>
prints the contents of both the general and special purpose registers
for the given
<I>Mach</I>.
<TT>machregs</TT>
calls
<TT>machspr</TT>
then
<TT>machgpr</TT>
to display the contents of the registers.
<DT><DT> <DD>
<DL><DT><DD><TT><PRE>
<br>
acid: machregs(machp[1])
PC 0x80196e10 runproc+0x53 proc.c:531
SP 0x80016f80 ECODE 0x801006f8 EFLAG 0x00000202
CS 0x00000010 DS 0x80010008 SS 0x0003b000
GS 0x0000001b FS 0x0000001b ES 0x00000008
TRAP 0x00000002 nonmaskable interrupt
AX 0xc9121c18 BX 0x8025f34c CX 0x000000d8 DX 0x00000000
DI 0x80279b44 SI 0x0005d203 BP 0x80016000
</PRE></TT></DL>
<br>
</dl>
<br> <br>
<DL>
<DT><DT> <DD>
<TT>machs</TT>
prints summaries for all
<TT>Mach</TT>s
in the system.
<DT><DT> <DD>
<DL><DT><DD><TT><PRE>
<br>
acid: machs()
0x80017000 0 up 0x00000000
0x8003d000 1 up 0x00000000
</PRE></TT></DL>
<br>
</dl>
<br> <br>
<DL>
<DT><DT> <DD>
<TT>machspr</TT>
prints the special purpose registers for the given
<I>Mach</I>.
While
<TT>machspr</TT>
may be used interactively, this function is typically only called by
<TT>machregs</TT>.
<DT><DT> <DD>
<DL><DT><DD><TT><PRE>
<br>
acid: machspr(machp[1])
PC 0x80196e10 runproc+0x53 proc.c:531
SP 0x80016f80 ECODE 0x801006f8 EFLAG 0x00000202
CS 0x00000010 DS 0x80010008 SS 0x0003b000
GS 0x0000001b FS 0x0000001b ES 0x00000008
TRAP 0x00000002 nonmaskable interrupt
</PRE></TT></DL>
<br>
</dl>
<br> <br>
<DL>
<DT><DT> <DD>
<TT>machstacks</TT>
prints a stack trace for all
<TT>Mach</TT>s
in the system, similar to
<TT>stacks</TT>.
<DT><DT> <DD>
<DL><DT><DD><TT><PRE>
<br>
acid: machstacks()
=====================================================
0x80017000 0 up 0x00000000
runproc()+0x14d proc.c:530
sched()+0x165 proc.c:164
schedinit()+0x90 proc.c:107
main()+0x158 main.c:130
idle l.s:233
=====================================================
0x8003d000 1 up 0x00000000
runproc()+0x53 proc.c:531
sched()+0x165 proc.c:164
schedinit()+0x90 proc.c:107
squidboy(apic=0x8026b144)+0x96 mp.c:421
0x80003091 ?file?:0
</PRE></TT></DL>
<br>
</dl>
<br> <br>
<DL>
<DT><DT> <DD>
<TT>machstk</TT>
produces a short format stack trace for the given
<I>Mach</I>,
similar to
<TT>stk</TT>.
Unlike
<TT>stk</TT>,
the
<TT>:</TT>
operator should not be used to address variables on processors other
than the current
<TT>Mach</TT>
(see
<TT>machno</TT>).
This is a limitation imposed by the current implementation of
<I>acid</I>
and
<I>libmach</I>.
<DT><DT> <DD>
<DL><DT><DD><TT><PRE>
<br>
acid: machstk(machp[1])
runproc()+0x53 proc.c:531
sched()+0x165 proc.c:164
schedinit()+0x90 proc.c:107
squidboy(apic=0x8026b144)+0x96 mp.c:421
0x80003091 ?file?:0
</PRE></TT></DL>
<br>
</dl>
<br> <br>
<DL>
<DT><DT> <DD>
<TT>machunwind</TT>
dumps the contents of the stack for the given
<I>Mach</I>.
This is a function of last resort; it is primarily used to debug the
above functions.
<DT><DT> <DD>
<DL><DT><DD><TT><PRE>
<br>
acid: machunwind(machp[1])
...
0x8003dfd8: schedinit+0x90
0x8003dfdc: 0x80279948
0x8003dfe0: microdelay+0x3c
0x8003dfe4: 0x32e8
0x8003dfe8: 0x0
0x8003dfec: squidboy+0x96
0x8003dff0: 0x64
0x8003dff4: 0x0
0x8003dff8: 0x80003091
0x8003dffc: mplapic+0xb0
</PRE></TT></DL>
<br>
</center></dl>
<H4>Support Functions
</H4>
<br> <br>
These functions provide utility to other library functions.
<br> <br>
<DL>
<DT><DT> <DD>
<TT>machaddr</TT>
converts the given
<I>integer</I>
address to a global address.
If the address is mapped to processor-local memory,
<TT>machaddr</TT>
will provide an alternative that can be addressed by any processor.
This function is idempotent; any address, regardless of mapping, may
be passed to this function.
<DT><DT> <DD>
<DL><DT><DD><TT><PRE>
<br>
acid: MACHADDR = KZERO+0x16000;
acid: print(machaddr(machp[1], MACHADDR)\X)
0x8003d000
</PRE></TT></DL>
<br>
</dl>
<br> <br>
<DL>
<DT><DT> <DD>
<TT>machpc</TT>
provides the program counter for the given
<I>Mach</I>.
<DT><DT> <DD>
<DL><DT><DD><TT><PRE>
<br>
acid: print(machpc(machp[1])\X)
0x80196e10
</PRE></TT></DL>
<br>
</dl>
<br> <br>
<DL>
<DT><DT> <DD>
<TT>machsp</TT>
provides the stack pointer for the given
<I>Mach</I>.
<DT><DT> <DD>
<DL><DT><DD><TT><PRE>
<br>
acid: print(machsp(machp[1])\X)
0x80016f80
</PRE></TT></DL>
<br>
</dl>
<br> <br>
<DL>
<DT><DT> <DD>
<TT>machureg</TT>
provides the
<I>Ureg</I>
structure for the given
<I>Mach</I>.
<DT><DT> <DD>
<DL><DT><DD><TT><PRE>
<br>
acid: print(machureg(machp[1])\X)
0x8003df3c
</PRE></TT></DL>
<br>
</center></dl>
<H4>Redefined Functions
</H4>
<br> <br>
A handful of functions defined in other modules are redefined to
augment behavior on multiprocessors.
This necessitates the
<TT>mach</TT>
library be defined after augmented libraries on the command line.
<br> <br>
<DL>
<DT><DT> <DD>
<TT>proclstk</TT>
produces a long format stack trace for the given
<I>Proc</I>,
similar to
<TT>lstk</TT>.
Unlike
<TT>lstk</TT>,
the
<TT>:</TT>
operator should not be used to address variables unless the process is
scheduled on the current
<TT>Mach</TT>
(see
<TT>machno</TT>).
This is a limitation imposed by the current implementation of
<I>acid</I>
and
<I>libmach</I>.
This function was added to supplement the redefined
<TT>procstk</TT>
function below.
<DT><DT> <DD>
<DL><DT><DD><TT><PRE>
<br>
acid: proclstk(0x8027ca08)
gotolabel(label=0x80016030)+0x0 l.s:1000
sched()+0x160 proc.c:164
p=0x286
sysrendezvous(arg=0x8027cc64)+0x143 sysproc.c:836
rendval=0x0
tag=0x624bc
l=0x8b1ce87c
val=0x7c86d798
p=0x8027d3c8
syscall(ureg=0x8b20a1a4)+0x238 trap.c:726
sp=0xcfffee9c
scallnr=0x22
startns=0x0
ret=0xffffffff
i=0x1
stopns=0x0
s=0x0
_syscallintr()+0x18 plan9l.s:44
0x8b20a1a4 ?file?:0
</PRE></TT></DL>
<br>
</dl>
<br> <br>
<DL>
<DT><DT> <DD>
<TT>procstk</TT>
produces a short format stack trace for the given
<I>Proc</I>,
similar to
<TT>stk</TT>.
Unlike
<TT>stk</TT>,
the
<TT>:</TT>
operator should not be used to address variables unless the process is
scheduled on the current
<TT>Mach</TT>
(see
<TT>machno</TT>).
This is a limitation imposed by the current implementation of
<I>acid</I>
and
<I>libmach</I>.
This function overrides the definition in
<TT>kernel</TT>.
This was necessary as
<TT>kernel</TT>
assumes any given process terminates in a call to
<TT>gotolabel</TT>.
This is not always the case on a multiprocessor where the process may
be actively scheduled at debug time.
<DT><DT> <DD>
<DL><DT><DD><TT><PRE>
<br>
acid: procstk(0x8027ca08)
gotolabel(label=0x80016030)+0x0 l.s:1000
sched()+0x160 proc.c:164
sysrendezvous(arg=0x8027cc64)+0x143 sysproc.c:836
syscall(ureg=0x8b20a1a4)+0x238 trap.c:726
_syscallintr()+0x18 plan9l.s:44
0x8b20a1a4 ?file?:0
</PRE></TT></DL>
<br>
</dl>
<br> <br>
<DL>
<DT><DT> <DD>
<TT>reason</TT>
uses machine-dependent information to generate a string explaining why
a
<TT>Mach</TT>
has stopped.
The
<I>integer</I>
argument is the value of an architecture dependent status register.
This function overrides the builtin definition.
This was necessary as the builtin function would discard the
<I>integer</I>
argument and always consult the status register on the debugging
<TT>Mach</TT>.
<DT><DT> <DD>
<DL><DT><DD><TT><PRE>
<br>
acid: print(reason(machureg(machp[1]).trap))
nonmaskable interrupt
</PRE></TT></DL>
<br>
</center></dl>
<H4>Future Work
</H4>
<br> <br>
Only
<TT>pc</TT>
kernels are supported.
<br> <br>
In-situ debugging is not supported.
<br> <br>
Floating point is not supported.
This is further complicated by support for XSAVE/XRESTOR and
YMM register state in
<TT>pc</TT>
kernels.
<br> <br>
<I>Acid</I>
and its constituent libraries assume uniprocessor, which causes
complications in kernel context. Use of redefined functions have
largely addressed these issues.
<br> <br>
Users must be aware of processor-local address ranges.
A good understanding of kernel memory mapping is essential.
The addition of per-processor address translation to
<I>acid</I>
and
<I>libmach</I>
could ease this burden considerably.
<br> <br>
The somewhat portable nature of the
<TT>kernel</TT>
library is at odds with the
<TT>pc</TT>-specific
<TT>mach</TT>
library.
Once remaining portability issues are resolved, both libraries could
be merged.
</center><H4>Acknowledgements
</H4>
<br> <br>
Portions of this document use descriptions and formatting from the
``Acid Manual'', by Phil Winterbottom.
</center><H4>References
</H4>
<DL COMPACT>
<DT>[1]<DD>
P. Winterbottom,
``Acid: A Debugger Built from A Language'',
USENIX Proc. of the Winter 1994 Conf.,
San Francisco, CA.
</dl>
<br> <br>
<A href=http://www.lucent.com/copyright.html>
Copyright</A> © 2019 Alcatel-Lucent Inc. All rights reserved.
</body></html>
|
