[nSLUG] kernel debugging?

Peter Cordes peter at llama.nslug.ns.ca
Thu Oct 23 21:11:56 ADT 2003

On Tue, Oct 21, 2003 at 06:21:10PM -0300, Dop Ganger wrote:
> On Mon, 20 Oct 2003, Peter Cordes wrote:
> >  Anyway, there's supposed to be a way to run GDB on the kernel, right?
> > What tools are there to examine kernel data structures?  The processes are
> > stuck, so I don't need to single-step the execution, just look at variables
> > in memory.  I have multiple computers in the cluster I admin at Dal, so
> > remote-debugging stuff is an option.
> OProfile (http://oprofile.sourceforge.net) has usually done the trick for
> me and might be worth looking at as a first line of attack. It's a
> profiler and lets you see what's being called and is quite handy.
> Recompile the kernel with profiling support, install oprofile, and off you
> go.

 I hadn't thought of using oprofile for debugging.  Hmmm.  It's in the
kernel.org tree in 2.6, BTW.

> Digging deeper, there's kdb (http://oss.sgi.com/projects/kdb/) and kgdb
> (http://kgdb.sourceforge.net/), neither of which I've had to use for a
> couple of years or so, so I'm probably not the best person to say which to
> go for. kgdb runs from a second machine, and kdb runs on the machine being
> diagnosed. I *think* that kdb is more likely to be what you need, although
> kgdb is more powerful.

 Thanks for the suggestions.  I actually compiled a kernel with kdb, but it
turned out that I didn't need it (yet).  I noticed that there are
instructions for using GDB on a running kernel at the bottom of

compile a kernel with debugging symbols (add -g to CFLAGS)
save the vmlinux object file (hit ^Z at the right time with make-kpkg...) 
boot the kernel
gdb vmlinux /proc/kcore

Then you can do stuff like
print *init_tasks[0]
p *init_tasks[0].next_task.next_task...
(the index for init_tasks is the SMP processor ID.)

 You can keep tacking on .next_task until you find the process you're looking
for in the kernel's circularly linked task list.  (they're in the order ps
lists them in).

 Is there a kernel data structure that lets you look up stuff about a pro
when you know the PID?  And how do I find out stuff about file
descriptors/pipes?  I want to check on the data structures associated with a
pipe that cat is reading from on fd 0.  I suspect that the reader thinks it
is empty and the writer thinks it is full, or something like that, because
of a race condition.  (nobody on the openMosix mailing lists has seen this
problem with non-SMP nodes, but everybody with SMP nodes running 2.4.22 can
reproduce it.)

 Using gdb is read-only, so no single stepping or breakpoints. That's what
kdb and kgdb are for.  However, kdb only runs on the console or a serial
port, and I don't really want to sit in the noisy machine room with no chair
all day.  (I hooked up a null modem cable between two of the cluster
machines, so I'll be able to ssh in and run minicom on one of them to debug
the other one, if I eventually want to use kdb :)  This way of using gdb is
what I'd been thinking of when I posted my question, BTW.

#define X(x,y) x##y
Peter Cordes ;  e-mail: X(peter at cor , des.ca)

"The gods confound the man who first found out how to distinguish the hours!
 Confound him, too, who in this place set up a sundial, to cut and hack
 my day so wretchedly into small pieces!" -- Plautus, 200 BC
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 351 bytes
Desc: Digital signature
URL: <http://nslug.ns.ca/mailman/private/nslug/attachments/20031023/6b51bf1f/attachment.pgp>

More information about the nSLUG mailing list