Linux kernel debugging has always been a tough thing to do and requires pretty much a dedicated setup for regular effective debugging. But with the release of VMware 6, an inbuilt kernel debugger provided with it makes life pretty much easier in debugging the linux kernel for various requirements.

In case VMware 6 in Linux b0rks out for you with an error message:

neo@sauron ~ $ /opt/vmware/bin/vmware
/opt/vmware/lib/vmware/bin/vmware: symbol lookup error:
/opt/vmware/lib/vmware/lib/libvmwareui.so.0/libvmwareui.so.0: undefined
symbol: _ZN3Gtk13RecentManager11get_defaultEv

Try running VMware:

neo@sauron ~ $ VMWARE_USE_SHIPPED_GTK="yes" /opt/vmware/bin/vmware

To start with, do a regular installation of any GNU/Linux distro inside the VMware and recompile the kernel with debugging options enabled. My debug options looks something like this:

#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_UNUSED_SYMBOLS=y
CONFIG_DEBUG_KERNEL=y
CONFIG_LOG_BUF_SHIFT=15
CONFIG_DETECT_SOFTLOCKUP=y
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_VM=y
CONFIG_FRAME_POINTER=y
CONFIG_EARLY_PRINTK=y
CONFIG_4KSTACKS=y
CONFIG_DOUBLEFAULT=y

Make sure you have turned on Use of Frame pointers while compiling the kernel from "Kernel Hacking" option during the kernel configuration. This will help gdb in the debugging process.

To make VMware listen for debug connections to a TCP port in localhost just append the following line to your VMware Guest OS configuration file (.vmx):

debugStub.listen.guest32 = "1"

Now when you start your VMware guest OS, you will notice it is listening for a debug connection on 0.0.0.0:8832.
In order to make gdb do a source level debugging, you need to provide it with the entire linux source tree which was used to compile the Guest OS kernel including the uncompressed vmlinux binary.

So basically in the Host OS, you need to go to the directory where you copied the entire linux kernel source tree on which the Guest OS kernel was compiled and fire up gdb

BIG NOTE: I am *not* using the VMware console for accessing the Guest OS, instead I have logged into the Guest OS through SSH and do all the stuff required for debug/trigger actions. This is because when the debugger is attached from the Host OS to the Guest OS kernel running inside VMware, the kernel is interrupted by default untill you do continue in the debugger. In such scenario, if you are using the console, there is a high chance you will be trapped inside it since all keyboard actions will be trapped by the VMware and sent to the Guest OS kernel which is basically trapped by the debugger. In my case I got locked up and had to reboot.

neo@sauron ~/hacks/kern-debug $ gdb
GNU gdb 6.6
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu".
(gdb) file ./vmlinux
Reading symbols from /home/neo/hacks/kern-debug/vmlinux...done.
Using host libthread_db library "/lib/libthread_db.so.1".
(gdb) target remote 127.0.0.1:8832
Remote debugging using 127.0.0.1:8832
[New Thread 1]
warning: shared library handler failed to enable breakpoint
0xc0102c58 in irq_entries_start ()
(gdb) bt
#0  0xc0102c58 in irq_entries_start ()
#1  0xc010104e in default_idle () at include/asm/irqflags.h:57
#2  0xc01010ab in cpu_idle () at arch/i386/kernel/process.c:192
#3  0xc010033e in rest_init () at init/main.c:432
#4  0xc03fa737 in start_kernel () at init/main.c:620
#5  0x00000000 in ?? ()
(gdb)

From here, you can do almost everything you would do in a normal user space application debugging.

(gdb) break do_execve
Breakpoint 2 at 0xc014f117: file fs/exec.c, line 1133.
(gdb) c
Continuing.

Breakpoint 2, do_execve (filename=0xc5ceb000 "/usr/bin/ls", argv=0xc015096e,
envp=0xc4dc6f9c, regs=0xc4dc6fb8) at fs/exec.c:1133
1133            int retval;
(gdb)

Looks pretty neat!

But I think its pretty much necessary to mention about the awesome User Mode Linux and the KGDB patch. Using those you can pretty much have the same debugging experience, just that you need to give some effort in setting up and maintaing the environment.

VMware 6 Kernel Debugger Reference: http://stackframe.blogspot.org

posted at: 12:01 | path: / | permanent link to this entry