2

The problem: Recently I had to perform a kernel debugging on two MS windows VMWare virtual machines connected via a virtual serial port, and while running on a GNU/Linux host.

The layout: GNU/Linux host with Arch Linux distribution (even though the distro does not play a crucial role in this scenario), with VMWare Workstation 11 installed, hosting two MS Windows virtual machines - MS Windows 7SP1 x64 (the DEBUGGER), and MS Windows 8.0 x64 (the DEBUGGEE).

The solution: I did quite a lot of Googling to figure this out, and there was some information here and there, but I was no able to find a solution that would work for my case. Therefore, I have compiled my findings and a working step-by-step approach in the solution written below. Hopefully this will be also useful for someone else as well.

  • this was on an old vmwareplayer (the physical machine that hosts the vms shouldn't matter theoratically) woodmann.com/forum/… – blabb Jul 25 '16 at 15:19
  • The named socket creation differs per physical host OS. And there are differences across VMware versions with how to set up the VM (debugger) to VM (debuggee) communication. – lockout Jul 25 '16 at 15:37
  • I followed your steps..but no output.Debugger m/c not able to connect debuggee machine. – Pranav Jun 19 '17 at 3:08
2

This is a step-by-step approach, excluding the actual host system installation, VMware setup and Virtual Machine creation, since these are rudimentary steps and anyone willing to perform kernel debugging should be capable doing. The approach is split into three parts - the VM hardware settings, MS Windows configuration, and establishing a debugging session.

I. VM hardware settings

  1. Start with the DEBUGGER VM (the one from where you will be connecting to the DEBUGGEE to perform kernel debugging). In virtual machine settings hardware tab click 'Add...' to add new 'Serial port' device.

    VMWare VM serial port

  2. Set Serial port type to 'Output to socket'.

    Output to socket

  3. Give the name to the Socket (named pipe) a one you desire in a writeable location. I chose a '/tmp' folder and give the name 'com1', so that the full path of the socket is '/tmp/com1'. Since the DEBUGGER will connect to a DEBUGGEE specify the direction of connection as 'From: Client' and 'To: A Virtual Machine'. Also, make sure the device status is selected as 'Connect at power on' unless you require otherwise.

    Named pipe

  4. Once you click 'Finish' the DEBUGGER VM is prepared to use the serial connection.

    Debugger ready

  5. Configure the DEBUGGEE VM (the one which will be debugged). In virtual machine settings hardware tab click 'Add...' to add new 'Serial port' device.Set Serial port type to 'Output to socket'. Give the name to the Socket exactly the same one as you set it for the DEBUGGER, in this case - '/tmp/com1'. Specify the direction of connection as 'From: Server' and 'To: A Virtual Machine'. Check that device status is selected as 'Connect at power on' unless you require otherwise.

    Debuggee named pipe

  6. For the I/O mode make sure that 'Yield CPU on poll' is checked. Once done, this will prepare your VM to use the serial connection, and both VMs should be able to communicate over that. Debuggee ready

II. MS Windows configuration

  1. Start you DEBUGGEE machine in order to enable debugging mode on a MS Windows system. Launch the command prompt 'cmd.exe' as Administrator and execute the 'bcdedit' command to view the current boot settings.

    bcdedit

  2. Execute (and adjust identifiers as needed) the following commands to create an additional boot option with debugging enabled:

    bcdedit /copy {current} /d "Windows 8 Debugging"

    bcdedit /set {ec04b3f3-16f4-11e6-b020-9ef119952d26} debug on

    bcdedit /set {ec04b3f3-16f4-11e6-b020-9ef119952d26} debugtype serial

    bcdedit /set {ec04b3f3-16f4-11e6-b020-9ef119952d26} debugport 1

    bcdedit /set {ec04b3f3-16f4-11e6-b020-9ef119952d26} baudrate 115200

    This should get your MS Windows DEBUGGEE be ready to boot in a debugging mode once restarted.

III. Establishing a debugging session

  1. Start the DEBUGGER VM. Download and install the MS Windows SDK with debugging tools. The Microsoft Windows SDK for Windows 7 can be found here. Run the WinDbg and choose 'File/Kernel Debug (Ctrl+K)' to configure a debugging session over serial connection. In kernel Debugging settings COM tab, set baud rate to 115200 and port - com1 (or the name you specified). Once you click OK, the WinDbg will start waiting on com1 for incoming debugging sessions.

    WinDbg Setup

  2. Start the DEBUGGEE VM. Make sure you choose the "Windows 8 Debugging" boot option to start the machine in the debugging mode.

  3. On your DEBUGGER VM in WinDbg you should see an incoming kernel debugging connection. Select 'Debug/Break (Ctrl+Break)' to issue an interrupt and start debugging the kernel.

    Kernel debugging connection

These steps should be enough to get the Kernel Debugging session over serial connection between two MS Windows VMs on a GNU/Linux host up and running.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.