How to connect two Windows VMWare virtual machines over a virtual serial port for kernel debugging on a Linux host

Question

The problem: Recently I had to perform a kernel debugging on two MS windows VMWare virtual machines connected via a virtual serial port, and while running on a GNU/Linux host.

The layout: GNU/Linux host with Arch Linux distribution (even though the distro does not play a crucial role in this scenario), with VMWare Workstation 11 installed, hosting two MS Windows virtual machines - MS Windows 7SP1 x64 (the DEBUGGER), and MS Windows 8.0 x64 (the DEBUGGEE).

The solution: I did quite a lot of Googling to figure this out, and there was some information here and there, but I was no able to find a solution that would work for my case. Therefore, I have compiled my findings and a working step-by-step approach in the solution written below. Hopefully this will be also useful for someone else as well.

Answer 1

This is a step-by-step approach, excluding the actual host system installation, VMware setup and Virtual Machine creation, since these are rudimentary steps and anyone willing to perform kernel debugging should be capable doing. The approach is split into three parts - the VM hardware settings, MS Windows configuration, and establishing a debugging session.

I. VM hardware settings

1. Start with the DEBUGGER VM (the one from where you will be connecting to the DEBUGGEE to perform kernel debugging). In virtual machine settings hardware tab click 'Add...' to add new 'Serial port' device.

   

2. Set Serial port type to 'Output to socket'.

   

3. Give the name to the Socket (named pipe) a one you desire in a writeable location. I chose a '/tmp' folder and give the name 'com1', so that the full path of the socket is '/tmp/com1'. Since the DEBUGGER will connect to a DEBUGGEE specify the direction of connection as 'From: Client' and 'To: A Virtual Machine'. Also, make sure the device status is selected as 'Connect at power on' unless you require otherwise.

   

4. Once you click 'Finish' the DEBUGGER VM is prepared to use the serial connection.

   

5. Configure the DEBUGGEE VM (the one which will be debugged). In virtual machine settings hardware tab click 'Add...' to add new 'Serial port' device.Set Serial port type to 'Output to socket'. Give the name to the Socket exactly the same one as you set it for the DEBUGGER, in this case - '/tmp/com1'. Specify the direction of connection as 'From: Server' and 'To: A Virtual Machine'. Check that device status is selected as 'Connect at power on' unless you require otherwise.

   

6. For the I/O mode make sure that 'Yield CPU on poll' is checked. Once done, this will prepare your VM to use the serial connection, and both VMs should be able to communicate over that.

II. MS Windows configuration

7. Start you DEBUGGEE machine in order to enable debugging mode on a MS Windows system. Launch the command prompt 'cmd.exe' as Administrator and execute the 'bcdedit' command to view the current boot settings.

   

8. Execute (and adjust identifiers as needed) the following commands to create an additional boot option with debugging enabled:

   bcdedit /copy {current} /d "Windows 8 Debugging"

   bcdedit /set {ec04b3f3-16f4-11e6-b020-9ef119952d26} debug on

   bcdedit /set {ec04b3f3-16f4-11e6-b020-9ef119952d26} debugtype serial

   bcdedit /set {ec04b3f3-16f4-11e6-b020-9ef119952d26} debugport 1

   bcdedit /set {ec04b3f3-16f4-11e6-b020-9ef119952d26} baudrate 115200

   This should get your MS Windows DEBUGGEE be ready to boot in a debugging mode once restarted.

III. Establishing a debugging session

9. Start the DEBUGGER VM. Download and install the MS Windows SDK with debugging tools. The Microsoft Windows SDK for Windows 7 can be found here. Run the WinDbg and choose 'File/Kernel Debug (Ctrl+K)' to configure a debugging session over serial connection. In kernel Debugging settings COM tab, set baud rate to 115200 and port - com1 (or the name you specified). Once you click OK, the WinDbg will start waiting on com1 for incoming debugging sessions.

   

10. Start the DEBUGGEE VM. Make sure you choose the "Windows 8 Debugging" boot option to start the machine in the debugging mode.

11. On your DEBUGGER VM in WinDbg you should see an incoming kernel debugging connection. Select 'Debug/Break (Ctrl+Break)' to issue an interrupt and start debugging the kernel.

    

These steps should be enough to get the Kernel Debugging session over serial connection between two MS Windows VMs on a GNU/Linux host up and running.