Windows 7x86 System Freezes when running assembly program with int 3h

Question

Recently was doing some pydbg testing, so I had to remove the AeDebug entry (which determines the JIT debugger) in the registry, and Windbg was (and is) my JIT debugger. Exported the key before deleting it. It was:

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug

and the values were:

(Default) REG_SZ (value not set)
Auto REG_SZ 1
Debugger REG_SZ "C:\...\windbg.exe" -p %ld -e %ld -g
UserDebuggerHotKey REG_DWORD 0x00000000 (0)

Now I had to test some other stuff in Windbg, so I added the exported AeDebug key in the .reg file.

However, when I attempt to run a few test assembly programs that contact int 3h/0xCC (which should make Windbg come up as my JIT debugger) instead my system freezes with no BSOD, and I have to power off/turn back on.

I've deleted all of AeDebug and re-ran "windbg -I" to register it as my JIT debugger. However, I still get system freezes!

Please help! It only occurs so far when code Im running contains an int 3h breakpoint in it.

I tried running an assembly (masm32) program that I compiled with the following to test:

.586
.model flat, stdcall

option casemap:none

includelib \masm32\lib\kernel32.lib
includelib \masm32\lib\user32.lib

include \masm32\include\kernel32.inc
include \masm32\include\user32.inc
include \masm32\include\windows.inc

.code
start:
xor eax, eax
xor ebx, ebx
xor ecx, ecx
xor edx, edx
int 3h
invoke ExitProcess, 0
END start

However, the system still freezes. When I run windbg first and I step out of the breakpoint, it exits like normal but doesn't feeze my whole system.

How can I get normal breakpoints again that dont cause undefined behavior?

Answer 1

This occurs when you have kernel debugging enabled, but have not attached a kernel debugger.

In short the freeze occurs when,

• Kernel debugging is enabled.
• A kernel debugger is not attached. (If it was, control would be transferred to it).
• Jit debugger is enabled/disabled (doesn't matter).
• App is not being actively debugged by a usermode debugger like WindDbg or OllyDbg.

For Windows XP, see the contents of boot.ini. If boot.ini has a /debug flag it means kernel debugging is enabled.

For Windows 7 and above you can use bcdedit or msconfig. You can also use the command bcdedit | findstr debug to check if debug flag has been enabled.

To fix the problem, just remove the debug flag from your boot entry.

Answer 2

It wasn't clear from your question if the x86 in the title is concerning the processor or the process.

If you are actually running on a 64-bit processor/OS, but running a 32-bit app, then you possibly set the wrong AeDebug key. There is one for 32-bit applications and one for 64-bit applications. Make sure you are setting both of them to the appropriate version of windbg.

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug
HKLM\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\AeDebug