how do I debug full screen applications

Question

I need to debug an old (1999) full screen application. When I window the application and attach olly the program crashes.

I have heard of remote debugging. Would running the program on a VM and then attaching windbg to the application on the remote machine do the trick?

Answer 1

you can use windbg like an user-mode debugger from a kernel connection

open the vm
install windbg in vm
install the full screen game in vm

connect the vm to a kernel debugger in host

execute ntsd -d {the fullscreen app in vm}

the kernel debugger in host will kick to life as a plain user-mode debugger debugging the app ( the syntax for continuing is kinda tricky refer windbg manual for syntax of using ntsd -d )

Answer 2

I usually do one of the following when dealing with full screen software:

1. As blabb said setting up a remote debugger is an option, although I usually find it a bit slow and annoying to use and set up.
2. Occasionally there's a configuration option to switch to windowed mode. Its not always there but when it is its the best choice.
3. Since you are already debugging and reversing it, you can always hook it's fullscreen request API (usually ChangeDisplaySettings or ChangeDisplaySettingsEx on Windows).
4. If you can't hook or find the fullscreen API (if the software has anti-debugging it might be kind of a chicken and egg problem), you can always call ChangeDisplaySettings yourself and disable fullscreen mode after getting a window handle for the program's fullscreen window.
5. Or you could simply use one of many existing tools for exactly that, they're covered in this SO question: https://superuser.com/questions/318748/force-fullscreen-games-to-in-window-mode

Hope that helps :)