The technique of jumping to 64bit code from a 32bit WOW64-ed process is commonly called "Heaven's gate" when performed manually. This is usually done to use 64bit features (such as manipulating 64bit processes by calling 64bit versions of windows APIs) or by malware to make debugging more difficult, which is coincidentally what you seem to be experiencing ;).
Searching for that term online may yield more results.
Most user-mode debuggers indeed don't handle that transition well for multiple reasons (one being the debugger assumes a 32bit process, while you're now executing 64bit code, another being user-mode debugging APIs don't support that).
TL;DR: The "solution" for that situation is to debug using a debugger that explicitly made the effort to support 32 and 64 bit interleaved processes. Although this is often easier for a kernel mode debugger), such as windbg, or other debuggers that support both 32 and 64 bit modes (
x64dbg should be able to do it, although I never tried As mentioned in the comments x64dbg is unable to do that).
A brief explanation of Heaven's gate
There's a special DLL loaded into 32bit processes on 64bit windows environments, this DLL is
wow64cpu.dll. This DLL is responsible for most of the WOW64 magic, and specifically for implementing the transition from 32 to 64 bit inside 32 bit processes (WOW64-ed processes).
This is mandatory because as the operating system is natively 64 bit, all utilities, APIs and low level functionalities are implemented using 64 bit code (otherwise what's the point of having 64 bit OSes??). Therefore, every time an WOW64-ed process requires a OS assistance, it must first translate from 32bit CPU mode to 64 bit CPU mode.
So, how's the transition done, and what's that wow64cpu.dll you mentioned got to do with it?
To make a long story short, the value at
fs:[0c0h] is set to an address inside
wow64cpu.dll. This field is called
WOW32Reserved and it points to a far jump to a specific address, using the
033 segment. Changing the segment selector to
23, which is used for 32bit code) does not change the code selector or the base addressing, you're merely changing the GDT entry used to execute the target code, specifically - you replace the 4th GDT with the 6th.
GDT entries 4 and 6 only differ in a couple of flags being set - those controlling whether the CPU is executing in 16, 32, and 64 modes (well, those GDT entries only have flags set for 32 and 64 bit modes, but transitioning to 16 bit can be achieved in a similar manner).
Where to go on from here?
As this is a big and complicated topic, I prefer redirecting you to relevant articles over touching the low-level details in here more than I already did. Here are several useful articles, on the more theoretical side of things:
- Explanation of the "weird jump to
- Explanation of the entire WOW64 mechanism from the process's perspective, including how to debug it to see it happening.
- A very brief explanation of the flow executed, and the article historically keying the term "Heaven's gate".
- Another in-depth explanation.
- The Windows Internals book series.
Additionally, here are several actual open source implementations of natively executing 64 bit code inside
WOW64-ed 32 bit process:
- This is a code snippet that implements some direct 64 bit OS operations.
- This is a full blown heaven's gate implementation.
- This is another, later, full blown heaven's gate implementation.