Windows x86 compability on x64 operating system

Question

I know some about wow64 system, in x64 windows version, and i looked up into this wow64* dlls, to see how the actual syscall transition happen, but something i can't find, and this is probably done by windows kernel, is how the actual x86 assembler instruction processing works? I know that there are two selectors for process, 0x23 for x86, and 0x33 for x64, how does kernel handles the assembly code in 0x23 selector for the process ? Is there some sort of virtual machine, that emulates x86 instructions, using x64 instructions?

Answer 1

There is no need for emulation because x64 processors can execute x86 (32-bit) code natively using so-called compatibility mode. You only need to set up proper descriptors in the GDT (L=0 for 32-bit code, L=1 for 64-bit code) and it "just works". This is what Windows does. E.g. from this OSR post, the windows 7 x64 GDT has these entries:

GDT[0x04] = 32 bit ring 3 Code: Execute/Read
GDT[0x05] = 32 bit ring 3 Data: Read/Write, Accessed
GDT[0x06] = 64 bit ring 3 Code: Execute/Read, Accessed

Index 4 corresponds to the selector 0x23 (4*8=0x20) and 6 to 0x33 (6*8=0x30).

So by switching to selector 0x23 you can have the CPU execute your code as x86, and then go back to x64 by switching to 0x33.

wow64cpu.dll contains both 32-bit and 64-bit code and is responsible for most of house-keeping necessary when switching between x64 and x86. Kernel itself is x64 only AFAIK, 32-bit code is supported only in user mode.

For more details on Wow64 implementation see this nice article by George Nicolaou:

http://rce.co/knockin-on-heavens-gate-dynamic-processor-mode-switching/