What is the protocol for x87 floating point emulation in MS-DOS?

Question

I'm trying to get useful results using the Reko decompiler on a dusty old MS-DOS binary compiled with Borland C++ that appears to be performing a lot of floating point arithmetic. I'm seeing code sequences like

mov ax,0x4D8C    ; segment selector
mov es,ax
int 0x3C         ; call x87 emulator??
fld dword ptr [<some address>]
sub sp,8
int 0x39         ; call x87 emulator??

...etc. A cursory search engine search strongly hints that the int instructions are invoking an x87 emulation library; when the x87 is present, it lets the coprocessor execute the instruction, but when it isn't, the emulator emulates.

I am well familiar with how to implement FPU operations, with shifts and whatnot. What I'd like to find out more about is the protocol of these invocations to the emulator. I have been unable to locate documentation. Perhaps one of the members of the RE community does?

Answer 1

Nothing like asking a question on stackexchange, only to be humiliated by finding the answer (or at least part of it). After finding the following source file, it started making sense:

https://github.com/alexhenrie/wine/blob/master/dlls/krnl386.exe16/fpu.c

On old 8086 machines, where there is no trap for invalid instructions, the Elders of the Past came up with an emulation strategy. All x87 instructions are in the D8-DF range (8 possible values) followed by modrm and other goodness. If you prefix the instruction with a FWAIT (opcode 9B), you guarantee that there always be two bytes of code before the modrm byte, looking something like 9B Dx. However, instead of emitting those two bytes, the compiler emits CD xx, where xx ranges 34-3B (8 possible values). As we all know, CD is the encoding of the x86 int instruction.

When the CPU executes the int instruction and arrives at the handler for 34-3B, it vectors off to the interrupt handler. If there isn't an x87 coprocessor available, the handler will emulate the floating point instruction, maintaining the coprocessor state in memory. If however there is an x87 coprocessor present, the handler will peek at the return stack to see where the int instruction is located, and overwrite it with the appropriate 9B Dx byte sequence, corresponding to the CD 3x byte sequence. It then returns control to the patched instruction so that it gets executed. Now that it has been patched, the instruction is an actual FPU instruction, and future executions of the instructions will no longer take the long detour through the emulator.

The documentation for how to deal with interrupt 3E is still not forthcoming. However, for the time being, I have enough information to implement x87 emulation support in the Reko decompiler.