Nothing like asking a question on stackexchange, only to be humiliated by finding the answer (or at least part of it). After finding the following source file, it started making sense:
On old 8086 machines, where there is no trap for invalid instructions, the Elders of the Past came up with an emulation strategy. All x87 instructions are in the
DF range (8 possible values) followed by modrm and other goodness. If you prefix the instruction with a
9B), you guarantee that there always be two bytes of code before the modrm byte, looking something like
9B Dx. However, instead of emitting those two bytes, the compiler emits
CD xx, where xx ranges
3B (8 possible values). As we all know, CD is the encoding of the x86
When the CPU executes the
int instruction and arrives at the handler for
3B, it vectors off to the interrupt handler. If there isn't an x87 coprocessor available, the handler will emulate the floating point instruction, maintaining the coprocessor state in memory. If however there is an x87 coprocessor present, the handler will peek at the return stack to see where the
int instruction is located, and overwrite it with the appropriate
9B Dx byte sequence, corresponding to the
CD 3x byte sequence. It then returns control to the patched instruction so that it gets executed. Now that it has been patched, the instruction is an actual FPU instruction, and future executions of the instructions will no longer take the long detour through the emulator.
The documentation for how to deal with interrupt
3E is still not forthcoming. However, for the time being, I have enough information to implement x87 emulation support in the Reko decompiler.