OllyDbg FPU anti-debug

Question

While playing around with FPU instructions I discovered an anti-debugging trick for OllyDbg. I haven't found it in popular references so far.

First of all here it is.

fnsave [esp-100h]
cmp word ptr [esp-0EEh], 07FFh      ; 07FFh (all bits set) in Olly 2
                                    ; 0000h in Olly 1
                                    ; something different otherwise
je @being_debugged

The code fragment checks the "Last Instruction Opcode" at +0x12. According to Intel manuals it's set if and only if an exception occured.

The value is set only once, it doesn't work when restarting (<<) the program.

But I don't know where the values come from. When Olly calls GetThreadContext() of debuggee, the value is already set. Other debuggers does not seem to behave like this.

Can anyone confirm this? Tested on Win XP SP3, Intel P4.

Answer 1

Last Instruction Opcode is set mainly for the use of exception handlers, but it's set for every non-control FPU instruction (that is, set by loads, stores, compares, etc).

The value is non-zero on my system, but it is different value, and it's not constant.

Olly 1 doesn't request as many context fields as Olly 2 does, and that's part of the reason why the values are different. The context structure initialization leaves some fields unassigned. It's not specific to Olly that some of the bits are set. Also note that the value of (0xd800+)0x7ff does not decode to any meaningful instruction.

In short, not a reliable detection method.