IDA Pro - configure register clobbering

Question

TL;DR:

The __fastcall convention in IDA Pro assumes that all the registers which could be used for transferring parameters are clobbered after a call (tested on x86, clobbered regs are eax, edx, ebx, ecx). I'm wondering how can I change that.

Full explanation:

I have a chunk of code (used Watcom name mangling):

cseg01:00062A6B                 mov     edx, [ebp+tng1]
cseg01:00062A6E                 mov     eax, [ebp+this1]
cseg01:00062A71                 call    W?MyMethod$_Whatever$n_x__pn$SubObject$$
cseg01:00062A76                 call    W?MySubMethod$_SubObject$n_pn$Thing$$_l
cseg01:00062A7B                 cmp     eax, 0Ah

The functions are:

SubObject *__fastcall Whatever::MyMethod(Whatever *__hidden this);
int __fastcall SubObject::MySubMethod(SubObject *__hidden this, Thing *tng);

IDA Pro produces:

(v3 = Whatever::MyMethod(this1), SubObject::MySubMethod(v3, v4) < 10)

So the proper code clearly is:

(this1->MyMethod()->MySubMethod(tng1) < 10)

The problem is — IDA did not use tng1 as the 2nd argument of MySubMethod() — instead, it defined a separate variable v4 with an unset value.

Because we use __fastcall for both functions, first arg is passed through eax, second by edx. The code so happens to set the value of edx before the first call, even though it's really an argument for the second call. IDA seem to assume that MyMethod() overwrote (or with proper nomenclature, clobbered) the value of EDX).

I tried __usercall and __userpurge, and it had no impact on the code — IDA still assumes EDX was clobbered by the first call.

Is there a way to configure/change that behavior, making IDA know that function calls does not clobber registers?

I remember fixing it in the past, so there is a way, just can't remember what it was exactly...

Also, it would be nice to configure that on a project level instead of defining a custom calling convention for every function.

Answer 1

In the absence of an explicit __spoils declaration on the called function's prototype, Hex-Rays makes decisions about which registers should be spoiled at a call site based on the compiler that has been chosen for the binary, which can be viewed and modified under Options->Compiler. Your first step should be to ensure that the Compiler dropdown box says "Watcom C++".

The authors of Hex-Rays need to implement support for each compiler's ABI separately. That is to say, if you were to reverse engineer the part of Hex-Rays that chose the standard clobber set to be used throughout call analysis, you'd find that the logic made use of if-statements that were checking the database's compiler settings (as described in the previous paragraph).

Unfortunately, as of the last time I reverse engineered that part of Hex-Rays, they had not yet implemented support for Watcom's ABI. It's possible this has changed in the meantime -- again, your first step should be to set the compiler to Watcom and see what happens. I get the impression they might have added it in 7.3, based on the following changelog:

BUGFIX: TYPES: corrected the list of spoiled registers for watcom files

However, if setting the compiler doesn't work, then there's nothing you can do as a user to fix this without manually annotating each function's __spoils set. If so, this would be a great time to contact Hex-Rays support with your license key, and a copy of the binary if possible, and ask them if they can implement ABI support for Watcom in the decompiler. It may not happen in the next release, but if enough customers have enough requests involving Watcom binaries, it'll happen. After all, it's probably not that much work on their end to implement it, and it's a nice feature to have out-of-the-box.