The __fastcall convention in IDA Pro assumes that all the registers which could be used for transferring parameters are clobbered after a call (tested on x86, clobbered regs are eax, edx, ebx, ecx). I'm wondering how can I change that.

Full explanation:

I have a chunk of code (used Watcom name mangling):

cseg01:00062A6B                 mov     edx, [ebp+tng1]
cseg01:00062A6E                 mov     eax, [ebp+this1]
cseg01:00062A71                 call    W?MyMethod$_Whatever$n_x__pn$SubObject$$
cseg01:00062A76                 call    W?MySubMethod$_SubObject$n_pn$Thing$$_l
cseg01:00062A7B                 cmp     eax, 0Ah

The functions are:

SubObject *__fastcall Whatever::MyMethod(Whatever *__hidden this);
int __fastcall SubObject::MySubMethod(SubObject *__hidden this, Thing *tng);

IDA Pro produces:

(v3 = Whatever::MyMethod(this1), SubObject::MySubMethod(v3, v4) < 10)

So the proper code clearly is:

(this1->MyMethod()->MySubMethod(tng1) < 10)

The problem is — IDA did not use tng1 as the 2nd argument of MySubMethod() — instead, it defined a separate variable v4 with an unset value.

Because we use __fastcall for both functions, first arg is passed through eax, second by edx. The code so happens to set the value of edx before the first call, even though it's really an argument for the second call. IDA seem to assume that MyMethod() overwrote (or with proper nomenclature, clobbered) the value of EDX).

I tried __usercall and __userpurge, and it had no impact on the code — IDA still assumes EDX was clobbered by the first call.

Is there a way to configure/change that behavior, making IDA know that function calls does not clobber registers?

I remember fixing it in the past, so there is a way, just can't remember what it was exactly...

Also, it would be nice to configure that on a project level instead of defining a custom calling convention for every function.

  • 1
    Oh right, the clobbering is configured per-function using __spoils. Details here: hex-rays.com/products/ida/support/idadoc/1361.shtml But I don't think there's project-wide setting for it. Or is there? Nov 27 '21 at 21:12
  • I'm not sure defining a calling convention is necessary. I think defining the prototype of mySubMethod to accept two arguments (I assume it accepts one now) will do the trick. I think the main issue here is not with clobbering but with incorrectly identifying the number of arguments in mySubMethod
    – NirIzr
    Nov 30 '21 at 8:26

In the absence of an explicit __spoils declaration on the called function's prototype, Hex-Rays makes decisions about which registers should be spoiled at a call site based on the compiler that has been chosen for the binary, which can be viewed and modified under Options->Compiler. Your first step should be to ensure that the Compiler dropdown box says "Watcom C++".

The authors of Hex-Rays need to implement support for each compiler's ABI separately. That is to say, if you were to reverse engineer the part of Hex-Rays that chose the standard clobber set to be used throughout call analysis, you'd find that the logic made use of if-statements that were checking the database's compiler settings (as described in the previous paragraph).

Unfortunately, as of the last time I reverse engineered that part of Hex-Rays, they had not yet implemented support for Watcom's ABI. It's possible this has changed in the meantime -- again, your first step should be to set the compiler to Watcom and see what happens. I get the impression they might have added it in 7.3, based on the following changelog:

BUGFIX: TYPES: corrected the list of spoiled registers for watcom files

However, if setting the compiler doesn't work, then there's nothing you can do as a user to fix this without manually annotating each function's __spoils set. If so, this would be a great time to contact Hex-Rays support with your license key, and a copy of the binary if possible, and ask them if they can implement ABI support for Watcom in the decompiler. It may not happen in the next release, but if enough customers have enough requests involving Watcom binaries, it'll happen. After all, it's probably not that much work on their end to implement it, and it's a nice feature to have out-of-the-box.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.