The __fastcall convention in IDA Pro assumes that all the registers which could be used for transferring parameters are clobbered after a call (tested on x86, clobbered regs are
ecx). I'm wondering how can I change that.
I have a chunk of code (used Watcom name mangling):
cseg01:00062A6B mov edx, [ebp+tng1] cseg01:00062A6E mov eax, [ebp+this1] cseg01:00062A71 call W?MyMethod$_Whatever$n_x__pn$SubObject$$ cseg01:00062A76 call W?MySubMethod$_SubObject$n_pn$Thing$$_l cseg01:00062A7B cmp eax, 0Ah
The functions are:
SubObject *__fastcall Whatever::MyMethod(Whatever *__hidden this); int __fastcall SubObject::MySubMethod(SubObject *__hidden this, Thing *tng);
IDA Pro produces:
(v3 = Whatever::MyMethod(this1), SubObject::MySubMethod(v3, v4) < 10)
So the proper code clearly is:
(this1->MyMethod()->MySubMethod(tng1) < 10)
The problem is — IDA did not use
tng1 as the 2nd argument of
MySubMethod() — instead, it defined a separate variable
v4 with an unset value.
Because we use
__fastcall for both functions, first arg is passed through
eax, second by
edx. The code so happens to set the value of
edx before the first call, even though it's really an argument for the second call. IDA seem to assume that
MyMethod() overwrote (or with proper nomenclature, clobbered) the value of
I tried __usercall and __userpurge, and it had no impact on the code — IDA still assumes
EDX was clobbered by the first call.
Is there a way to configure/change that behavior, making IDA know that function calls does not clobber registers?
I remember fixing it in the past, so there is a way, just can't remember what it was exactly...
Also, it would be nice to configure that on a project level instead of defining a custom calling convention for every function.