For cdecl the caller needs to clean up the stack. So somewhere after the function call there HAS to be a stack operation that removes the parameters from the stack (remove as in adjust the stack pointer)
PUSH 0xABCDEF01
CALL function
ADD ESP, 4 ; sizeof(DWORD) (the parameter)
It does NOT need to be directly after the call. (I think VS08 sometimes puts the return value check before it)
So this is equally valid
PUSH 0xABCDEF01
CALL function
TEST EAX, EAX
ADD ESP, 4 ; sizeof(DWORD) (the parameter)
; A conditional jump probably
as is this
PUSH 0xABCDEF01
CALL function
POP EAX
Because EAX holds a DWORD and thus removes sizeof(DWORD)
== 4 bytes from the stack. (This obviously also holds true for all other general purpose extended registers).
(It is POSSIBLE that e.g. for N
parameters one could use N
POP
but I'm certain that no compiler does that and for every N
> 2 it will lead to bigger code so it is unrealistic to happen in handwritten assembly too.)
There doesn't exists a maximum N
of instructions that you could check after the call to determine whether that stack operation is the cleanup code. However I think that 5 will work for almost all cases (If you by chance do already know that this function is CDECL then check as many as you want)
Also for functions where the return value isn't important but is called before the return value of another function is inspected, one can save a variable like this:
CALL function1
PUSH EAX ; Save EAX
PUSH something
CALL function2
POP EAX ; Remove the something parameter
POP EAX ; Restore the return value of function1
You could try to also count the PUSH
but that comes with it's own sort of intermediate instructions - this is really common:
PUSH something
LEA EAX, something_else ; Get a pointer to something_else
PUSH EAX
PUSH more
CALL function
So you cannot stop working 'up' as soon as you hit an instruction that isn't a PUSH
.
In conclusion there isn't a perfect solution but I think both can lead to a pretty reliable value; Which one you choose (or maybe both?) is up to you, IMO the first one (Searching for the stack cleanup) is easier to implement.