0

I was practising reverse engineering on some Windows x64 applications when I came across this function:

call alloca_probe

This alloca_probe function has some strange implementation:

  1. EAX is used as a function argument (for the allocation size)
  2. The prolog saves current state of R10 and R11 even though they are considered volatile registers by convention.

As I can recall, the x64 calling convention by Microsoft indicates:

  • RCX, RDX, R8, and R9 should be used as the first 4 function arguments.
  • RAX, RCX, RDX, R8, R9, R10, R11 are all volatile registers.

alloca_probe function clearly doesn't follow this convention...

My question is:

Why doesn't this function follow the convention, and how does the compiler know how to use these type of functions (e.g using EAX as first argument)?

0

This is not a standard function but a compiler helper, used by the compiler to perform some necessary housekeeping (allocate extra stack space) so it doesn’t have to follow the ABI for “normal” functions.

Because it is called in the prolog of the parent function while the user code has not started executing yet, special care needs to be taken.

EAX use likely comes from the x86 version which has similar constraints and could not use stack arguments. In theory any register could have been used but mov eax, imm has a short encoding which saves space in the prolog. The normally volatile registers need to be saved because they need to be preserved by the function’s prolog we’re executing and if they get trashed there may be all kinds of wrong results.

  • Thanks! Is this compiler helper actually an intrinsic function? If so, aren't those functions suppose to be inlined? – YAYAdest Nov 7 at 20:24
  • Intrinsics are pseudo functions which map to processor instructions. This one is different as it’s inserted by the compiler and not by programmer. – Igor Skochinsky Nov 11 at 15:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.