Why doesn't alloca_probe follow x64 calling convention?

Question

I was practising reverse engineering on some Windows x64 applications when I came across this function:

call alloca_probe

This alloca_probe function has some strange implementation:

1. EAX is used as a function argument (for the allocation size)
2. The prolog saves current state of R10 and R11 even though they are considered volatile registers by convention.

As I can recall, the x64 calling convention by Microsoft indicates:

• RCX, RDX, R8, and R9 should be used as the first 4 function arguments.
• RAX, RCX, RDX, R8, R9, R10, R11 are all volatile registers.

alloca_probe function clearly doesn't follow this convention...

My question is:

Why doesn't this function follow the convention, and how does the compiler know how to use these type of functions (e.g using EAX as first argument)?

Answer 1

This is not a standard function but a compiler helper, used by the compiler to perform some necessary housekeeping (allocate extra stack space) so it doesn’t have to follow the ABI for “normal” functions.

Because it is called in the prolog of the parent function while the user code has not started executing yet, special care needs to be taken.

EAX use likely comes from the x86 version which has similar constraints and could not use stack arguments. In theory any register could have been used but mov eax, imm has a short encoding which saves space in the prolog. The normally volatile registers need to be saved because they need to be preserved by the function’s prolog we’re executing and if they get trashed there may be all kinds of wrong results.