I was practising reverse engineering on some Windows x64 applications when I came across this function:
This alloca_probe function has some strange implementation:
- EAX is used as a function argument (for the allocation size)
- The prolog saves current state of R10 and R11 even though they are considered volatile registers by convention.
As I can recall, the x64 calling convention by Microsoft indicates:
- RCX, RDX, R8, and R9 should be used as the first 4 function arguments.
- RAX, RCX, RDX, R8, R9, R10, R11 are all volatile registers.
alloca_probe function clearly doesn't follow this convention...
My question is:
Why doesn't this function follow the convention, and how does the compiler know how to use these type of functions (e.g using EAX as first argument)?