Reverse Engineering Stack Exchange is a question and answer site for researchers and developers who explore the principles of a system through analysis of its structure, function, and operation. It only takes a minute to sign up.
Sign up to join this community
Hello best people, my question is this example is telling me about cdecl calling convention. I have confusion that when analyzing the code in disassembly how would I recognize that this is the cdecl call plus
they are saying that the highlighted line in the picture showing that the stack is cleaned by the caller as the function is complete, my question is how it is cleaning the stack, I'm according to my knowledge seeing that the value 12 is added in esp, HOW IT IS CLEANING? please explain.
esp register is the stack pointer and it points to the head of the stack, which is the lowest address in the binary.
#1 Initial stack frame:
|local var 1| <- esp (lowest address)
|local var 2|
|-----------|
|-----------|
|-----------|
|local var n| <- ebp
After the push instructions the stack will look at:
#2 Stack frame just before the call:
|a| <- esp
|b|
|c|
|local var 1|
|local var 2|
|-----------|
|-----------|
|-----------|
|local var n| <- ebp
So now 3 extra values were added to the stack frame of the current function, in order to pass arguments to the test function. 3 arguments * 4 bytes size of an argument = 12 bytes.
After test returns we need to get back to our #1 stack frame from #2 - we want esp to point back to the local var 1. in order to do that, we need to add 12 to the esp - remember that the stack grows from highest address to lowest and that the size of our 3 arguments is 12 bytes. This is the add esp, 12 instruction, and it's called cleanup the stack.
In cdecl call, the caller, which is the current function, is the one that responsible for cleanup the stack. In other calling conventions, like stdcall the callee is responsible for cleaning the stack - restore the state of the stack pointers to prior the call.
Just to add to the previous answer, “cleaning” here is not used in the literal sense but to indicate the the code returns the stack pointer (esp) to the value it had before the call. Unfortunately this term is quite entrenched so will probably continue to confuse new learners in future as well.
