Skip to main content
Reverse Engineering

Return to Answer

added technical correction, additional discussion
Source Link
julian ♦
julian ♦
  • 7.2k
  • 3
  • 23
  • 56

If this extends to AMD64 architecture as well, it means that memory can be allocated by the compiler in the area of a function's stack frame between the function stack frame base pointer %rbp and the "red zone" in a way determined by the compiler itself, so long as the stack is properly aligned to a 16-byte boundary. In other words, memory allocation for this area is compiler-dependent. MSVC may not allocate memory for this area of stack frames differently, for example. More specifically, a different compiler may not allocate 48 bytes for program 2's main() function like GCC did.

Update (thank you to Igor Skochinksy for pointing this out):

The MS doc is not totally correct; leaf function is a function which does not call other functions (it's a leaf on the function call graph); it may or may not use a stack frame (in MS ABI it doesn't but that's not necessarily the case elsewhere).

In addition to this, there are some important differences between the AMD64 ABI and the Windows x64 ABI:

Windows on x64 implements an ABI of its own, which is somewhat different from the AMD64 ABI...These are the main differences:

  1. Only 4 integer/pointer arguments are passed in registers (rcx, rdx, r8, r9). 2. There is no concept of "red zone" whatsoever. In fact, the ABI explicitly states that the area beyond rsp is considered volatile and unsafe to use. The OS, debuggers or interrupt handlers may overwrite this area. 3. Instead, a "register parameter area" (also called "home space" sometimes) is provided by the caller in each stack frame. When a function is called, the last thing allocated on the stack before the return address is space for at least 4 registers (8 bytes each). This area is available for the callee's use without explicitly allocating it. It's useful for variable argument functions as well as for debugging (providing known locations for parameters, while registers may be reused for other purposes). Although the area was originally conceived for spilling the 4 arguments passed in registers, these days the compiler uses it for other optimization purposes as well (for example, if the function needs less than 32 bytes of stack space for its local variables, this area may be used without touching rsp).

All these points are important, but the reason I have added emphasis to the first sentence of point 3 is that it pertains specifically to our discussion of why space is allocated to stack frames the way they are by the compiler.

main() in program 1 has no locals or arguments passed to it, but it does indeed have a 32-byte (4 registers * 8 bytes) register parameter area.

main() in program 2 uses 8 bytes of space for locals, so 16 bytes of space is allocated for its stack frame plus the 32-byte register parameter area. 16 bytes are allocated for the local variables in order to maintain alignment to a 16-byte boundary.

This accounts for the size difference between the stack frames created for main() in program 1 and program 2.

If this extends to AMD64 architecture as well, it means that memory can be allocated by the compiler in the area of a function's stack frame between the function stack frame base pointer %rbp and the "red zone" in a way determined by the compiler itself, so long as the stack is properly aligned to a 16-byte boundary. In other words, memory allocation for this area is compiler-dependent. MSVC may not allocate memory for this area of stack frames differently, for example. More specifically, a different compiler may not allocate 48 bytes for program 2's main() function like GCC did.

If this extends to AMD64 architecture as well, it means that memory can be allocated by the compiler in the area of a function's stack frame between the function stack frame base pointer %rbp and the "red zone" in a way determined by the compiler itself, so long as the stack is properly aligned to a 16-byte boundary. In other words, memory allocation for this area is compiler-dependent. MSVC may allocate memory for this area of stack frames differently, for example. More specifically, a different compiler may not allocate 48 bytes for program 2's main() function like GCC did.

Update (thank you to Igor Skochinksy for pointing this out):

The MS doc is not totally correct; leaf function is a function which does not call other functions (it's a leaf on the function call graph); it may or may not use a stack frame (in MS ABI it doesn't but that's not necessarily the case elsewhere).

In addition to this, there are some important differences between the AMD64 ABI and the Windows x64 ABI:

Windows on x64 implements an ABI of its own, which is somewhat different from the AMD64 ABI...These are the main differences:

  1. Only 4 integer/pointer arguments are passed in registers (rcx, rdx, r8, r9). 2. There is no concept of "red zone" whatsoever. In fact, the ABI explicitly states that the area beyond rsp is considered volatile and unsafe to use. The OS, debuggers or interrupt handlers may overwrite this area. 3. Instead, a "register parameter area" (also called "home space" sometimes) is provided by the caller in each stack frame. When a function is called, the last thing allocated on the stack before the return address is space for at least 4 registers (8 bytes each). This area is available for the callee's use without explicitly allocating it. It's useful for variable argument functions as well as for debugging (providing known locations for parameters, while registers may be reused for other purposes). Although the area was originally conceived for spilling the 4 arguments passed in registers, these days the compiler uses it for other optimization purposes as well (for example, if the function needs less than 32 bytes of stack space for its local variables, this area may be used without touching rsp).

All these points are important, but the reason I have added emphasis to the first sentence of point 3 is that it pertains specifically to our discussion of why space is allocated to stack frames the way they are by the compiler.

main() in program 1 has no locals or arguments passed to it, but it does indeed have a 32-byte (4 registers * 8 bytes) register parameter area.

main() in program 2 uses 8 bytes of space for locals, so 16 bytes of space is allocated for its stack frame plus the 32-byte register parameter area. 16 bytes are allocated for the local variables in order to maintain alignment to a 16-byte boundary.

This accounts for the size difference between the stack frames created for main() in program 1 and program 2.

fixed spelling
Source Link
julian ♦
julian ♦
  • 7.2k
  • 3
  • 23
  • 56

If this extends to AMD64 architecture as well, it means that memory can be allocated by the compiler in the area of a function's stack frame between the function stakcstack frame base pointer %rbp and the "red zone" in a way determined by the compiler itself, so long as the stack is properly aligned to a 16-byte boundary. In other words, memory allocation for this area is compiler-dependent. MSVC may not allocate memory for this area of stack frames differently, for example. More specifically, a different compiler may not allocate 48 bytes for program 2's main() function like GCC did.

If this extends to AMD64 architecture as well, it means that memory can be allocated by the compiler in the area of a function's stack frame between the function stakc frame base pointer %rbp and the "red zone" in a way determined by the compiler itself, so long as the stack is properly aligned to a 16-byte boundary. In other words, memory allocation for this area is compiler-dependent. MSVC may not allocate memory for this area of stack frames differently, for example. More specifically, a different compiler may not allocate 48 bytes for program 2's main() function like GCC did.

If this extends to AMD64 architecture as well, it means that memory can be allocated by the compiler in the area of a function's stack frame between the function stack frame base pointer %rbp and the "red zone" in a way determined by the compiler itself, so long as the stack is properly aligned to a 16-byte boundary. In other words, memory allocation for this area is compiler-dependent. MSVC may not allocate memory for this area of stack frames differently, for example. More specifically, a different compiler may not allocate 48 bytes for program 2's main() function like GCC did.

Source Link
julian ♦
julian ♦
  • 7.2k
  • 3
  • 23
  • 56

The stack is for data storage, not code

I figured it out. It looks like the sub statement is allocating space for the entire program rounded to the nearest qword. Program 1 requires only 0x17 bytes 23 bytes of code while program 2 requires 0x28 or 40 bytes of space.

This is not correct: space on the stack is not allocated for a function's code (or any other kind of code). Instead, the stack is used as storage space for variables used by functions. When a function is called, a space on the stack called a stack frame is created to accommodate the arguments passed to that function as well as local variables declared in the function. This space is created by decrementing the stack pointer, %rsp. Here is a visual depiction of 2 stack frames:

AMD64 caller and callee stack frames

There are 2 frames depicted here:

  • the caller frame, which is the stack frame created for the function that has called the current function
  • the callee frame, which is the frame created for the currently executing function

Note that code resides in a different region in the virtual memory address space than the stack (picture is of Win32 processes but applies to PE32+ process address space as well, except the address space is much larger):

PE32 layout in virtual memory

Stack frames should be aligned to 16-byte boundaries

In order to fully appreciate the situation, the full disassembly of both functions in question should be included, especially since in this case the disassembly is conveniently small.

Full disassembly of main() from prog1:

00000000004015b0 <main>:
  4015b0:   55                      push   %rbp           // save caller's frame base address
  4015b1:   48 89 e5                mov    %rsp,%rbp      // current top of stack becomes base of caller's frame
  4015b4:   48 83 ec 20             sub    $0x20,%rsp     // allocate space for new stack frame
  4015b8:   e8 93 01 00 00          callq  401750 <__main>
  4015bd:   b8 00 00 00 00          mov    $0x0,%eax
  4015c2:   48 83 c4 20             add    $0x20,%rsp     // callee stack frame now out of scope
  4015c6:   5d                      pop    %rbp           // restore caller's base frame pointer
  4015c7:   c3                      retq   
  4015c8:   90                      nop
  4015c9:   90                      nop
  4015ca:   90                      nop
  4015cb:   90                      nop
  4015cc:   90                      nop
  4015cd:   90                      nop
  4015ce:   90                      nop
  4015cf:   90                      nop

Full disassembly of main() from prog2:

00000000004015b0 <main>:
  4015b0:   55                      push   %rbp
  4015b1:   48 89 e5                mov    %rsp,%rbp
  4015b4:   48 83 ec 30             sub    $0x30,%rsp
  4015b8:   e8 a3 01 00 00          callq  401760 <__main>
  4015bd:   c7 45 fc 05 00 00 00    movl   $0x5,-0x4(%rbp)
  4015c4:   c7 45 f8 06 00 00 00    movl   $0x6,-0x8(%rbp)
  4015cb:   8b 55 fc                mov    -0x4(%rbp),%edx
  4015ce:   8b 45 f8                mov    -0x8(%rbp),%eax
  4015d1:   01 d0                   add    %edx,%eax
  4015d3:   48 83 c4 30             add    $0x30,%rsp
  4015d7:   5d                      pop    %rbp
  4015d8:   c3                      retq   
  4015d9:   90                      nop
  4015da:   90                      nop
  4015db:   90                      nop
  4015dc:   90                      nop
  4015dd:   90                      nop
  4015de:   90                      nop
  4015df:   90                      nop

This code is unoptimized, which makes interpretation significantly easier and allows us to clearly discern the calling convention being followed.

According to Microsoft's documentation on function types,

There are basically two types of functions. A function that requires a stack frame is called a frame function. A function that does not require a stack frame is called a leaf function.

A frame function is a function that allocates stack space, calls other functions, saves nonvolatile registers, or uses exception handling. It also requires a function table entry. A frame function requires a prolog and an epilog. A frame function can dynamically allocate stack space and can employ a frame pointer. A frame function has the full capabilities of this calling standard at its disposal.

If a frame function does not call another function then it is not required to align the stack.

The functions above both call __main, so they would be considered frame functions. Thus, a stack frame is allocated for main in both cases, as we can see from the disassembly above.

According to Microsoft's documentation on stack allocation,

The stack will always be maintained 16-byte aligned, except within the prolog (for example, after the return address is pushed), and except where indicated in Function Types for a certain class of frame functions.

This brings us to the original question:

Why does this program allocate 48 bytes instead of the 32 bytes from the first program?

It is not easy to answer this directly. The short answer is stack frame alignment + vagaries of the compiler.

The compiler is responsible for stack memory allocation (except when alloca is used at runtime to dynamically allocate memory on the stack, but that does not apply to the disassembly in question). It should be noted that in the disassembly of main() in program 1, nothing is saved in the created stack frame (except %rip, which gets pushed onto the stack 32-byte stack frame when __main is called) but 32 bytes are allocated anyway.

In program 2, __main is also called, and in addition 2 4-byte integer values are written to the stack frame created for main().

This allocation of "extra" memory is commonly done by GCC - stack frames are not allocated as compactly as possible in 64-bit and 32-bit runtime environments. A clue as to why this is the case is provided in the System V ABI AMD64 Architecture Processor Supplement Draft Version 0.99.7 section 3.2.1 "Registers and the Stack Frame":

SYS V AMD64 Stack Frame figure

Specifically, the "clue" is the "unspecified" area in the current stack frame. This area is not discussed further in the AMD64 ABI, but it is discussed in the System V ABI, Intel386 Architecture Processor Supplement section 3 "Low Level System Information", page 37:

Other areas depend on the compiler and the code being compiled. The standard calling sequence does not define a maximum stack frame size, nor does it restrict how a language system uses the "unspecified" area of the standard stack frame.

If this extends to AMD64 architecture as well, it means that memory can be allocated by the compiler in the area of a function's stack frame between the function stakc frame base pointer %rbp and the "red zone" in a way determined by the compiler itself, so long as the stack is properly aligned to a 16-byte boundary. In other words, memory allocation for this area is compiler-dependent. MSVC may not allocate memory for this area of stack frames differently, for example. More specifically, a different compiler may not allocate 48 bytes for program 2's main() function like GCC did.