Computing subtracted stack space for a function manually

Question

Lets assume this function frame :

How to calculate and check if subtracted stack space by line 3 is 58h ?: sub esp, 58h

And this is the code, the compiler is also Dev C++:

#include <stdio.h>

char *the_good_one = "gb_master";

void cookie()
{
    printf("Get a cookie!\n");
}

char check_password()
{
    char password[64];

    printf("Enter password: ");
    scanf("%s", password);

    return (!strcmp(password, the_good_one));
}

int main(void)
{
    if(check_password())
    {
        printf("GOOOOOOOOOOD!\n");
        cookie();
    }
    else
    {
        printf("Wrong password\n");
    }

    return 0;
}

check_password() function disassembly added :

Answer 1

Older compilers made space for function parameters on the stack by pushing them, and popping from the stack after the function call; newer compilers optimize this. For example, while a function gets executed, the stack changed like this:

 start        calling       after         before scanf   after scanf 
              printf        printf                                   
+-----------+ +-----------+ +-----------+ +-----------+ +-----------+
|return addr| |return addr| |return addr| |return addr| |return addr|
+-----------+ +-----------+ +-----------+ +-----------+ +-----------+
|           | |           | |           | |           | |           |
| local     | | local     | | local     | | local     | | local     |
| variables | | variables | | variables | | variables | | variables |
|           | |           | |           | |           | |           |
|           | |           | |           | |           | |           |
|           | |           | |           | |           | |           |
|           | |           | |           | |           | |           |
|           | |           | |           | |           | |           |
|           | |           | |           | |           | |           |
+-----------+ +-----------+ +-----------+ +-----------+ +-----------+
              | "Enter.." |               | password  |              
              +-----------+               +-----------+              
                                          | "%s"      |              
                                          +-----------+              

You see how sp (the bottom of the stack) changes with every function call. Newer versions of gcc change this; they make enough space on the stack (for local variables and all possible function parameters) from the beginning, and just move the parameters to addresses relative to the stack pointer:

 start        calling       after         before scanf   after scanf 
              printf        printf                                   
+-----------+ +-----------+ +-----------+ +-----------+ +-----------+
|return addr| |return addr| |return addr| |return addr| |return addr|
+-----------+ +-----------+ +-----------+ +-----------+ +-----------+
|           | |           | |           | |           | |           |
| local     | | local     | | local     | | local     | | local     |
| variables | | variables | | variables | | variables | | variables |
|           | |           | |           | |           | |           |
|           | |           | |           | |           | |           |
|           | |           | |           | |           | |           |
|           | |           | |           | |           | |           |
|           | |           | |           | |           | |           |
|           | |           | |           | |           | |           |
+-----------+ +-----------+ +-----------+ +-----------+ +-----------+
|           | |           | |           | | password  | |           |
+-----------+ +-----------+ +-----------+ +-----------+ +-----------+
|           | | "Enter.." | |           | | "%s"      | |           |
+-----------+ +-----------+ +-----------+ +-----------+ +-----------+

Note how, from the beginning, the stack has the size it needs for the 4th step (before scanf), and how the "Enter.." string is moved directly to where the stack pointer is (so it's the first parameter on the stack), not to the space directly below local variables.

So to calculate the stack size from the source code, you need to know

• how many bytes your return frame needs
• how many bytes to reserve for the stack canary, if any
• how many bytes local variables need
• how many bytes to reserve for function arguments; this may include special treatment for structures, or doubles, that don't follow standard conventions
• how many bytes to subtract from the function arguments because they're passed in registers, this applies to 64 bit code especially.

This might change per compiler version as well. It seems that your compiler reserved 8 bytes for stack frame/canary; 64 (0x40) bytes for the password array, and another 16 bytes for function parameters, where 8 would have been sufficient (maybe for alignment reasons)?

I wouldn't rely on any formula for the number of bytes needed; instead, check with the specific compiler i'm using, and prepare for this number to change whenever a different compiler gets used.