3

Lets assume this function frame :

enter image description here

How to calculate and check if subtracted stack space by line 3 is 58h ?: sub esp, 58h

And this is the code, the compiler is also Dev C++:

#include <stdio.h>

char *the_good_one = "gb_master";

void cookie()
{
    printf("Get a cookie!\n");
}

char check_password()
{
    char password[64];

    printf("Enter password: ");
    scanf("%s", password);

    return (!strcmp(password, the_good_one));
}

int main(void)
{
    if(check_password())
    {
        printf("GOOOOOOOOOOD!\n");
        cookie();
    }
    else
    {
        printf("Wrong password\n");
    }

    return 0;
}

check_password() function disassembly added :

enter image description here

  • 1
    I added the source code – Vlad Jun 7 '15 at 13:16
  • 2
    Could you give fully disassembled code of "check_password"? Since the compiler may insert some canaries for stack protection. – Ta Thanh Dinh Jun 7 '15 at 14:01
  • 1
    @tathanhdinh added – Vlad Jun 7 '15 at 14:09
  • 1
    Thanks Vlad. Sorry to bother you again, could you please give the compiler options you have been used? – Ta Thanh Dinh Jun 7 '15 at 14:30
  • 2
    Frankly, I find it quite bizzare. A stack of size 72 is enough for "check_password' since we need 64 bytes for password, and 8 bytes for arguments of printf, scanf, and strcmp. I do not find neither any evidence for data alignment. And when I try your code with gcc 4.9.0, the compiler generates "check_password" with stack of size 84!!! – Ta Thanh Dinh Jun 7 '15 at 22:50
6

Older compilers made space for function parameters on the stack by pushing them, and popping from the stack after the function call; newer compilers optimize this. For example, while a function gets executed, the stack changed like this:

 start        calling       after         before scanf   after scanf 
              printf        printf                                   
+-----------+ +-----------+ +-----------+ +-----------+ +-----------+
|return addr| |return addr| |return addr| |return addr| |return addr|
+-----------+ +-----------+ +-----------+ +-----------+ +-----------+
|           | |           | |           | |           | |           |
| local     | | local     | | local     | | local     | | local     |
| variables | | variables | | variables | | variables | | variables |
|           | |           | |           | |           | |           |
|           | |           | |           | |           | |           |
|           | |           | |           | |           | |           |
|           | |           | |           | |           | |           |
|           | |           | |           | |           | |           |
|           | |           | |           | |           | |           |
+-----------+ +-----------+ +-----------+ +-----------+ +-----------+
              | "Enter.." |               | password  |              
              +-----------+               +-----------+              
                                          | "%s"      |              
                                          +-----------+              

You see how sp (the bottom of the stack) changes with every function call. Newer versions of gcc change this; they make enough space on the stack (for local variables and all possible function parameters) from the beginning, and just move the parameters to addresses relative to the stack pointer:

 start        calling       after         before scanf   after scanf 
              printf        printf                                   
+-----------+ +-----------+ +-----------+ +-----------+ +-----------+
|return addr| |return addr| |return addr| |return addr| |return addr|
+-----------+ +-----------+ +-----------+ +-----------+ +-----------+
|           | |           | |           | |           | |           |
| local     | | local     | | local     | | local     | | local     |
| variables | | variables | | variables | | variables | | variables |
|           | |           | |           | |           | |           |
|           | |           | |           | |           | |           |
|           | |           | |           | |           | |           |
|           | |           | |           | |           | |           |
|           | |           | |           | |           | |           |
|           | |           | |           | |           | |           |
+-----------+ +-----------+ +-----------+ +-----------+ +-----------+
|           | |           | |           | | password  | |           |
+-----------+ +-----------+ +-----------+ +-----------+ +-----------+
|           | | "Enter.." | |           | | "%s"      | |           |
+-----------+ +-----------+ +-----------+ +-----------+ +-----------+

Note how, from the beginning, the stack has the size it needs for the 4th step (before scanf), and how the "Enter.." string is moved directly to where the stack pointer is (so it's the first parameter on the stack), not to the space directly below local variables.

So to calculate the stack size from the source code, you need to know

  • how many bytes your return frame needs
  • how many bytes to reserve for the stack canary, if any
  • how many bytes local variables need
  • how many bytes to reserve for function arguments; this may include special treatment for structures, or doubles, that don't follow standard conventions
  • how many bytes to subtract from the function arguments because they're passed in registers, this applies to 64 bit code especially.

This might change per compiler version as well. It seems that your compiler reserved 8 bytes for stack frame/canary; 64 (0x40) bytes for the password array, and another 16 bytes for function parameters, where 8 would have been sufficient (maybe for alignment reasons)?

I wouldn't rely on any formula for the number of bytes needed; instead, check with the specific compiler i'm using, and prepare for this number to change whenever a different compiler gets used.

  • 1
    I think you are right. In fact, I still do not understand why the function needs a stack of 88 bytes, while 64 (local variable) + 8 (arguments) = 72 should be sufficient. – Ta Thanh Dinh Jun 7 '15 at 22:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.