How to predict address space layout differences between real and gdb-controlled executions?

Question

It's something that puzzle me for a long time. I can observe that there is a difference between the real execution of a program and the gdb-controlled one.

But, here is an example:

1. First, here is the example code (we use an automatic variable to get the location of the stack):

   #include <stdio.h>
   #include <stdlib.h>
   
   int main ()
   {
     char c = 0;
   
     printf ("Stack address: %p\n", &c);
   
     return EXIT_SUCCESS;
   }
   

2. Then, we disable the ASLR (we use the personality flags of the process and not the system-wide method through /proc/sys/kernel/randomize_va_space):

   $> setarch `uname -m` -R /bin/bash
   

3. Then, get a run in the real memory environment:

   Stack address: 0x7fffffffe1df
   

4. And, the same through gdb:

    (gdb) r
    Starting program: ./gdb-against-reality
    Stack address: 0x7fffffffe17f
    [Inferior 1 (process 5374) exited normally]
    (gdb) 
   

So, here we have a difference of 96 bytes between the two runs. But, how can I predict this difference for a given program without having it running in the real memory layout (just by knowing the gdb memory layout) ?

And, also, from where/what is coming this difference ?

Answer 1

There could be other factors involved, but my guess would be that changes in the process environment variables, which are stored on the stack, are what's causing this issue.

Running a small program that just prints out the environment variables reveals a couple variations in environment variables when run inside vs outside gdb on my system.

int main(int argc, char **argv, char** envp)
{
  char** env;
  for (env = envp; *env != 0; env++)
  {
    char* thisEnv = *env;
    printf("%s\n", thisEnv);    
  }
}

First, when running under gdb, there is a LINES variable that isn't present when the process is started outside of gdb:

LINES=83

Secondly, the underscore environment variable is different. When running outside of gdb, it is set to the name of the executable:

_=./gdbtest

But when started from inside gdb, it is set to the path of the gdb binary:

_=/usr/bin/gdb

You can try to run the program normally, then attach to it with gdb/gdbserver, which should avoid these variations in environment variables (assuming that is in fact what is causing your problem).

If your process is short-lived, it can be hard to pause the process before it exits. Maybe someone else has some good suggestions on starting a process in a paused state; I usually use a second program like this one to catch the process as it is starting and pause it so I can attach a debugger to it.

Answer 2

Just to add to the answers, I can tell how to get close to a clean environment despite gdb. In fact, there are two methods to reach this:

1. We can get rid of the extra environment variables added by gdb as follow:

   (gdb) unset environment LINES
   (gdb) unset environment COLUMNS
   

   Write these commands before running the program, and you should be close to the normal environment. Note that you still have to take care of the _ variable.

2. One can also generate a memory core of the vulnerable program and analyze it with gdb:

   $> gdb vuln_program core
   

   You should just look at the memory and never run, next, step, ... because doing so will force you to restart the program with a fresh memory (with the shift).

That was two methods you can use with gdb to follow a program without too much differences with the real execution. But, they are many others!