Debug vs Release binaries - Overflow detection

Question

I'm reading the IDA Pro book and in chapter 20 the author shows the following code from a debug build:

push ebp
mov ebp, esp
sub esp, 0F0h
push ebx
push esi
push edi 
lea edi, [ebp+var_F0]
mov ecx, 3Ch
mov eax, 0CCCCCCCCh
rep stosd
mov[ebp+var_8], 0
mov [ebp+var_14], 1
mov [ebp+var_20], 2
mov [ebp+var_2C], 3

As we can see the local variables are not adjacent to each other. Chris Eagle outlines that this makes it easier to detect overflows from one variable that may spill into and corrupt another variable, and then he just left it at that. That doesn't make sense to me, isn't it easier to just set a breakpoint after a specific operation that could cause an overflow and then check the value of variable? How exactly is this more useful?

Answer 1

the latest visual studio compilers use runtime checks to detect overflows it performs them using a variety of run-time checks

you can use them in un-optimized builds only /Od (these don't work in optimized builds not with /O1 or /O2 or /Ox)

these can be either #pragmas or /RTC1 /RTCS | U | C command-line switches

the stack corruption is detected by means of allocating a larger buffer than is required and filling it up with a known pattern

since the compiler knows how much space should be used it can check if the bounds have been trampled with unknown pattern

(yes some clever pattern matching exploits can possibly still try to fool this but it works for genuine usage where you are writing an inadvertently overflowing code)

take for example this code

#define CRT_SECURE_NO_WARNING
#include <string.h>
#include <stdio.h>
void foo(void){
    char flowoverme[0x10];
    strcpy(flowoverme,"yaddaaayadddaaafoo");
}
int main(void){
    foo();
    printf("checking overflows by pattern pasting \n");
}

(if you use /analyze compiler switch it will spit out this code will overflow

:\>cl /nologo /Zi /RTC1 /analyze /Od /EHsc rtcchk.cpp /link /nologo /debug
rtcchk.cpp
rtcchk.cpp(8) : warning C6386: Buffer overrun while writing to 'flowoverme':  the wr
itable size is '16' bytes, but '19' bytes might be written.: Lines: 7, 8

but assume you just did cl foo.cpp

:\>cl /nologo /Zi /RTC1 /Od /EHsc rtcchk.cpp /link /nologo /debug
rtcchk.cpp

if you execute this compiled code the printf wont be reached if runtime checks were enabled

:\>rtcchk.exe

:\>

we can disassemble and see what is happening inside the function foo and why printf() is not executed

lets open up the binary in windbg go to start of foo() and ask windbg to go up (gu that is return to main() back) as below and you will notice windbg doesn't return to main but stops with an error message

:\>cdb -c "g rtcchk!foo;gu" rtcchk.exe

Microsoft (R) Windows Debugger Version 10.0.16299.15 X86

0:000> cdb: Reading initial command 'g rtcchk!foo;gu'

rtcchk!failwithmessage+0x255:
013d75da cc              int     3

and the call stack would show

0:000> kP
ChildEBP RetAddr
0028f544 013d72a9 rtcchk!failwithmessage(
                        void * retaddr = 0x013d698a,
                        int crttype = 0n1,
                        int errnum = 0n2,
                        char * msg = 0x0028f568 "Stack around the variable 'flowoverme' was corrupted.")+0x255
0028f96c 013d6c3d rtcchk!_RTC_StackFailure(
                        void * retaddr = 0x013d698a,
                        char * varname = 0x013d69b8 "flowoverme")+0x94
0028f98c 013d698a rtcchk!_RTC_CheckStackVars(
                        void * frame = 0x0028f9b8,
                        struct _RTC_framedesc * v = 0x013d69a4)+0x42
0028f9b8 013d69d8 rtcchk!foo(void)+0x4a
0028f9c0 013d6ecd rtcchk!main(void)+0x8
(Inline) -------- rtcchk!invoke_main+0x1c
0028fa08 76a9ed6c rtcchk!__scrt_common_main_seh(void)+0xf9
0028fa14 77cb37eb kernel32!BaseThreadInitThunk+0xe
0028fa54 77cb37be ntdll!__RtlUserThreadStart+0x70
0028fa6c 00000000 ntdll!_RtlUserThreadStart+0x1b
0:000>

if you still want to know what or how those functions operate open up either crt sources in vs or disassemble the functions

the compiler knows the required size and where the bounds are

0:000> dx -r3 (_RTC_framedesc *) 0x013d69a4
(_RTC_framedesc *) 0x013d69a4  : 0x13d69a4 [Type: _RTC_framedesc *]
    [+0x000] varCount         : 1 [Type: int]
    [+0x004] variables        : 0x13d69ac [Type: _RTC_vardesc *]
        [+0x000] addr             : -24 [Type: int]
        [+0x004] size             : 16 [Type: int]
        [+0x008] name             : 0x13d69b8 : "flowoverme" [Type: char *]
0:000>

Answer 2

I've checked the mentioned chapter and there's also an info about detecting an execution of the instruction on the stack. This is I think more common scenario.

As for the overflow detection I can only speculate, but for me it's easier to check, in one place, if all the values that should be 0xCC are still in fact intact the same than to do this every time after an operation that could overflow and check if the value that should be the result of an operation. Consider also an arrays that with this approach could be check if the do not span outside the range than they should.

With having the use of OxCC both checks can be covered.

Answer 3

What you pasted is Visual C++'s /GZ switch or for newer versions /RTC. It fills the stack for local variables with 0xCC. What I couldn't find is if there's an actual function that checks at runtime if any of the gaps have been tainted.

I assume there is because the compiler should be able to generate those at compile time knowing the stack layout, so at some point (end of function, or exception?) he could automatically verify if the gaps are clean.

Without the gap, you would not be able to tell if one variable write spilled because it would just end up in the next one. I guess that's what the book author was implying.

Unfortunately I could not find clear documentation for those flags, except a description that it does fill the stack and use it to verify the integrity.