Yes, this is an implementation of what's often called "Stack Canaries", a method of stack Buffer Overflow Protection. That example you're describing is specifically the method used by Visual Studio, enabled by default since Visual Studio 2005, implemented since Visual Studio 2003. It is also called
GS protection due to the fact Visual Studio provided the flags,
/GS to enable and
/GS- to disable, that protection and override the default behavior.
What is stack buffer overflow protection?
There are multiple stack buffer overflow protection techniques implemented by different compilers and 3rd party protection tools, but they all revolve around the same fundamental idea: When a stack buffer overflow is exploited, the attacker often overwrites the return address located on the stack to redirect code execution to a controlled address. Stack canaries work by prefixing the
ret instruction with some sort of validation that the stack, and specifically the return address, was not altered by an attacker prior to the
ret instruction being executed (which will result in a pop from the stack and placing the popped value in the instruction pointer).
How does the provided example protect the stack from such exploits?
To answer that, we'll need to provide the full implementation details of which you just provided the first half. Most stack overflow canary protections usually include inserting function prolog and epilog, and you only provided the former.
Here's a commented example prolog:
sub esp, 8 // allocate 8 bytes for cookie
mov eax, DWORD PTR ___security_cookie
xor eax, esp // xor cookie with current esp
mov DWORD PTR [esp+8], eax // save in stack
This prolog starts with allocating the stack variable's space just as any "regular" function would. It then continues to fetch a value randomly generated at the process start, and stored in a specific memory address,
EAX and then
xor-ing it with the current stack pointer. The resulting value is then stored on the stack.
And a commented example epilog:
mov ecx, DWORD PTR [esp+8] // Read saved cookie
xor ecx, esp // Xor saved cookie, should result in the same value
call @__security_check_cookie@4 // Call a short function to validate resulting value is legit, and terminate safely otherwise
add esp, 8
Then, just before the function's
ret instruction is executed, which under normal buffer overflow attack circumstances will fetch the overwritten return address and will execute attacker controlled code, a validation that the original stack cookie value was kept is made, triggering a safe failure in case the value differs from the expected stack cookie.
The logic in most of the cookie/canary based protections is the following:
- On function entrance store a specific value, deterministic but unpredictable from the attacker's perspective. Preferably, one that's dependant on the function's execution conditions (such as the current ESP). That value should be placed between the function's return address and any buffer-overflow potential buffer or variable.
- On function's exit, just before using the potentially overwritten return address, validate the stored cookie is same as expected, fail if it was somehow changed.
- As classic buffer overflow attacks are rewriting the entire stack in a sequential manner (meaning that to overwrite the return address from stack variable
buf, all data located on the stack between the return address and
buf might also be overwritten), any overwrite of the return address must also modify the stack canary. As long as the attacker cannot predict the canary before triggering the function call, the attack will fail due to the canary validation performed just before
ret is executed.
Was that the end of stack based buffer overflows?
No, for several reasons the battle over stack overflow exploitation and protection carried on (and is still ongoing, in certain circumstances):
- Soon after the first canary protections implemented in Visual Studio, attacks against SEH exception structures (which are allocated on the stack to handle exceptions) began, and provided several anti-SEH buffer overflow protections (such as SafeSEH, which underwent several versions until it was fully reliable in protecting against such attacks including the later
- Additionally, information leakage bugs were used to predict (and increase prediction chances) of canary values, which enabled bypassing the canary check and made stack based buffer overflows possible.
- Slightly similar to #2 but specific to canary protections, in certain cases an attacker could exploit the flow of the process's execution to slowly extract the canary values byte by byte, thus reducing a 2**32 (4294967296 possibilities) brute force to only 256*4 (1024 possibilities) brute force, making many attacks a lot more plausible.
- Buffer overflow bugs allowing non-linear overwrites were also being used, to "skip" overwriting the stack canary (or most of it) to either completely avoid the need to predict the canary value, or reduce the modification range to a lower, 1 byte range. Such common examples are looks that overwrite only on certain conditions or those whom overwrite only 1 byte of a dword. These would also make the return address modification limited but were still useful (and occasionally more so, in certain cases where ASLR was also used).