What could be the purpose of XORing esp with a global variable? (stack canary protection)

Question

I am currently looking at a function inside a Win32 executable's main module. After allocating memory on the stack (sub esp) and saving some registers on the stack, the value of esp is XORed with a global variable.

mov eax, esp
xor eax, DWORD PTR ds:[<some address>]
push eax

I wonder whether this is some kind of stack protection technique?

Edit: I wrote the question from memory. The actual instruction sequence is:

sub esp, 0xf0
mov eax, DWORD PTR ds:[<some address>]
xor eax, esp
mov DWORD PTR ss:[esp+0xec], eax

Answer 1

Yes, this is an implementation of what's often called "Stack Canaries", a method of stack Buffer Overflow Protection. That example you're describing is specifically the method used by Visual Studio, enabled by default since Visual Studio 2005, implemented since Visual Studio 2003. It is also called GS protection due to the fact Visual Studio provided the flags, /GS to enable and /GS- to disable, that protection and override the default behavior.

What is stack buffer overflow protection?

There are multiple stack buffer overflow protection techniques implemented by different compilers and 3rd party protection tools, but they all revolve around the same fundamental idea: When a stack buffer overflow is exploited, the attacker often overwrites the return address located on the stack to redirect code execution to a controlled address. Stack canaries work by prefixing the ret instruction with some sort of validation that the stack, and specifically the return address, was not altered by an attacker prior to the ret instruction being executed (which will result in a pop from the stack and placing the popped value in the instruction pointer).

How does the provided example protect the stack from such exploits?

To answer that, we'll need to provide the full implementation details of which you just provided the first half. Most stack overflow canary protections usually include inserting function prolog and epilog, and you only provided the former.

Here's a commented example prolog:

sub         esp,    8           //  allocate 8 bytes for cookie
mov eax,    DWORD   PTR ___security_cookie
xor eax,    esp                 //  xor cookie with current esp
mov DWORD   PTR [esp+8],    eax //  save in stack

This prolog starts with allocating the stack variable's space just as any "regular" function would. It then continues to fetch a value randomly generated at the process start, and stored in a specific memory address, into register EAX and then xor-ing it with the current stack pointer. The resulting value is then stored on the stack.

And a commented example epilog:

mov ecx,    DWORD   PTR     [esp+8]  // Read saved cookie
xor ecx,    esp                      // Xor saved cookie, should result in the same value
call        @__security_check_cookie@4  // Call a short function to validate resulting value is legit, and terminate safely otherwise
add         esp,    8

Then, just before the function's ret instruction is executed, which under normal buffer overflow attack circumstances will fetch the overwritten return address and will execute attacker controlled code, a validation that the original stack cookie value was kept is made, triggering a safe failure in case the value differs from the expected stack cookie.

The logic in most of the cookie/canary based protections is the following:

1. On function entrance store a specific value, deterministic but unpredictable from the attacker's perspective. Preferably, one that's dependant on the function's execution conditions (such as the current ESP). That value should be placed between the function's return address and any buffer-overflow potential buffer or variable.
2. On function's exit, just before using the potentially overwritten return address, validate the stored cookie is same as expected, fail if it was somehow changed.
3. As classic buffer overflow attacks are rewriting the entire stack in a sequential manner (meaning that to overwrite the return address from stack variable buf, all data located on the stack between the return address and buf might also be overwritten), any overwrite of the return address must also modify the stack canary. As long as the attacker cannot predict the canary before triggering the function call, the attack will fail due to the canary validation performed just before ret is executed.

Was that the end of stack based buffer overflows?

No, for several reasons the battle over stack overflow exploitation and protection carried on (and is still ongoing, in certain circumstances):

1. Soon after the first canary protections implemented in Visual Studio, attacks against SEH exception structures (which are allocated on the stack to handle exceptions) began, and provided several anti-SEH buffer overflow protections (such as SafeSEH, which underwent several versions until it was fully reliable in protecting against such attacks including the later SEHOP).
2. Additionally, information leakage bugs were used to predict (and increase prediction chances) of canary values, which enabled bypassing the canary check and made stack based buffer overflows possible.
3. Slightly similar to #2 but specific to canary protections, in certain cases an attacker could exploit the flow of the process's execution to slowly extract the canary values byte by byte, thus reducing a 2**32 (4294967296 possibilities) brute force to only 256*4 (1024 possibilities) brute force, making many attacks a lot more plausible.
4. Buffer overflow bugs allowing non-linear overwrites were also being used, to "skip" overwriting the stack canary (or most of it) to either completely avoid the need to predict the canary value, or reduce the modification range to a lower, 1 byte range. Such common examples are looks that overwrite only on certain conditions or those whom overwrite only 1 byte of a dword. These would also make the return address modification limited but were still useful (and occasionally more so, in certain cases where ASLR was also used).

Answer 2

Yes, this is Microsoft's stack overflow protection, commonly known as "GS cookie". From Compiler Security Checks In Depth:

When the function is compiled with /GS, the functions prolog will set aside an additional four bytes and add three more instructions as follows:

sub   esp,24h
mov   eax,dword ptr [___security_cookie (408040h)]
xor   eax,dword ptr [esp+24h]
mov   dword ptr [esp+20h],eax

The prolog contains an instruction that fetches a copy of the cookie, followed by an instruction that does a logical xor of the cookie and the return address, and then finally an instruction that stores the cookie on the stack directly below the return address. From this point forward, the function will execute as it does normally. When a function returns, the last thing to execute is the function’s epilog, which is the opposite of the prolog.

When compiled with /GS, the security checks are also placed in the epilog:

mov   ecx,dword ptr [esp+20h]
xor   ecx,dword ptr [esp+24h]
add   esp,24h
jmp   __security_check_cookie (4010B2h)

The stack's copy of the cookie is retrieved and then follows with the XOR instruction with the return address. The ECX register should contain a value that matches the original cookie stored in the __security_cookie variable. The stack space is then reclaimed, and then, instead of executing the RET instruction, the JMP instruction to the __security_check_cookie routine is executed.