GS Cookie and exception handlers

Question

I'm trying to understand how this GS cookie implementation is working. From what I've read on the topic, a cookie is set during the prologue then checked again in the epilogue. Well I can see the cookie being set, but it is not like the examples I've seen online.

prologue:

push ebp
mov ebp,esp
push FFFFFFFF
push sdk.FAB99E9               ; New Exception handler
mov eax,dword ptr fs:[0]       ; Old Exception handler
push eax
sub esp,14                     ; Allocate 5 DWORDs
push ebx
push esi
push edi
mov eax,dword ptr ds:[FAC635C] ; The GS cookie (.data section)
xor eax,ebp
push eax                       ; The GS Cookie
lea eax,dword ptr ss:[ebp-C]
mov dword ptr fs:[0],eax       ; Set new Exception Handler
mov edi,ecx                    ; Address of a vector passed as arg1
mov dword ptr ss:[ebp-20],edi  ; local variable
mov dword ptr ss:[ebp-1C],0    ; local variable
mov dword ptr ds:[edi],0       ; zero the vector
mov dword ptr ds:[edi+4],0
mov dword ptr ds:[edi+8],0
mov dword ptr ss:[ebp-4],0     ; sets the 0xFFFFFFFF above to 0x0
mov dword ptr ss:[ebp-1C],1
mov byte ptr ss:[ebp-D],0

epilogue:

mov eax,edi
mov ecx,dword ptr ss:[ebp-C]   ; Old Exception handler
mov dword ptr fs:[0],ecx       ; Restore Exception handler
pop ecx                        ; Pop the GS Cookie
pop edi
pop esi
pop ebx
mov esp,ebp
pop ebp
ret 

The function commented as "New Exception Handler", does contain a Cookie check. But I've never been able to step into it.

Can anybody explain how this works and what it might look like in C++? Or, more specific to my needs, is this a try/catch block in the original code or added by the compiler?

Answer 1

The function commented as "New Exception Handler", does contain a Cookie check. But I've never been able to step into it.

Since the exception handler is called only during an exception, it won't be called during normal execution, so this is expected behavior.

As for the missing cookie check in the epilog, apparently your function is not using the plain old GS cookie (return address overwrite protection via simple comparison) but only the improved EH+GS cookie (local variables + EH data + return address protection with the extra XOR check).

Commented function prolog (comments refer to final offsets in the frame):

 push    ebp             ; [EBP+0] save old EBP
 mov     ebp, esp        ; set up ebp frame (ESP=EBP)
 push    0FFFFFFFFh      ; [EBP-4] push initial state
 push    offset SEH_1000BF50 ; [EBP-8] SEH handler for the function
 mov     eax, large fs:0 ; read SEH chain list head
 push    eax             ; [EBP-C] pointer to previous SEH record
 sub     esp, 14h        ; allocate space for local vars and move to the saved registers area.
 push    ebx             ; [EBP-18] save ebx
 push    esi             ; [EBP-1C] save esi
 push    edi             ; [EBP-20] save edi
 mov     eax, ___security_cookie
 xor     eax, ebp        ; xor the cookie to make it harder to forge
 push    eax             ; EBP-24 push XOR'ed cookie
 lea     eax, [ebp-0Ch] ; eax = &SEH_record
 mov     large fs:0, eax ; insert our SEH entry at start of list

final stack frame layout:

EBP-30 xored EH cookie  
EBP-2C saved edi
EBP-28 saved esi
EBP-24 saved ebx
EBP-20 <local variables>
EBP-0C SEH pointer to previous record\
EBP-08 SEH handler                    | extended SEH record
EBP-04 EH state                      /
EBP+00 saved EBP
EBP+04 return address

In case of an exception, the check happens in the function-specific SEH handler before jumping to ___CxxFrameHandler3 which performs C++ exception handling or stack unwinding:

SEH_1000BF50:
  mov     edx, [esp+8]    ; (1) get pointer to current SEH record
  lea     eax, [edx+0Ch]  ; edx <- original frame pointer (function's EBP)
  mov     ecx, [edx-24h]  ; get xor'ed cookie at orig_ebp-30h
  xor     ecx, eax        ; xor them together
  call    @__security_check_cookie@4 ; check expected value
  mov     eax, offset stru_10034344
  jmp     ___CxxFrameHandler3   ; handle C++ exceptions/unwinding

(1) gets the second argument of the SEH handler (EstablisherFrame), which points to the active SEH record which was stored at EBP-C in the function, so we adjust it by 0Ch to get the original EBP value which is used for xoring with the cookie value (originally stored at [EBP-30h], or at -24h from the SEH record), and the xor result is compared with the expected value by __security_check_cookie().

I guess the compiler decided that there is no need to protect just the return address because there are no buffers in the function (see "GS Buffers" in MSDN), and any other overwrite will trash the EH cookie and will be detected in case of exception.

BTW, the state variable at [ebp-4] uniquely identifies a state in the function's execution. This allows the C++ exception handler (e.g. __CxxFrameHandler) to destroy any automatic objects leaving their scope due to the exception and unwind the state to the starting one (usually value -1). So this exception handler setup does not correspond to a "try/catch block" but has been added by the compiler for the whole function. However, the state transitions (writes to the try level variable may correspond to try block boundaries (or just scope boundaries).

For more background on Visual C++ exception handling implementation (both C-style SEH and C++ EH), check out my article on the topic (and its references).