There are two things you must check:
- You overwritten exception handler but, is there any exception occurs?
- Are there any protection mechanism active for exception handlers?
For first one, most easy exception which you can trigger and not caught until your handler is obviously access violation. You can try to trigger an exception with overwrite a value which referenced before stack cookie checked. Try to overwrite as long as possible for increasing your chance.
There may be another methods like overwriting some variables and structures but not overwriting stack cookie depending your example. Most of the times compiler arranges overflowing buffers close to stack cookie.
For second one, there may be SafeSEH on and off modules loaded in program memory. But, it is not matter for this stage. Despite SafeSEH, SEHOP (Structured Exception Handling Overwrite Protection) may be blocking program to access exception handler. If you done all tests on Windows XP most probably your problem is not SEHOP. I couldn't find proper resources if Microsoft published any update including SEHOP for Windows XP, last time I checked there is no SEHOP support for Windows XP.
SEHOP specifications need two things to be assured according to this article:
- Each SEH structure points to the next structure.
- Last one contains a specific handler pointing to ntdll!_except_handler4