How does Windows SafeSEH mechanism work?
I understand how the overwriting SEH exploitation technique works. I do not quite understand how the SafeSEH mechanism works preventing this technique from working. This question is about inner workings of the SafeSEH mechanism.
Regarding your original question: Upon ntdll.dll loading each PE image, the list of exception handler addresses in the PE image is parsed and each address is inserted into a sorted list that ntdll.dll internally uses. When an exception hits, ntdll.dll!KiUserExceptionDispatcher will try to figure out which exception in the SEH chain to use. In turn, each exception handler address is scanned for in this list (by ntdll.dll). If it is found, then the exception filter is allowed to run. If not, then ntdll.dll will throw exception and essentially fastfail the thread with 0xC00001A5: An invalid exception handler routine has been detected.
If you want to browse the "code" yourself, break on KiUserExceptionDispatcher, throw an exception in a dummy program, and step through. You can also scan ntdll in IDA for 0xC00001A5 (should not occur very often) and when you see that used, that is the point at which ntdll has decided to fastfail your thread (may be hard to step back due to codepath locality optimizations).
I post my own answer based on my own research. I am not sure about correctness of the parts in bold. Comments correcting my answer or fuller and better answers are welcome. I will edit out the boldness after I receive comments confirming or refuting my assumptions.
Assume that your malicious buffer overwrote the _EXCEPTION_REGISTRATION_RECORD and you found its offset. Now as a first step of SEH exploitation you need to overwrite the record's second field, PEXCEPTION_ROUTINE, with address of pop pop ret instruction sequence. At this point, if you do not try to find this sequence in the .exe module being exploited, it does not matter if it was compiled with /SAFESEH.
Next, you turn to picking pop pop ret instruction sequence address from one of DLLs loaded by the vulnerable executable. After you picked that address and overwrote the PEXCEPTION_ROUTINE field with it, at the time of exception the OS takes that address, determines the .dll module the address belongs to, and if the .dll was compiled with /SAFESEH, checks if that .dll's module safe exception handler table contains the given address. If the .dll was built with /SAFESEH and the address you used is not present in the table, the execution flow is not passed to that address and exploitation fails. If the .dll was built without /SAFESEH, then OS has nothing to check against and this step of the exploitation works.
