I have been reading a lot about the different techniques for Windows API hooking (a technique I'm particularly fascinated by and fond of), and it seems a major problem in implementing a realiable hook function is ensuring that the hook is written in a way that is thread-safe. Of course, there are some techniques where this problem has been solved or can be solved trivially, such as hotpatching the Windows API, but hot patching is not guaranteed to work on all win32 or third party API functions and the techniques that do support hooking them are not normally thread-safe.
A very common technique that is has problems caused by multithreading is a inline hook that replaces the normal function prologue code with a jump instruction to the hook procedure, and then calls the original function as needed through a trampoline.
There are several inherent issues to the inline hook technique, which makes it a very complicated method to use and debug. A primary issue is as I mentioned, that it is not safe in a orthodox multithreaded environment in the real world. This is due to that when changing the bytes of the function, you can not guarantee that the instruction pointer will not be in the middle of your newly injected code, which may then cause the target application to crash from executing an invalid mix of the old opcodes mixed with the opcodes you inserted.
There are some solutions to this problem, with one being suspending all the threads in the process and then checking the instruction pointer in each thread to ensure that no thread is currently executing the target instructions you wish to replace. Then if there happens to be a thread or two executing that particular function, then you can respond accordingly by doing something such as performing a stack trace to place a breakpoint at the return address, resuming the thread, and then handling the exception when the thread has returned from the target function.
Of course, this method is still unsafe, because there is nothing stopping one sneaky thread from using
CreateThread() before you can suspend all of the running threads in the process (some applications on my computer run with 40+ threads at once). There could even be a related process that uses
CreateRemoteThread() in your target application and then calls the function you are hooking before it's safe.
A solution to that problem could be trying to debug the process and receive notifications of when a process creates a new thread, and then respond by suspending that thread. Of course, many event notification systems provided by the Windows API or a third party API will not be sent in real time, which may allow that thread to perform an unsafe operation before it is suspended.
Another solution could be to statically patch the executable with the hook function before the process is launched, presumably by hooking the EAT/IAT. This is not an option for me because I need to have an implementation that will work process wide, regardless of how a function is resolved or in the event of a new unhooked module calling the function.
There are many other issues to overcome with the inline hook technique that I did not mention. Which brings me back again to my question:
What methodology can be used to change code flow atomically during program execution ?
I was curious to see if there was a more robust solution out there that overcomes some of the shortcomings of the methods I covered in this post.
Please no third party library suggestions for hooking functions. I want to implement my own for the educational benefits.
I prefer hooking technique documentation and examples that use the C programming language.
My processor is an AMD Athlon II X2 250 that is x86-x64 compatible, and my operating system is Windows 7.