Ways to detect Function hooking?

Question

I making a game and I'm trying to find a way to prevent against function hooking (IAT/VMT/etc), or at the least detect it, I understand how they work but dont see a very reliable way to detect or prevent them. Does anyone have any ideas on how to do this? it would be greatly appreciated.

Answer 1

Here are several possible solutions for the problem. Generally specking you will need a combination of watchdog and hidden traps. This will not be a bullet proof but will be good enough to make it hard on the attacker to hook your game.

Vary Basic:

• buy commercial protector for the game

Basic:

• Do not use IAT but resolve everything dynamically.
• Create watchdog thread that will check all critical apis/functions for patches randomly - if the preamble is different from standard one (mov edi, edi or push ebp, mov ebp, esp)

Advanced Basic (in addition to Basic):

• Create watchdog process to check and monitor main process for alternation.
• Main process also will check for the presence of the watchdog process
• Add integrity calculation functions which will check checksums for the critical code blocks for alternation. This could be done from main process and watchdog process.
  • Distribute traps that will check for code validity randomly in various parts of the game. If possible move the checks in the code on every execution of the game (your binary will always be rewritten after each execution)
  • Store the on-disk binary in obfuscated state and unpack it during loading

Advanced (in addition to the above):

• Implement anti-injection techniques to prevent process alternation from outside.
• Control the loading of all the libraries that your need by implementing your own loader, previously validating that they are authentic (signature, checksums...)
• Alternatively check that all system libraries in your process are not local copies using the VAD trees. Most already loaded libraries will be mapped between processes with Copy-On-Write flag to save resources. If the patch was applied anywhere in the library address space, OS will create a local copy.

More Advanced:

• Create kernel driver (right one) to assist on process alternation protection. It would better protect from code injection, memory alternation. It would provide loading protection and etc.

That's for now. I'll think about more options along the way.

Answer 2

This ties into the Turing-old problem of execution prevention. So, I'll try to answer a very general question in a very general way.

Perhaps the most common method for function hooking is editing the IAT at runtime or during execution(Or the constant address table in .rdata for local functions). But detecting function hooks by checking the IAT is completely unreliable. It's like assuming a page that has been torn out of a book is still there because the index says it is. No runtime-filled tables (IAT, VMT, Windows Executable Header) containing references to instructions are guaranteed to provide any conclusive information as to what is actually being executed in memory.

A common method for hooking that is invisible to a program's address tables is applied by placing a jump instruction into the assembly of the targeted function directly after the stack pointer is moved into the base pointer to be pushed and before the stack is expanded.

So, speaking in terms of reliable automatic prevention, we are led to the conclusion that we must derive the conditions upon which our "Hook Detector" detects function hooks from the predictability of our program's execution procedure.