I am trying to catch anti debugging routines, and I don't quite understand what INT3 does, and I have read that it can be used to detect debuggers and mask calls. The program eventually sends me into a nonsensical loop if I am stepping, and if I am running, it will notify me that a debugger has been found.

77BE0000   8B4424 04        MOV EAX,DWORD PTR SS:[ESP+4]
77BE0004   CC               INT3
77BE0005   C2 0400          RETN 4
77BE0008 > CC               INT3
77BE0009   90               NOP
77BE000A   C3               RETN
77BE000B   90               NOP
77BE000C > CC               INT3
77BE000D   C3               RETN

I think INT3 might be something going on here. Since my intel instruction set reference (http://faydoc.tripod.com/cpu/index_i.htm) says this has something to do with a debugger interrupt. I also read something about how these can be used to mask debugger detection from reverse engineers.

Am I barking up the wrong tree or is there something in this line of code questionable?


This technique uses the fact that when the interrupt instructions INT3 (breakpoint) and INT1 (single-step) are stepped thru inside a debugger, by default, the exception handler will not be invoked since debuggers typically handle the exceptions generated by these interrupts. Thus, a packer can set flags inside the exception handler, and if these flags are not set after the INT instruction, it means that the process is being debugged. Additionally, kernel32!DebugBreak() internally invokes an INT3 and some packers use the said API instead.


  • Thank you very much for the link! If you have any other tips or data regarding catching non-trivial anti-debugging code, I'd love to read it. I'm still reading all the info (as this links out to several others), hopefully I get to the part it discusses circumventing it.
    – Chemistpp
    Aug 3 '15 at 5:51

Not the answer you're looking for? Browse other questions tagged or ask your own question.