I am trying to catch anti debugging routines, and I don't quite understand what INT3 does, and I have read that it can be used to detect debuggers and mask calls. The program eventually sends me into a nonsensical loop if I am stepping, and if I am running, it will notify me that a debugger has been found.
77BE0000 8B4424 04 MOV EAX,DWORD PTR SS:[ESP+4] 77BE0004 CC INT3 77BE0005 C2 0400 RETN 4 77BE0008 > CC INT3 77BE0009 90 NOP 77BE000A C3 RETN 77BE000B 90 NOP 77BE000C > CC INT3 77BE000D C3 RETN
I think INT3 might be something going on here. Since my intel instruction set reference (http://faydoc.tripod.com/cpu/index_i.htm) says this has something to do with a debugger interrupt. I also read something about how these can be used to mask debugger detection from reverse engineers.
Am I barking up the wrong tree or is there something in this line of code questionable?