18

I am trying to scan all possible techniques to disrupt the usage of a debugger on a Unix platform (ie POSIX and a bit more).

I am thinking about techniques such as the PTRACE test at the beginning of a program (or at various points of its execution), the insertion of fake breakpoints (eg int3/0xcc x86 opcode) or time checks. But, also global strategies defined on the program to slow down the analysis of the program.

For now, all the techniques I found on Internet were easily worked around as soon as the anti-debug technique has been understood. So, I wonder if there are stronger ones.

|improve this question
20

Here's some I've seen or heard about:

  • Stripping the section headers. A simple and fully legal action that stops GDB dead in its tracks. Does not work against some other debuggers (e.g. IDA). Can be done using the sstrip tool.

  • Using syscall function or direct syscalls instructions instead of calling specific functions like ptrace(). Can be defeated by setting the breakpoint on syscall function or just stepping through the file, but may be non-obvious if you don't know about it.

  • Performing anti-debugging actions before main(), e.g. in constructors of global objects or using __attribute__((constructor)). Since GDB usually sets initial breakpoint in main(), it takes care of the default situation. Workaround is simple: put breakpoint on the actual file's entrypoint (info file in GDB).

  • Sending itself debugging-related signals like SIGTRAP. (Note that this can be ignored with handle SIGTRAP nostop in GDB.)

  • Forking and tracing itself with ptrace.

  • Fake breakpoints insertion: Inserting int3/0xcc will force the debugger to stop on these bytes as they will be treated as software breakpoints. If they are numerous it might slow down considerably the analysis.

  • Breakpoint detection: I saw this technique in this paper, you can attach a function that will be triggered when a breakpoint is encoutered. This paper covers also some other tricks.

|improve this answer
  • Ok, this seems a reasonable list of techniques but nothing surprising til now. And, you are missing the breakpoints tricks (insertion of fake breakpoints to confuse the debugger when PTRACEd and checking the hash of the next code block to see if new software breakpoints have been inserted or not). But, would you know a paper describing all these techniques for Unix (I know some papers for MS-Windows but none for Unix systems). – perror Mar 22 '13 at 17:42
  • @Emmanuel: feel free to edit it and add more, that's why I made it a wiki. I'm not aware of any papers, but possibly you can find some more tricks in something like Phrack. – Igor Skochinsky♦ Mar 22 '13 at 17:46
  • My IDA died while trying to load an ARM binary that had its section headers stripped. So #1 may still be a useful trick (though I ultimately wrote a script to reconstitute the headers). – nneonneo Jun 9 '14 at 1:38
  • @nneonneo: if that happens with the current version, please send a bug report. thanks. – Igor Skochinsky♦ Jun 9 '14 at 21:09
  • The real fun is fork and use ptrace() on yourself as a form of IPC. – Joshua Sep 7 '18 at 4:02
9

My understanding of anti-debugging technology is that it is a game of cat and mouse (or cat and also cat). A technique works until it is understood by the opposing side and then it doesn't work any more. I think that ultimately the advantage is on the side of the debugger. Consider dynamic binary instrumentation or virtual machine systems. You can detect the presence of DBI or emulation by looking for cracks or faults in the emulation of the platform, but those are just errors in the emulation/translation software. If you had a "perfect" emulation system, then a debugged application could not know that it was being traced?

So I think the best you can do is little things that are easily worked around. The Shiva system still I think represents "okay" protection from debugging on Unix platforms, since it decrypts small portions of itself to run on demand and then re-encryptes them, as I recall.

|improve this answer
  • I would tag the Shiva system as an advanced packer but not really an anti-debug technique (even if its main effect is to render the usage of a debugger extremely tedious because your window in memory where you can effectively set breakpoints is extremely small compared to a "normal" program). – perror Mar 22 '13 at 17:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.