(linux) How to debug a "debugger"?

Question

I'm trying to understand a program. This programm implements an antidebug technique.

From what I've understand so far, the father ptrace a forked process.

An the child does something like that:

some code
Int 3
some code
Int 3
some code
Int 3

and so on

I think the father implements a kind of debugger because with strace -i I see a lot of:

[77b7457c] waitpid(2468, [{WIFSTOPPED(s) && WSTOPSIG(s) == SIGTRAP}], __WALL) = 2468
[77b7457c] --- SIGCHLD (Child exited) @ 0 (0) ---
[77baf52c] ptrace(PTRACE_GETREGS, 2468, 0, 0x7fb68be4) = 0
[77baf52c] ptrace(PTRACE_SETREGS, 2468, 0, 0x7fb68be4) = 0
[77baf52c] ptrace(PTRACE_CONT, 2468, 0, SIG_0) = 0

It reminds me this document http://www.alexonlinux.com/how-debugger-works where we found the same ideas: a child process, sending TRAP, catched by the father via ptrace calls, setting registers to the right value, then continue.

My problem now: How can I debug this "debugger"?

• If I bypass the ptrace, the child will get its SIGTRAP and stop.
• If I leave the ptrace, I can't use gdb
• If I bypass ptrace and use gdb with set follow-fork-mode child, I can't follow how and where the TRAP is caught in father process because gdb handle it. And if I send signal SIGTRAP the father detects it (how? that's one thing I want to understand) and terminates.

I know that gdb can't follow both father and child, so is there a way to debug this program?

Answer 1

You can break on the ptrace, bypass it, set it return value as if it was succesfuly called, use two instances of GDB: one for the child, and one for the father.

Answer 2

This binary is protected by a technique called "nanomites".

The father drives the child by getting and settings its registers via debug call (ptrace under linux, DebugActiveProcess under windows). It's an efficient antidebug method.

This was part of a challenge, you can find a lot of details in this post: http://doar-e.github.io/blog/2014/10/11/taiming-a-wild-nanomite-protected-mips-binary-with-symbolic-execution-no-such-crackme/