I have linux binary, written in C. During runtime I calculate the checksum of the binary, and it seems that this prevents setting up gdb software breakpoints in it. How safe is this from a reverse engineering perspective? Is there a way to step through on the assembly of this binary?


The code calculates checksum for a part of itself. I have tried setting up breakpoints in gdb, and the checksum failed.

  • You don't state how you calculate the checksum of the binary, but checking the file on disk won't influence gdb at all, since breakpoints in memory don't affect the disk file. Also, you can't prevent people from setting a breakpoint at the end of your checksum routine and faking a correct checksum, or intercepting the open call and redirect it to a file with the correct checksum. You can't prevent opening the file in IDA and doing static analysis either. Bottom line: unless you're a very experienced reverser yourself, don't even try to invent your own protection, you're 100% sure to fail. Nov 6, 2015 at 8:35
  • @GuntramBlohm, I modified the question. I do not calculate checksum for the file on disk.
    – robert
    Nov 6, 2015 at 8:46
  • This protection is against software breakpoints, but setting hardware breakpoints will be unnoticed. You should also add a PTRACE detection to make it a bit safer.
    – perror
    Nov 6, 2015 at 8:47
  • This won't do anything from hardware breakpoints, or even things like pin instrumentation since reads get pointed to the uninstrumented code.
    – broadway
    Nov 6, 2015 at 17:29

2 Answers 2


There's lots of ways your specific scheme can be broken - as perror says in his comments, setting hardware breakpoints will be unnoticed. The hardware breakpoints will also allow a reverser to check which part of your code reads the memory, pointing the attacker to the checksum algorithm. So, checksumming the process memory won't delay an experienced reverser by any significant amount.

If you want some modest protection, you should at least do some ptrace tricks to check if you're being debugged, as perror says in his comment. To get some better protection, set up ptrace in your main process, fork a child process, have the child throw exceptions in various places, catch these exceptions in the main process, have the main process do some of the work and patch the result back into the child's address space, and resume the child. Also, spawn something that will try to ptrace the parent process, which will fail if the parent is being run under the control of a debugger.

This will at least prevent attackers who have nothing than gdb in their toolbox to fail. It will not help, for example, against anyone who compiles their own linux kernel and adds a sufficient amount of printk debugging output to the ptrace system call.

Really, there are companies who specialize in protecting software, and even they fail if the target is worthy enough / the attacker is determined enough. There are attacks on FlexLM, there are attacks on Themida (just google for Themida cracking tutorial), and not even Microsoft is able to prevent cracked versions of windows from floating around in the internet. As i said in my comment, anything you could invent yourself won't prevent a determined reverser from cracking your software, unless you're a very experienced reverser yourself.

I consider myself moderately experienced, not an expert by far, and i know that i couldn't devise anything that would stop anyone who has my experience for more than a few days at most. There are some members of this forum who could rip anything i'd invent into shreds within a day.

So honestly, if your software is worth protecting at all, get something commercial like flexlm. Any effort you spend on it yourself is wasted time.

  • Thank you for the answer. I have found examples where the child process tries to ptrace the parent. However I do not really understand the other method with ptrace, which uses exceptions.
    – robert
    Nov 10, 2015 at 9:26
  • By the way, it is much simpler trying to trace itself, instead of creating a child process. Is there any gain using the child process?
    – robert
    Nov 10, 2015 at 9:38

Any self-protection in your binary can be detected and removed, it's only a matter of how hard and annoying you want it to be for the reverser. It's impossible by design to avoid someone from bypassing your check.

There are numerous resources about all the existing anti-debug techniques, but it's an endless arms race, just identify who you want to protect your binary from, and set the difficulty accordingly.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.