Reverse Engineering Stack Exchange is a question and answer site for researchers and developers who explore the principles of a system through analysis of its structure, function, and operation. It only takes a minute to sign up.
Sign up to join this community
I have linux binary, written in C. During runtime I calculate the checksum of the binary, and it seems that this prevents setting up gdb software breakpoints in it. How safe is this from a reverse engineering perspective? Is there a way to step through on the assembly of this binary?
EDIT:
The code calculates checksum for a part of itself. I have tried setting up breakpoints in gdb, and the checksum failed.
There's lots of ways your specific scheme can be broken - as perror says in his comments, setting hardware breakpoints will be unnoticed. The hardware breakpoints will also allow a reverser to check which part of your code reads the memory, pointing the attacker to the checksum algorithm. So, checksumming the process memory won't delay an experienced reverser by any significant amount.
If you want some modest protection, you should at least do some ptrace tricks to check if you're being debugged, as perror says in his comment. To get some better protection, set up ptrace in your main process, fork a child process, have the child throw exceptions in various places, catch these exceptions in the main process, have the main process do some of the work and patch the result back into the child's address space, and resume the child. Also, spawn something that will try to ptrace the parent process, which will fail if the parent is being run under the control of a debugger.
This will at least prevent attackers who have nothing than gdb in their toolbox to fail. It will not help, for example, against anyone who compiles their own linux kernel and adds a sufficient amount of printk debugging output to the ptrace system call.
Really, there are companies who specialize in protecting software, and even they fail if the target is worthy enough / the attacker is determined enough. There are attacks on FlexLM, there are attacks on Themida (just google for Themida cracking tutorial), and not even Microsoft is able to prevent cracked versions of windows from floating around in the internet. As i said in my comment, anything you could invent yourself won't prevent a determined reverser from cracking your software, unless you're a very experienced reverser yourself.
I consider myself moderately experienced, not an expert by far, and i know that i couldn't devise anything that would stop anyone who has my experience for more than a few days at most. There are some members of this forum who could rip anything i'd invent into shreds within a day.
So honestly, if your software is worth protecting at all, get something commercial like flexlm. Any effort you spend on it yourself is wasted time.
ptrace the parent. However I do not really understand the other method with ptrace, which uses exceptions.
Any self-protection in your binary can be detected and removed, it's only a matter of how hard and annoying you want it to be for the reverser. It's impossible by design to avoid someone from bypassing your check.
There are numerous resources about all the existing anti-debug techniques, but it's an endless arms race, just identify who you want to protect your binary from, and set the difficulty accordingly.
PTRACEdetection to make it a bit safer.