I'm adding copyright protection to a demo version of a shared library. I've seem some recommendations to add timing around license violation detection codes -- which would be debugger + patch targets -- for example RDTSC, GetTickCount(), etc. Unfortunately this shared library doesn't have timing related codes, so adding would stand out and make them relatively easy to find (I see that IDAPro can find all all occurrences of a single instruction such as RDTSC, so I assume OllyDbg can also). Another suggestion is to strip section headers, but that seems applicable only for an exe, not a shared library.

Currently, what is the leading edge way to either prevent a debugger from handling a shared library, or detect inside the lib that's it being debugged ?

  • Do you have the source code of this library (so you can modify it) or only compiled version? Which operating systems is it created to run on?
    – bart1e
    Commented Jul 9, 2019 at 11:35
  • @bart1e, yes I have source (mostly C, a couple of C++ files). Linux Commented Jul 10, 2019 at 17:07

1 Answer 1


I don't know whether the advice I'll give you is the "leading edge way", but I will describe some steps that you can perform in order to protect your software.

Is it possible to protect it such that no one will be able to crack it?

No, and it's important to know that and keep in mind that what you really want to do is to keep it as hard to crack as possible and dishearten potential violator from doing that. So, you want to make it so complex and obfuscated that it would require a lot of work to change anything both statically and during the runtime.

Anti-debugging techniques

  • Software breakpoints detection

    As you probably know, to trigger a software breakpoint, debuggers use int 3 instruction, i.e. when you put a software breakpoint at some address, debugger will overwrite the byte at that address to 0xCC (int 3 opcode) and when this instruction triggers the interrupt, a debugger will restore the byte that was previously there. That means that every software breakpoint will change some byte in the code section, so to detect it, you may create a function searching for 0xCC.

  • Control sum checking

    It's a generalization of the previous technique. Instead of detecting only 0xCC, you will detect any change made in code, including fragments replaced by NOPs. You want to apply it at the end of protection process so that you don't need to change required control sum value at each change made in the code. I would advise you to put functions checking checksums at many different places in your code and if possible make each of them slightly different (so that it's not so easy to find them all at once).

  • Hardware breakpoint detection

    Since hardware breakpoints don't require any changes in a process image, they cannot be detected using the above mentioned methods. They are implemented using DR registers and it's possible to create 4 of them as a maximum at given moment. To defend against them, you can just reset their values from time to time. More info about debug registers.

  • Time / cycles checking

    Since you have the source code, you can import other libraries even during runtime. On Linux you can do it via dlopen (to load library) and dlsym to get specific procedure from it.. Of course you should encrypt every string so that it's not straightforward to find out which functions you are using for measuring the time elapsed.

Other tricks

  • Inlining every function

    That will stop debuggers from recognising library functions, thus will be very annoying for everyone who wants to analyse your code. Of course, using C++ templates will also help. ;)

  • Inserting junk code

    You can put (and inline of course) many garbage functions that won't change your program behaviour, but it won't be obvious for person analysing disassembly. You don't have to write them yourself - you may use library functions as well.

  • Being nasty

    Of course, if someone is determined enough, he will eventually find all your control sum / breakpoint checking functions and NOP them out. But what if you put there some code that will have an impact on the rest of application? Consider:

  bool isBeingDebugged
    if (controlSum(address) & rand() == requiredValue)
      return true;
    return false;

Above function not only checks for code integrity, but also modifies the global state of a program. For instance, if you use later on some string decoding function relying on rand() results, the application will likely behave differently if that function is simply replaced by NOPs.

  • Being creative

    The best anti debugging technique is such that no one expects, so search, experiment and apply your own ideas.

Additional information

  • excellent, comprehensive answer. How do I mark it as the accepted answer ? Can you clarify one thing: in your "bool isBeingDebugged ... " example, I don't understand the use of rand(). Are you saying (a) rand() is the C lib rand() and would return a sequence of values the same for every Linux and could thus be used as part of a license violation detection code, and (b) it would be difficult for someone to nop out just part of that code, because then the expected rand() value sequence would change ? Thanks. Commented Jul 12, 2019 at 17:49
  • To accept an answer, you can click "tick" symbol next to (under?) upvote panel.
    – bart1e
    Commented Jul 12, 2019 at 18:02
  • a) That was just an example, you can use some other function instead. Yes, I mean just rand() function that will return pseudorandom values, thus the same among program executions (for the same starting seed).
    – bart1e
    Commented Jul 12, 2019 at 18:03
  • b) No, but it can be surprising first of all, because usually debugger detecting functions will not affect the global state - just check for debugger. And potential hacker will have to be very careful not to nop some important part of code, so he has to fully understand it, which may be very time consuming and difficult.
    – bart1e
    Commented Jul 12, 2019 at 18:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.