28

If I am building a C++ application and I want to make it more difficult to reverse engineer, what steps can I take to do this?

  • Does the choice of compiler affect this?
  • What about compiler flags, presumably a high optimization level would help, what about other flags?
  • Does stripping symbols help and not building with debug symbols?
  • Should I encrypt any internal data such as static strings?
  • What other steps might I take?
35

Compiler

The choice of a compiler has minimal effects on the difficulty to reverse engineer your code. The important things to minimize are all related to information leaks from your code. You want to at least disable any runtime type information (RTTI). The leakage of type information and the simplicity of the instruction set of the virtual machine is one of the reasons CLR and JVM code is easier to reverse engineer. They also have an JIT which applies optimizations to code which may reduce the strength of obfuscation. Obfuscation is basically the opposite of optimization and a lot of obfuscations are solved by first applying an optimization pass.

Debugging information

I would advice you to also turn off any debugging information, even if it doesn't leak any majorly important information today it might do so tomorrow. The amount of information leakage from the debug information varies from compiler to compiler and from binary format to binary format. For instance Microsoft Visual C++ keeps all important debugging information in an external database, usually in the form of a PDB. The most you might leak is the path you used when building your software.

Strings

When it comes to strings you should definitely encrypt them if you need them at all. I would aim to replace all the ones that are for error tracing and error logging with numeric enumerations. Strings which reveal any sort of information about what is going on right now in your binary needs to be unavailable. If you encrypt the strings, they will be decrypted. Try to avoid them as much as possible.

System APIs

Another strong source of information leakage is imports of system APIs. You want to make sure that any imported function which has a known signature is well-hidden and can not be found using automatic analysis. So an array of function pointers from something like LoadLibrary/GetProcAddress is out of the question. All calls to imported functions need to go through a one way function and needs to be embedded within an obfuscated block.

Standard runtime libraries

Something a lot of people forget to take into consideration is the information leaked by standard libraries, such as the runtime of your C++ compiler. I would avoid the use of it completely. This is because most experienced reverse engineers will have signatures prepared for a lot of standard libraries.

Obfuscation

You should also cover any critical code with some sort of heavy obfuscation. Some of the heavier and cheaper obfuscations right now are CodeVirtualizer/Themida and VMProtect. Be aware that these packages have an abundance of defects though. They will sometimes transform your code to something which will not be the equivalent of the original which can lead to instability. They also slow down the obfuscated code significantly. A factor of 10000 times slower is not uncommon. There's also the issue of triggering more false positives with anti-virus software. I would advice you to sign your software using a reputable certificate authority.

Separation of functional blocks

The separation of code into functions is another thing which makes it easier to reverse engineer a program. This applies especially when the functions are obfuscated because it creates boundaries around which the reverse engineer can reason about your software. This way the reverse engineer can solve your program in a divide and conquer manner. Ideally you would want your software in one effective block with obfuscation applied uniformly to the entire block as one. So reduce the number of blocks, use inlining very generously and wrap them in a good obfuscation algorithm. The compiler can easily do some heavy optimizations and stack ordering which will make the block harder to reverse engineer.

Runtime

When you hide information it is important that the information is well hidden at runtime as well. A competent reverse engineer will examine the state of your program as it is running. So using static variables that decrypt when loaded or by using packing which is completely unpacked upon loading will lead to a quick find. Be careful about what you allocate on the heap. All heap operations go via API calls and can be easily logged to a file and reasoned about. Stack operations are generally harder to keep track of just because of how frequent they are. Dynamic analysis is just as important as static. You need to be aware of what your program state is at all times and what information lies where.

Anti-debugging

Anti-debugging is worthless. Do not spend time on it. Spend time on making sure your secrets are well hidden independent of whether your software is at rest or not.

Packing and encrypting code segment

I will group encryption and packing into the same category. They both serve the same purpose and they both have the same issues. In order to execute the code, the CPU needs to see the plain text. So you have to provide the key in the binary. The only remotely effective way of encrypting and packing code segments is if you encrypt and decrypt them at functional boundaries and only if the decryption happens upon function entry and then re-encryption happens when leaving the function. This will provide a small barrier against dumping your binary as it is running but is must be coupled with strong obfuscation.

Finally

Study your software in something like the free version of IDA. Your goal is to make sure that it becomes virtually impossible for the reverse engineer to find a steady mental footing. The less information you leak and the more changing the environment is, the harder it will be to study. If you're not an experienced reverse engineer, designing something hard to reverse engineer is almost impossible.

If you're designing a copy protection system prepare for it to be broken mentally. Make sure you have a plan for how you will deal with the break and how to make sure the next version of your software adds enough value to drive upgrades. Build your system on a solid ground which can not be broken, do not resort to generating your own license keys using some custom algorithm hidden in the manner I described above. The system needs to be built on a sound cryptographic foundation for unforgeability of messages.

  • 5
    "use inlining very generously and wrap them in a good obfuscation algorithm" <- just add Boost. Boost + inlining + LTCG = hell on wheels. Tens of copies of the same functions with different registers used to pass arguments and differently inlined subfunctions, twenty kinds of smart pointers, argh! – Igor Skochinsky Mar 21 '13 at 14:24
  • Haha, yeah you can definitely pick complex heavily templated code in order to get some free obfuscation. Wasn't there an article on /r/re showing the use of templates only in order to create a pure C++ obfuscation framework with opaque predicates and using template meta programming to generate random numbers using a linear congruential generator? I seem to remember one at least. – Peter Andersson Mar 21 '13 at 14:46
  • "So using static variables that decrypt when loaded [..] will lead to a quick find". What is the alternative? – user2005 Jun 20 '14 at 21:41
  • @Sosukodo I would put the variables into a local cache, probably on the stack, decrypt them there, use them, then zero out the memory. I would also make sure that the decryption algorithm, usage and key is completely embedded in VM based obfuscation. It's important that there's not an easy way of finding and decrypting all variables by some automated means. Embedding the key into virtualized code and not into data makes this much harder. It's of course very hard to make it impossible but we're trying to make the work of the reverse engineer as tedious and boring as possible. – Peter Andersson Jun 21 '14 at 10:18
  • I agree - we're just trying to build a better lock. Can you give any links to articles that discuss these things you mention? – user2005 Jun 21 '14 at 15:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.