I'm reversing a code that is heavily obfuscated. It uses opaque predicates, call stack tampering, junk code and control flow flattering. Call stack tampering uses indirect branches (push register / retn) calculated with complex operations at runtime. It seems near impossible to reverse engineer. What would be the correct way to deal with code that uses many obfuscation techniques? I know some people use dynamic symbolic execution but I want to know if there are other ways to deal with this. Here is an example of one obfuscated function hex-ray output:
Usually, I try to detect which obfuscator has been used. Knowing your enemy is the first step towards the victory. If you can detect the obfuscator, you may be lucky and find the corresponding "deobfuscator". You won't have the original source, but you'll significantly decrease the entropy and the noise.
A good starting point is to look at how the strings are handled. Every obfuscator has its own way to do that, and it could be a clear indicator.
Another place I usually look for is the binary initialization: each obfuscator bootstraps it's own helper functions in a different way.
When I am unlucky, I tend to rely on symbolic execution combined with dynamic analysis via ollydbg and radare2.
To add on too Yennefer's wonderful and respectable answer:
Going through the memory segments can definitely help, a lot of times obfuscators will tamper with those. Giving a possible clue as to what the obfuscator is.
Also look at the imported functions, how do those look? Some obfuscators mess around with the IAT, and some of the functions inside of the IAT can be tampered with, but "signature" code from those functions can be found, which can lead to what the obfuscator is.
Sometimes, there is even pre-built programs lying around the web that can automatically deobfuscate the program for you.
Good luck! :-)