I'm guessing that tools like PeiD detect packers by looking for signatures inside the packed binary and by measuring the entropy. Is there any other work that tries to determine if a binary is obfuscated in some way based on some property of the binary? Would this be able to detect control flow obfuscations as in the case of ollvm?
2 Answers
I wouldn't say that obfuscation detection is strictly related to entropy. When I detected obfuscated code areas I did it by simple statistical comparison of probabilities of appearances of specific assembly commands.
For example number of jump related commands will be significantly larger in a binary with control flow obfuscation.
Number of arithmetic related commands will be significantly larger in a binary with opaque predicates.
Number of some specific code sequences such as jmp $+5 or push addr; ret will be significantly larger when such an obfuscations are applied.
These heuristics usually working pretty well on the code which is not packed (packed code, however, can be detected by entropy measurement and existence of segments related to well known packers or with unknown meaning).
-
One of the tricks I use to detect obfuscation is to count the number of functions and the number of basic blocks and then check the exit basic blocks to determine if the "exit instructions" make any sense. May 18, 2015 at 7:32
-
Counting (number of function prologues + number of function epilogues)/(code size) may be also good heuristic.– w sMay 18, 2015 at 7:33
-
2Generally speaking there a lot of different metrics which can be relatively simply accounted. Relation between code and data sizes, number of jumps, number of ALU ops, number of load/store operations related to code size - all this may give a lot of information. Using the famous triad of multidimensional statistics (factor, discriminant and cluster analysis) on a these metrics should give a good detection ratio.– w sMay 18, 2015 at 7:39
I think that, in general, it may be difficult to distinguish obfuscated code from code generated by a compiler that is either extremely bad or extremely good.
- A compiler that does a lousy job of code generation/optimization will generate all sorts of unnecessary/dead instructions.
- As examples of high levels of optimization leading to difficult-to-understand code, consider Massalin's work on superoptimizers, and GCC's use SSE2 instructions and/or aggressive instruction scheduling, e.g., under gcc -Ofast. Switch statements lead to all sorts of interesting code generation possibilities that may look a lot like obfuscation.
Extreme cases of obfuscation are relatively esay to detect. But an obfuscator that's trying to be stealthy will not be easy to detect.
