I'm guessing that tools like PeiD detect packers by looking for signatures inside the packed binary and by measuring the entropy. Is there any other work that tries to determine if a binary is obfuscated in some way based on some property of the binary? Would this be able to detect control flow obfuscations as in the case of ollvm?

  • I'm sure in case of .NET each obfuscation software has some unique "footprint" on the binary produced. If glancing on the binary is not an option :) – Dominik Antal May 18 '15 at 7:20

I wouldn't say that obfuscation detection is strictly related to entropy. When I detected obfuscated code areas I did it by simple statistical comparison of probabilities of appearances of specific assembly commands.

For example number of jump related commands will be significantly larger in a binary with control flow obfuscation. Number of arithmetic related commands will be significantly larger in a binary with opaque predicates. Number of some specific code sequences such as jmp $+5 or push addr; ret will be significantly larger when such an obfuscations are applied.

These heuristics usually working pretty well on the code which is not packed (packed code, however, can be detected by entropy measurement and existence of segments related to well known packers or with unknown meaning).

  • One of the tricks I use to detect obfuscation is to count the number of functions and the number of basic blocks and then check the exit basic blocks to determine if the "exit instructions" make any sense. – joxeankoret May 18 '15 at 7:32
  • Counting (number of function prologues + number of function epilogues)/(code size) may be also good heuristic. – w s May 18 '15 at 7:33
  • 2
    Generally speaking there a lot of different metrics which can be relatively simply accounted. Relation between code and data sizes, number of jumps, number of ALU ops, number of load/store operations related to code size - all this may give a lot of information. Using the famous triad of multidimensional statistics (factor, discriminant and cluster analysis) on a these metrics should give a good detection ratio. – w s May 18 '15 at 7:39

I think that, in general, it may be difficult to distinguish obfuscated code from code generated by a compiler that is either extremely bad or extremely good.

  • A compiler that does a lousy job of code generation/optimization will generate all sorts of unnecessary/dead instructions.
  • As examples of high levels of optimization leading to difficult-to-understand code, consider Massalin's work on superoptimizers, and GCC's use SSE2 instructions and/or aggressive instruction scheduling, e.g., under gcc -Ofast. Switch statements lead to all sorts of interesting code generation possibilities that may look a lot like obfuscation.

Extreme cases of obfuscation are relatively esay to detect. But an obfuscator that's trying to be stealthy will not be easy to detect.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy