I'm guessing that tools like PeiD detect packers by looking for signatures inside the packed binary and by measuring the entropy. Is there any other work that tries to determine if a binary is obfuscated in some way based on some property of the binary? Would this be able to detect control flow obfuscations as in the case of ollvm?
I wouldn't say that obfuscation detection is strictly related to entropy. When I detected obfuscated code areas I did it by simple statistical comparison of probabilities of appearances of specific assembly commands.
For example number of jump related commands will be significantly larger in a binary with control flow obfuscation.
Number of arithmetic related commands will be significantly larger in a binary with opaque predicates.
Number of some specific code sequences such as
jmp $+5 or
push addr; ret will be significantly larger when such an obfuscations are applied.
These heuristics usually working pretty well on the code which is not packed (packed code, however, can be detected by entropy measurement and existence of segments related to well known packers or with unknown meaning).
I think that, in general, it may be difficult to distinguish obfuscated code from code generated by a compiler that is either extremely bad or extremely good.
- A compiler that does a lousy job of code generation/optimization will generate all sorts of unnecessary/dead instructions.
- As examples of high levels of optimization leading to difficult-to-understand code, consider Massalin's work on superoptimizers, and GCC's use SSE2 instructions and/or aggressive instruction scheduling, e.g., under gcc -Ofast. Switch statements lead to all sorts of interesting code generation possibilities that may look a lot like obfuscation.
Extreme cases of obfuscation are relatively esay to detect. But an obfuscator that's trying to be stealthy will not be easy to detect.