I'm not asking about the difficulty of reverse engineering in general, but rather the difficulty of reverse engineering a particular hypothetical target. We'll assume that having the source code requires no reverse engineering and is thus the easiest. How does reverse engineering from object code, bytecode, and machine code compare? How would obfuscation affect the difficulty for each of the four formats? I'm looking for answers that try to quantify these comparisons.
IMHO, in general, order of difficulty:
- Object code
- Machine code
- Byte code
My reasoning is such:
Object code is easiest because it will usually contain lots of symbol or debug information. Even when it doesn't you have a bit of extra knowledge, since you know that all the code in that file relates to one compile object. That's pretty useful to know, and aids analysis (unless it's a giant monolithic program).
Machine code is next easiest because machine architectures are, in general, very well defined and very well documented. All you need is a disassembler and the code is at your mercy.
Byte code varies between easy and super hard because it can basically be anything. The byte code by itself is useless without knowledge of the virtual machine that it is written for. For instance, byte code for a JVM is easy, since we have a well defined spec for the JVM. Byte code generated for a Themida VM however, is difficult, because it's randomised and you have to analyze that particular VM before you can interpret the byte code in a meaningful (which itself is quite difficult).
Obfuscation almost always makes it harder to analyse the code underneath. But obviously it depends on the level of obfuscation. Some are trivial to bypass with a simple script, some can be bypassed by jumping over the obfuscated code (i.e junk), and others are highly complex and require complex emulators or step-by-step analysis.
It's pretty hard to quantify these differences without designing a specific multi-criteria analysis framework.
Just to add something from obfuscation perspective, I'd like to add to a good answer by @fileoffset.
Each code transformation (such as compilation, linking and obfuscation) on the way from the source code to production executable (probably obfuscated) looses some information and optionally adds some white noise . The simplest example of such information is comments in code. The simplest example of noise is indirect calls through PLT/GOT.
This information may be categorized as follows:
- Whole program semantics, such as algorithms, algorithms execution order, modules, revisions, comments, and relations of all this to all the program, etc. This information will be removed during the compilation from the program to the native code of the platform where it will be executed if the platform doesn't need it (which is not always the case for Java, .NET, and ActionScript for example because modules and classes are part of the platform low level language, but in contrary this is definitely correct for compilation to ARM, IA32-64 and other real world processors). This category of information can be corrupted by obfuscation only in case where it exists in the executable as a part of platform language, and that's exactly one of java obfuscators working methods.
Basic algorithm building blocks, such as loops, functions, conditional expressions, structures, global variables, calls to other functions, function prologues and epilogues, and almost everything else you can see within a function. Compilation will loose significant part of this information, but it may be recoverable from the context. This category of code artifacts can be significantly corrupted by obfuscation by using creative and non-standard patterns for standard tasks, for example control flow changes such as control flow flattening, loop unroll, not needed stack manipulations, etc. Please note that the obfuscation strength here depends on a platform requirements.
Platform level abstractions, such as registers, stack, single instructions, flags, etc. This may be obscured by obfuscation, for example virtual machine based obfuscators are just inventing random virtual machine on the fly. In addition most of obfuscators are adding not needed instructions.
Reverse engineering is a process of recovering lost information from the code, from lowest level up to the highest if it possible at all. This process is based on recognition of standard patterns. This process gets harder if standard patterns are removed. The number or order of magnitude of removed/changed/corrupted/obscured/white-noised standard constructions may be used as an incomplete and bad measure of obfuscation strength. Unfortunately there are no other reasonable measures I know about.
Let's try to re-categorize your formats from this perspective in order of obfuscation removing efforts needed.
- Source code
- Object code
- Not obfuscated production code for high level well known platforms such as .NET, ActionScript, and Java
- Obfuscated production code for high level well known platforms such as .NET, ActionScript, and Java
- Not obfuscated native code
- Native code obfuscated without Themida like VM
- Native code obfuscated with Themida like VM
The best (and funniest) heuristic to check and justify the order above is a cost of decompiler (which is automated reverse engineering tool) for corresponding platform, its existence and its quality.
For example some Java decompilers can do automatic partial deobfuscation and can almost always reconstruct the code on a class method level with cost of license $0
Native code decompiles, in contrary, are either doesn't work well enough (none of them reaches quality of JAD, very old java decompiler) or cost a lot such as Hex Rays decompiler, the only stable working decompiler I know (and I tried a lot of others).
By the way, there are some articles that trying to measure obfuscation strength, such as this, for example (the article is almost useless for real life binaries, but the list of references is interesting). None of them that I know working with reasonable size of executable under test. Most of them working with not realistic assumptions and/or exploiting specific obfuscation algorithm flaws.