As someone who has worked on a suite of Python decompilers, I have thought about this a bit.
I am going to take a different tack than other bottom-up answers; I start to describe the situation from a slightly more general and philosophical side which I think is easy to see.
Think of this as a common situation where someone is trying to reconstruct an object from its fragmented pieces. For example why might a smashed sandcastle be harder to reconstruct than a broken ceramic vase? From this point of view, there would be no surprise. For sure, a real explanation of the sand versus ceramic problem would involve adhesion strength of the materials, properties of fracturing for the materials involved, and mending techniques. Similarly, the properties of computer languages, compilers, runtime systems, machine code and decompilation techniques can vary just as much as sand castles and ceramic vases.
A little background
In general, compilation performs a kind of information entropy loss. This may seem surprising since the source code and the binary object are supposed to be semantically identical. Yes, that is true, but only with respect to how computers understand the world.
And although humans were the inventors or creators of computers, it seems strange that the world in which the computer operates, its "experience" of the world, is already so different than the experience of a human programmer.
It is easier to build a machine that just understands how to perform logic, arithmetic, read memory locations, and jump to different program states than it is to build a computer that inherently understands concepts in programming and its patterns and paradigms, algorithms, or data structures which is the level that programmers may operate in.
So in a sense, one thing that fascinates me here is the inherent problem of communicating between different forms of "intelligence". If it is this hard here, the simplistic idea that we would ever be able to communicate with some sort of alien life where our level of shared experience may be much less is, in my opinion almost hopeless. In addition to the problems seen in decompilation, you have additional problems with transmission speed and delay, and just topics that we might both be interested in.
As an example, using the cited "Universal Language" of Mathematics, if intelligent life is in a highly curved geometry where all "parallel" lines always touch or always diverge, then all this Euclidean Geometry would be super boring for that alien since it would have no practical use. And Shakespeare's Poetry - forget about it!
Factors that go into the level of goodness in translation
Although the question posed here is pretty specific, it is also a bit general or maybe vague. Here are factors that go into the ease (and thus quality) that a decompiler can provide:
- What was the source language used?
- How long, sophisticated and complex is the code to be decompiled?
- What specific compiler was used?
- What compiler/translator options were specified along the various pipelines that were involved in the translation?
- Which kind of native code?
- Which specific decompiler was used and what techniques were used there?
- Was there any other packaging, encryption, or code obfuscation done?
Before delving into each of the categories above, let me first give an analogy using that general principle that basically we are going against information entropy or information loss.
Why is it that reconstructing a sand castle is harder than a vase that has been shattered?
I suppose this question is a bit general as was question posed. The answer partially depends on how much effort was used to create the initial object, sand castle versus a ceramic vase. One can imagine a very simple primitive sand castle that does not have a lot of detail. Maybe it is just a box of some form. That would be easy to reconstruct while an intricate sand castle would be very hard to reconstruct.
The complexity of the program is analogous to the shape of the object we are trying to reconstruct. If your source code or machine code is small and simple, you'd might just dispense with the decompiler and try to understand the machine code.
And the material for the end result and processes used to create it are important too.
Reconstructing things made out of sand is much harder than ceramic that has a couple of well-defined factures. This is like the kind of object that gets run: binary code or bytecode. For bytecode, some are higher level then others. Python bytecode, Pascal P-code or GNU Emacs bytecode is more like ceramic than ARM assembly which is more like sand.
Also, how our sand castle or vase were destroyed is important too. Was the sand castle just kicked once or was at worn away by the tide repeatedly? And was the vase just accidentally dropped on the floor once or was it smashed to little pieces by a hammer?
The analogous process here has the compilation process. Was this a "one-pass" compiler like CPython or a multi-stage compilation pipeline that say GNU gcc or clang do where levels of "optimization" have been turned on?
And finally we get to the amount of effort or care you want to put in to recreating the initial object. If this vase is not of much value to you, you probably won't bother to do a careful and accurate job if you bother at all. Getting something approximately vase like, might be good enough for your needs. However if the vase is precious, well then you probably will spend a lot of effort to reconstruct the original. In fact when my mother broke a vase that she considered a precious heirloom, she hired professionals from the Smithsonian Museum in Washington DC to reconstruct the part of the vase that was broken.
Some of the Details
Having the above out of the way, we can get to the nitty gritty. Just as with sand castles versus ceramic vases, I'll focus on two examples which are at opposite extremes.
But first some factors to consider.
Decompilation quality depends on:
- the language the code was written in
- The compiler you started with
- compiler flags,
- code object
- the specific decompiler used
- Additional packaging and/or deliberate obfuscation
I will give an example that I know very well.
- language: Python (versions matter, but overall let's say in the 1.5 to 3.8 range)
- compiler: CPython
- compiler flags - none (which is the default)
- code object: : High-level custom bytecode
- specific decompiler: uncompyle6 or decompyle3 (for 3.7 and 3.8)
- maintenance effort of decompiler: basically one unpaid volunteer person, me
Many people have noted that this decompiles very well. Here are the factors why this is so:
Python stores docstring comments in the code which is nice for humans looking at decompiled code, although it doesn't really change the decompilation process.
Also there are standards for Python code formatting. So if the human and decompiler follow the same standard and there is basically only one, the the result will look similar.
The CPython compiler is a "one pass" compiler that doesn't do much in the way of code improving or code transformations. But as we move to later versions, More recent versions of CPython has been doing more here.
Python bytecode is also extremely high level. Variable, class, function and module names are all preserved. Python and Python bytecode are loose with type declarations.
The Python decompiler that I have been working on and maintaining makes use very specific idioms that can be found in instruction patterns. Since early Python translated everything one way (even though in theory there are many ways that it could choose), this kind of pattern driven approach is able to disambiguate between semantically identical code. For example
if x: if y: ... versus
if x and y. Earlier CPython create different kinds of code fragments for nested
Here I compare that with something at the other extreme and that I know less well. Some aspecs I know only vaguely.
- language: C
- compiler gcc
- compiler flags -O (typically used)
- code object ARM
- specific decompiler: Ghidra?
- maintenance effort of decompiler: probably more than one person, paid either via grant or as part of some other job-related duty
Python docstrings are, I suppose, a form of comment. In C any sort of document commenting is form of a comment; and all comments in C are stripped.
GCC is at least 3 or 4 distinct phases. C preprocessor, C compiler to assembly, assembly, and then a linking phase. The C compiler phase though can make several passes over the AST it produces, and or the instructions it produces. There is a lot of opportunity for code mangling that may need to be undone.
Unless you have symbol table generation turned on,
-g, the mapping of memory locations and/or registers to names is gone. At the assembly level, any structure or type information is gone.
C indention can come in one of several varieties. One could run the result through a formatter, assuming the Ghidra produces valid C. But I suspect it doesn't.
It is very possible that nested
&& could compile to the same instructions. And there may be more than one template that a compiler might use for a single construct.
But even if a particular compiler for a particular language is passed a particular set of options that compiles
&& differently, I doubt that the way Ghidra's decompiler hones in on a specific compiler's code idioms for most things.
My understanding is that the decompiler(s) in Ghidra are general purpose which means that they do not take into consideration that specific language, compiler or compiler options used. They work the same way on machine code whether the compiler used was
clang or some vendor's C compiler like the ones IBM or Intel have. Or whether the source language was C++ instead of C.
And this is an interesting aspect too. It takes a bit of effort to teach a decompiler about some specific compiler's quirks or habits. These things can change over time depending on the language and within a language like C, C compiler and compiler version, and the assembly language that is produced. Since there are:
- many different kinds of front end languages,
- many different translators,
- compilers releases constantly changing code-generation methods
- the variations in code generation for the same construct (sometimes randomness may play a role here), and
- many different back ends
it probably isn't worth the maintenance effort to hone in on idiosyncrasies of a particular compiler system at a particular point in time.
In the CPython/Python case, there is basically only one compiler CPython. Bytecode does vary from major release. For example between 3.5 to 3.6 there were a number of bytecode changes. However a Major release is on the order of a year. And many times the code generation doesn't change. However Python bytecode does vary a lot more than many other kinds of bytecode. Most bytecode stays the same. JVM stays largely the same. Emacs bytecode and P-Code stay the same over longer periods of time.
Because Ghidra is general purpose, it has a general purpose algorithm for detecting control flow. In uncompyle6 and decompyle3 we can do pretty well using the patterns approach. There has been some problem with control flow in the past, but recent not public versions of this code do very well by adding basic block and dominator information as pseudo instructions of the bytecode. Basically you can think of the instructions having additional parenthesis and comma marks to help detect nesting versus sequencing around jumps.
As for maintenance, I imagine there have been more than one person working on the decompilation aspect. And I imagine that person is funded at least partially by grants if this is not a part of the person's day job.
I mention the maintenance aspect because the person-hours that are spent on decompilers is much less than the person-hours spent in the compiler code generation.