Is decompilation significantly more limited when performed on handwritten assembly?

Question

I understand C language decompilers are designed to decompile code produced by a C compiler, and that such decompilers have a lot of limitations and sometimes are actually wrong (in the sense that they produce C code that is not functionally equivalent to the assembly code they are generated from, though I don't understand why they would produce code that was not functionally equivalent).

Does decompilation not really work if you are starting from hand-written assembly? I'm not talking heavily-optimized or obfuscated assembly; I can understand those might break decompilation in some way.

Answer 1

This is more of an extended comment than an empirically researched answer.

Perspective 1: Decompiler Design Assumption Violation

Abstractly, one way that we can think about this is in terms of tool design. A tool tends to perform best when applied to problems its designer developed the tool to solve. Keeping in mind that decompilers perform the reverse of compilation, an argument can be made that using a decompiler to analyze machine code produced from hand-written assembly is an example of applying it to a problem it was not intended to solve, since that machine code was not produced by compilation.

In other words, if it is the case that an assumption made by a decompiler is that the machine code being analyzed was produced by a compiler, then machine code produced from hand-written assembly violates this assumption. Attempting to decompile machine code produced from hand-written assembly amounts to attempting to approximate high-level language (HLL) source code for machine code when HLL source code never existed in the first place.

In the above screenshot we see output produced by Ghidra's decompiler for machine code in the unpacking stub of a variant of mumblehard. The unpacking stub was written in assembly (observe the system calls made directly at locations 0x08048064 and 0x08048095).

Perspective 2: Non-existent Source-to-Assembly Relationship

Decompilation is possible because of the relationship between HLL source (e.g. C++) and assembly code generated from that source - a translation accomplished by the compiler. With hand-written assembly, no such relationship ever existed, and no such translation ever took place. Therefore, when dealing with machine code produced from hand-written assembly, it is a mistake to assume that a meaningful mapping should be derived between that machine code and a HLL by a decompiler. Handcrafted assembly need not bear any relationship whatsoever to assembly produced from HLL code via compilation. If the handcrafted assembly takes on a form that for any reason - simply by virtue of the fact that the programmer has complete control over low-level operations - couldn't have been generated by compiling HLL source, it would not be feasible for a decompiler to synthesize an accurate HLL approximation.

Answer 2

I don't understand why they would be actually wrong

compilers convert high level code in human readable form (like C) to low level code (so that machine could understand it). During compilation most of the semantic as well as syntactic information gets lost due to language transformation, multiple optimizations, etc. Debug symbols may preserve some information (as they are for debug purpose and do not contribute to the actual functioning of the code), but they are generally stripped in COTS binaries. Thus it is sometimes hard (np-hard?) to recover all the information. Consider following program:

int main() {
  int a[10];
  a[4] = 4;
}

main:
        push    rbp
        mov     rbp, rsp
        mov     DWORD PTR [rbp-32], 4
        mov     eax, 0
        pop     rbp
        ret

The program is compiled without optimizations, and you can see that the information related is permanently lost (although, you can try to predict the array based on stack size, but it is impossible to conclude that it is indeed an array).

Does decompilation not really work if you are starting from hand-written assembly?

It may or may not. If a binary is compiled using standard compiler like gcc or llvm, then such compilers use certain format (calling conventions, optimization or code representation style, etc.). But in handwritten assembly, you can essentially do anything (use custom format, different calling conventions, etc). Thus, it is hard to predict the intension of the programmer.

Read more about decompilation correctness in this recently published paper:

Zhibo Liu and Shuai Wang. 2020. How far we have come: testing decompilation correctness of C decompilers. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2020). Association for Computing Machinery, New York, NY, USA, 475–487. DOI:https://doi.org/10.1145/3395363.3397370