Automated Static Unpacking Binaries

Question

I am reading the paper Automatic Static Unpacking of Malware Binaries (Kevin Coogan et al.) with the goal of trying to reproduce the given experimental results (with Hybris-C, MyDoom.q, tElock, etc), and studying how it can be expanded for other cases.

As far as I understand, the authors first use pointer analysis to extract the unpacking code by detecting transition points (i.e. the point separates the execution of the normal instructions from the runtime generated instructions), then the unpacker code is extracted using the backward slicing analysis from this point.

After a process of "code punning" and "reassembly" (e.g. patch out the defense code, fix some relocation problems, etc.), one could obtain a new binary whose each unpacked block is initially marked as a s-object (abbr. for section objects). Then the new binary can be emulated (if I understood correctly) where s-object(s) will be filled out by unpacked code.

While I might be able to imagine some limits of static analysis applied in the paper (e.g. inaccuracies of backward static slicing, pointer analysis, side effects...), and the implicit hypothesis about the existence of the transition points, I still cannot figure out how the described static unpacker works.

First, the authors say that the backward static slicing is applied since the context of the problem is unstructured binaries (and that is true since malicious codes are unstructured), but

Problem 1: how can the value-set analysis be applied?

since we have no hope to restore abstract locations (used by value-set analysis) in unstructured binaries. For example, in the following unpacking stub of Hybris Worm:

    mov edx, 0x135
    mov ebx, 0x401000
    mov eax 0x6bf00803

unpack:
    sub [ebx], eax
    nop
    sub eax, 0x15e3c0
    add ebx, 0x4
    dec ecx
    jne unpack
    jmp _oep

_oep:
    ...

I suppose that there would exist no abstract locations, no?

Second, the authors say that each s-object contains meta-data (i.e. name, size, ...) about some section, as cited in V.B.3.1 of the paper:

an s-object... contains meta-data about the section it presents... These meta-data are obtained from section table of the binary...

But

Problem 2: how can we be sure that the unpacked code must be fit in a section (whose info can be obtained by parsing the binary header)?

That would be probably true in the case of "pure" UPX where the unpacked codes are located in the section UPX0, but this is not true in general (e.g. the case above of Hybris).

Since there is a step of address translation described in V.B.3.1 which arise from the difference between the normal runtime unpacking process of the binary and the unpacking process of the static unpacker, I assume that the output of this unpacker is a new binary contains only unpacked code and no unpacking stub. But

Problem 3: how can it deal with multiple level packed programs?

For example, the experimental results given in the Figure 5 of the paper has the case of Peed-44 which uses a customized UPX contains at least 2 unpacking levels: which form should be considered as the "real" unpacked code?

So my question is

Is my understanding about the paper correct? (that should be not) then where did I misunderstand?

Answer 1

Regarding your first question:

If I understood the 3 papers necessary to answer the question correctly, your abstract locations would be ebx for example.

Now I hate academic papers with a passion but from glancing over your mentioned paper, the quoted paper for the used algorithm for Value-Set Analysis ("What You See Is Not What You eXecute") and in turn the paper quoted there to give an answer what Value-Set Analysis even is ("Generic Value-Set Analysis on Low-Level Code").

Basically, they track some slight abstraction of variables, which include registers (as explained in one of these papers, I don't want to go back and find out which one it was) over some code and observe how they change.

In your example of the Hybris worm, ebx would be an abstract location of interest, and Value-Set analysis would be used to observe what values ebx would assume over the loop code in order to deduce a potential location for unpacked code.

Answer 2

Regarding your 2nd question, usually you're not using sections but memory pages (and their associated permissions). You know the unpacked code will eventually have to be written then executed in a memory page, and you can easily detect such memory pages that have been writable, written to, and then executed.

Regarding your 3rd question, I'd say that a naive approach would be to run until the last stable "transition point" and then by definition, this last layer is the unpacked code. But more complex protections could break this heuristic very fast.

Regarding your first question, I'm not sure I understand any of those words.