Fixing/recompiling a heavily control flow obfuscated function

Question

I've recently come across a binary that has been obfuscated with a very obscure protection.

Obfuscation

From what I understand, the obfuscator is effective in several ways.

• Multiple JMPs via relative addressing (0xE9), JMPs to addresses in stack (e.g jmp [rsp-0x8]), and less frequently, JMPs using registers (jmp reg). These are extremely effective at rendering the CFG in IDA Pro useless.
• Constant stack arithmetic. The obfuscator is continuously using stack memory to hold vital parts of information such as variables and JMPs to the next block (as stated previously). More importantly, the instruction sub rsp, # and add rsp, # and substituted via lea rsp, [rsp+-#]. IDA's analyzer also fails to acknowledge a change in the stack pointer in the instruction lea rsp, [rsp+8].
• Junk code. Instructions that don't exactly have any effect or pertain to the original function's goal.
• Slight use of opaque predicates (however I haven't spotted too many).
• Conditional branching via cmov instructions, e.g

test rax, rax
... ; junk
mov rdx, sub_not_equal
mov rcx, [rsp-0x8]
cmovz rcx, rdx
... ; stack stuff
-> jmp to position in stack containing rcx

Examples

.text:00007FF639A15EA9                                                     loc_7FF639A15EA9:                       
.text:00007FF639A15EA9 048 48 89 04 24                                                     mov     [rsp+48h+var_48], rax
.text:00007FF639A15EAD 048 48 8D 64 24 08                                                  lea     rsp, [rsp+8]
.text:00007FF639A15EB2 040 48 8B 74 24 F8                                                  mov     rsi, [rsp+40h+var_48]
.text:00007FF639A15EB7 040 48 85 C0                                                        test    rax, rax
.text:00007FF639A15EBA 040 E9 F8 10 44 FF                                                  jmp     loc_7FF638E56FB7

.text:00007FF638E56FB7                                                     loc_7FF638E56FB7:                       
.text:00007FF638E56FB7 040 0F 85 F4 C2 B9 FB                                               jnz     near ptr sub_7FF6349F32B1
.text:00007FF638E56FBD 040 E9 33 EE 9F 02                                                  jmp     loc_7FF63B855DF5

.text:00007FF63B855DF5                                                     loc_7FF63B855DF5:                       
.text:00007FF63B855DF5 040 4C 89 74 24 F8                                                  mov     [rsp+40h+var_48], r14
.text:00007FF63B855DFA 040 48 8D 64 24 F8                                                  lea     rsp, [rsp-8]
.text:00007FF63B855DFF 048 48 8B 34 24                                                     mov     rsi, [rsp+48h+var_48]
.text:00007FF63B855E03 048 48 8D 64 24 08                                                  lea     rsp, [rsp+8]
.text:00007FF63B855E08 040 E9 86 B1 74 FD                                                  jmp     loc_7FF638FA0F93

.text:00007FF6392FCD6C                                                     loc_7FF6392FCD6C:                       
.text:00007FF6392FCD6C 048 48 8D 2D 42 65 6F FB                                            lea     rbp, loc_7FF6349F32B5
.text:00007FF6392FCD73 048 48 87 2C 24                                                     xchg    rbp, [rsp+48h+var_48]
.text:00007FF6392FCD77 048 48 8D 64 24 08                                                  lea     rsp, [rsp+8]
.text:00007FF6392FCD7C 040 FF 64 24 F8                                                     jmp     [rsp+40h+var_48]

So on and so forth..

Efforts

My goal was to create a tool that would take an obfuscated routine, follow along every jump, then create a peephole optimizer that would completely erase instruction sequences such as the lea rsp, [rsp+-], xhng reg, [rsp+x] and others shown and replace them with their "deobfuscated" variants. So I began working on a tool that would emulate certain instructions (especially anything involved with the stack) then create my own "virtual stack" that would hold the same information the original program would. This actually worked, except I didn't account for many things, such as JCCs. The end result shown in my tool was over 60 JMPs and over 200 instructions in the final combined dump (this did not even including the multiple JCC branches which can easily have the same or more CF obfuscation). I feel like my attempt was aimed in the wrong direction.

Guidance

Well. Using my tool, I could probably just skim through the entire assembly and figure out what is going on after adding in conditional branch support, however I still feel unsatisfied. My original goal was to produce a completely optimized, decompilable dump, however I feel like I've hit a dead end due to my design flaws.

I want to ask to everyone who has had the patience to read through this post, on their advice carrying forward, whether it's a library recommendation, or how to re-approach the deobfuscation process I had going here.

For context, I'm analyzing a unpacked dump of a process and using my tool coded in C++ using Zydis disassembler to decode and manually emulate some of the instructions.

Answer 1

Since this is obviously a pain to do statically or by hand in a dynamic fashion, I would suggest trying another approach.

I personally would try to do it by writing a small emulator.

Since it is mostly 'stack arithmetic' and junk code, this would be perfect. Take a look at the emulation framework named 'Unicorn'.

With this option, you don't have to re-implement anything, and it will support a lot more cases that a simple 'virtual stack' (as you call it).

You basically have to map the whole binary in memory, setup an initial stack frame, create a hook for every instructions, and maybe create a breakpoints system for debugging. You can then start the emulation, and inspect the program's behavior and the jmp that where taken.

If you want to make the code more readable, your hook routine may scan if the current instruction contains a reference to any stack pointer. And if, replace it with the content of the involved variable.

If you want to only focus on a small portion of code, this is also possible. But you'll have to setup the initial stack/registers state carefully and start at your custom entrypoint

Answer 2

Try ESET's deobfuscation tool. Basically it helps to remove some junk code, decrypt strings and restore CFF via a dynamic symbolic execution.