I've recently come across a binary that has been obfuscated with a very obscure protection.
From what I understand, the obfuscator is effective in several ways.
- Multiple JMPs via relative addressing (0xE9), JMPs to addresses in stack (
e.g jmp [rsp-0x8]), and less frequently, JMPs using registers (
jmp reg). These are extremely effective at rendering the CFG in IDA Pro useless.
- Constant stack arithmetic. The obfuscator is continuously using stack memory to hold vital parts of information such as variables and JMPs to the next block (as stated previously). More importantly, the instruction
sub rsp, #and
add rsp, #and substituted via
lea rsp, [rsp+-#]. IDA's analyzer also fails to acknowledge a change in the stack pointer in the instruction
lea rsp, [rsp+8].
- Junk code. Instructions that don't exactly have any effect or pertain to the original function's goal.
- Slight use of opaque predicates (however I haven't spotted too many).
- Conditional branching via
test rax, rax ... ; junk mov rdx, sub_not_equal mov rcx, [rsp-0x8] cmovz rcx, rdx ... ; stack stuff -> jmp to position in stack containing rcx
.text:00007FF639A15EA9 loc_7FF639A15EA9: .text:00007FF639A15EA9 048 48 89 04 24 mov [rsp+48h+var_48], rax .text:00007FF639A15EAD 048 48 8D 64 24 08 lea rsp, [rsp+8] .text:00007FF639A15EB2 040 48 8B 74 24 F8 mov rsi, [rsp+40h+var_48] .text:00007FF639A15EB7 040 48 85 C0 test rax, rax .text:00007FF639A15EBA 040 E9 F8 10 44 FF jmp loc_7FF638E56FB7
.text:00007FF638E56FB7 loc_7FF638E56FB7: .text:00007FF638E56FB7 040 0F 85 F4 C2 B9 FB jnz near ptr sub_7FF6349F32B1 .text:00007FF638E56FBD 040 E9 33 EE 9F 02 jmp loc_7FF63B855DF5
.text:00007FF63B855DF5 loc_7FF63B855DF5: .text:00007FF63B855DF5 040 4C 89 74 24 F8 mov [rsp+40h+var_48], r14 .text:00007FF63B855DFA 040 48 8D 64 24 F8 lea rsp, [rsp-8] .text:00007FF63B855DFF 048 48 8B 34 24 mov rsi, [rsp+48h+var_48] .text:00007FF63B855E03 048 48 8D 64 24 08 lea rsp, [rsp+8] .text:00007FF63B855E08 040 E9 86 B1 74 FD jmp loc_7FF638FA0F93
.text:00007FF6392FCD6C loc_7FF6392FCD6C: .text:00007FF6392FCD6C 048 48 8D 2D 42 65 6F FB lea rbp, loc_7FF6349F32B5 .text:00007FF6392FCD73 048 48 87 2C 24 xchg rbp, [rsp+48h+var_48] .text:00007FF6392FCD77 048 48 8D 64 24 08 lea rsp, [rsp+8] .text:00007FF6392FCD7C 040 FF 64 24 F8 jmp [rsp+40h+var_48]
So on and so forth..
My goal was to create a tool that would take an obfuscated routine, follow along every jump, then create a peephole optimizer that would completely erase instruction sequences such as the
lea rsp, [rsp+-],
xhng reg, [rsp+x] and others shown and replace them with their "deobfuscated" variants. So I began working on a tool that would emulate certain instructions (especially anything involved with the stack) then create my own "virtual stack" that would hold the same information the original program would. This actually worked, except I didn't account for many things, such as JCCs. The end result shown in my tool was over 60 JMPs and over 200 instructions in the final combined dump (this did not even including the multiple JCC branches which can easily have the same or more CF obfuscation). I feel like my attempt was aimed in the wrong direction.
Well. Using my tool, I could probably just skim through the entire assembly and figure out what is going on after adding in conditional branch support, however I still feel unsatisfied. My original goal was to produce a completely optimized, decompilable dump, however I feel like I've hit a dead end due to my design flaws.
I want to ask to everyone who has had the patience to read through this post, on their advice carrying forward, whether it's a library recommendation, or how to re-approach the deobfuscation process I had going here.
For context, I'm analyzing a unpacked dump of a process and using my tool coded in C++ using Zydis disassembler to decode and manually emulate some of the instructions.