Dealing with VM-obfuscated code and started learning about symbolic analysis but having a hard time wrapping my head around how to practically apply it to a problem. If someone can give me some pointers, I'd greatly appreciate it.
- With dynamic analysis, am I supposed to only emulate the VM to get a symbolic binding? If yes, how can I deobfuscate only that part of a binary without touching the rest and would reconstructing a new binary even work if I only touched a small part (Jump locations might be off or don't point anywhere)? How do I do that? A lot of papers I read synthesize a new C program or use llvm-ir but that's not the entire program but only the VM loop. Now how do I merge it?
- If I emulate the entire binary to get a symbolic representation of it (like I've seen in some CTF write-ups), wouldn't that mean I only have the representation of the main path that this binary is going? (No inputs). My reconstructed binary would lack the interactions that are not in the main trace from start to finish, no?
- VM-obfuscated code usually has an encryption key that changes with each opcode that is interpreted. If I have a symbolic representation, I would still not be able to arbitrarily decompile the entire virtual opcode because I don't know the encryption key at that time.
On top of the confusion, the binary I am looking at right now doesn't just have one obfuscated VM loop but multiple which is apparently not uncommon when using commercial VM obfuscation. A short trace gave me over 4 structures that look like a VM loop. Debugging it showed me that the VM stack/context/registers used for pointers of each machine is different and random. So now,
- Do I deal with the VMs 1-by-1? As in, I reverse one until it calls the second (say VM1 calling VM2 sometime), then recompile a binary without it, then go to the next?
- Can I emulate the entire thing even though it means it will emulate potential dozens of VM-loops? I am guessing symbolic analysis won't really work if the registers used for internal stack and such are different over time
- If I have multiple VMs, how do I even know which bytecode is interpreted by which VM? I can take a guess from the sections and analyzing my trace, but again the trace I have is only for the main path and other branches of the app are not in it.
My understanding from it is something like this:
- Identify the VM loop and analyze where it stores it's context, stack pointer, opcode pointer, vmkey
- Create a symbolic representation of the loop and how it pulls the next opcodes
- Find all the handlers of the VM and create a symbol representation of them
- Somehow use this knowledge then to de-virtualize it
All in all, I am really confused. I read a handful of papers on it but security researchers don't really like releasing tools/code samples so it's all just theoretical and abstract.