custom virtual machine protection

Question

How someone would go to reverse engineer a sample protected with a virtual machine? The problem is that it's not opcode anymore it's bytecode which I don't have any idea about because it's a private virtual machine so how you would go about solving this problem taking into account that ida pro and other tools doesn't work, some of you might say find the OEP will that's possible if it is a simple packer that unpack it self completely at the first hit but it's not the case in here.

I'm looking for ideas on how to approach this problem.

Answer 1

When a sample is protected with a VM, the only way to reverse engineer the code itself is to start with understanding the VM.

From what I've found, a lot of VMs have two things in common:

1. They are not complex (sometimes the bytecode instructions themselves are just decoded into x86 instructions - usually some VM instructions as well) Complexity in a VM very quickly degrades performance.

2. They are usually quite obfuscated

Aside from some generic solutions you could try (I.e. VMHunt) you will generally need to reverse engineer the VM and understand how it is decoding bytecode instructions - and what each VM instruction does. Then, you can use that knowledge to transform bytecode back into its original form - at which point you can analyze it in IDA Pro etc... Usually, the most difficult part is understanding the VM despite obfuscation.

It can be quite a lengthy / involved process, I have worked with code virtualizers before and encourage you to start by reading some of my work on x86virt:

Devirtualizer Source Code: https://github.com/JeremyWildsmith/x86devirt

Article Explaining Process: http://jeremywildsmith.com/?blog%2Fx86devirt