How someone would go to reverse engineer a sample protected with a virtual machine? The problem is that it's not opcode anymore it's bytecode which I don't have any idea about because it's a private virtual machine so how you would go about solving this problem taking into account that ida pro and other tools doesn't work, some of you might say find the OEP will that's possible if it is a simple packer that unpack it self completely at the first hit but it's not the case in here.

I'm looking for ideas on how to approach this problem.


1 Answer 1


When a sample is protected with a VM, the only way to reverse engineer the code itself is to start with understanding the VM.

From what I've found, a lot of VMs have two things in common:

  1. They are not complex (sometimes the bytecode instructions themselves are just decoded into x86 instructions - usually some VM instructions as well) Complexity in a VM very quickly degrades performance.

  2. They are usually quite obfuscated

Aside from some generic solutions you could try (I.e. VMHunt) you will generally need to reverse engineer the VM and understand how it is decoding bytecode instructions - and what each VM instruction does. Then, you can use that knowledge to transform bytecode back into its original form - at which point you can analyze it in IDA Pro etc... Usually, the most difficult part is understanding the VM despite obfuscation.

It can be quite a lengthy / involved process, I have worked with code virtualizers before and encourage you to start by reading some of my work on x86virt:

Devirtualizer Source Code: https://github.com/JeremyWildsmith/x86devirt

Article Explaining Process: http://jeremywildsmith.com/?blog%2Fx86devirt

  • okey I will take a look at it
    – zerocool
    Jun 1, 2019 at 9:37
  • Hi @Jeremy , Thanks for the answer. Is your blog URL no longer working? I couldn't see the article explaining the process. May 2, 2020 at 14:52
  • Hello @auspicious99, The domain is currently down, for now, you can use the IP:
    – Jeremy
    May 2, 2020 at 18:10
  • Thanks @Jeremy , that works. Looks impressive. Will need to dig through it soon. By the way, are you familiar with Ghidra? We were trying to reverse engineer VM obfuscated code with Ghidra. I've asked a question at reverseengineering.stackexchange.com/questions/24896/… May 3, 2020 at 2:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.