Virtual machine code obfuscation implementation details

Question

I want to implement a VM based simple proof-of-concept obfuscator. It should take an exe file as input and produce a new pe file with appended vm section. For simplicity let's say the exe file is compiled as a 32 bit pe.

The problem is that most materials I've found online explain only how to crack and not how to implement such a solution, or just explain how to implement a simple VM with a very limited number of instructions.

Architecture

I want to build such an architecture as described in this (Enhance virtual-machine-based code obfuscation security through dynamic bytecode scheduling) paper:

Let me quote it:

Fig. 1. A classical process for VM-based code obfuscation.
To obfuscate the code, we first dissemble the code region to be
protected into native assembly code (1).
The assembly code will be mapped into our virtual instructions (2)
which will then be encoded into a bytecode format (3). Finally,
the generated byecode will be inserted into a specific region of
the binary which is linked with a VM library (4).

Question

Let's say I have implemented a very basic virtual machine with a set of 20 instructions as presented in this [presentation]:

iadd, 
isub, 
imul, 
ilt, 
ieq, 
br addr, 
brt addr, 
brf addr, 
iconst value, 
load addr, 
gload addr, 
store addr, 
gstore addr, 
print, 
pop, 
call addr, numArgs
ret
hlt

In step 2 (Virtualization), I have to somehow map the extracted Intel instruction set to my virtualized instructions. The Intel instruction set is huge (over 200 instructions). And I have completely no idea how to do this. Starting with the fact that my virtual machine uses fewer of registers then Intel does.

Answer 1

You will need to go through each of the over 200 instructions, and figure out how to do the "same" thing in your instruction set. So you'll need to see how the stack may be changing, the stack pointer, the program counter, and also if any memory or register changes. For differences like not having enough registers, you could set aside a portion of memory to use to store the values that would otherwise be going into the Intel registers. Look out also for endianness of the data, and whether it is 32-bit, 64-bit, etc.

And that's if you can assume that each instruction is independent of the others. The instruction set you're starting from, may have quirks that break this assumption, and you'll need to take care of those. For example, it may have branch delay slots, like in MIPS (Understanding branch delay slots for reversing MIPS), where a condition is evaluated for the branching, and even if the jump/branch happens, the next instruction is evaluated.

As the mapping may be a challenging task, some people take an alternative approach and start from source code in a high level language (C for instance), and directly compile that to the bytecode of the virtual machine. For example, take a look at the virtualize transform of the Tigress obfuscator. The Tigress obfuscation is source-to-source (C to C), so when virtualize is selected, the resulting C code actually implements the desired virtual machine, and runs the bytecode (compiled for the VM by Tigress) on that virtual machine.

But in your case, if you have to start with an exe, then you'll need to work on the mapping.

Answer 2

If you want proof-of-concept VM based obfuscator, you can actually skip the virtualization step and bytecode generation step. You can write an interpreter of x86 instructions in x86 assembler so that the implementation is easy. You can add custom instructions one by one later through assembler macros, for example. This is much less complicated process than writing a full-blown VM.

The most trivial "VM" that runs some x86 instructions can be found here:

https://github.com/Barebit/trivial-vm

The vm1.c merely copies an x86 instruction to a buffer and runs it. It is not even an interpreter. The vm2 does the same but interprets few branch instructions that can't be run directly.