I want to implement a VM based simple proof-of-concept obfuscator. It should take an exe file as input and produce a new pe file with appended vm section. For simplicity let's say the exe file is compiled as a 32 bit pe.
The problem is that most materials I've found online explain only how to crack and not how to implement such a solution, or just explain how to implement a simple VM with a very limited number of instructions.
I want to build such an architecture as described in this (Enhance virtual-machine-based code obfuscation security through dynamic bytecode scheduling) paper:
Let me quote it:
Fig. 1. A classical process for VM-based code obfuscation.
To obfuscate the code, we first dissemble the code region to be
protected into native assembly code (1).
The assembly code will be mapped into our virtual instructions (2)
which will then be encoded into a bytecode format (3). Finally,
the generated byecode will be inserted into a specific region of
the binary which is linked with a VM library (4).
Let's say I have implemented a very basic virtual machine with a set of 20 instructions as presented in this [presentation]:
call addr, numArgs
In step 2 (Virtualization), I have to somehow map the extracted Intel instruction set to my virtualized instructions. The Intel instruction set is huge (over 200 instructions). And I have completely no idea how to do this. Starting with the fact that my virtual machine uses fewer of registers then Intel does.