By "VM obfuscated binary", I mean a binary that contains one or more implementations of a VM internally, and also some bytecode that can run on this VM. One of the reasons why this binary is this way, is to obfuscate some logic in the byte code. I see that a similar question has been asked for IDA Pro. The answer to that question indicated that IDA Pro does not support disassembly of such bytecode out of the box, but that there is a plugin that can help. Another question is more general, not specific to any tool like IDA Pro, Ghidra, etc.
NB: I'm not asking about the case where a binary is run in a virtual machine, say, Virtual Box, for example because it is suspected of being malware and running it in a virtual machine is a form of protection. Various questions have been asked on how to use IDA Pro, radare2, etc., to reverse engineer such a binary being run in a virtual machine. In the case of interest, we are talking about a binary that implements its own VM internally, and not about some external VM like Virtual Box.
So, it is not a problem disassembling the parts of the binary that implement the VM. The problem at hand is the bytecode. The bytecode architecture is not natively supported by Ghidra, so I'm working on using sleigh to specify it, etc.; that is another question.
The present question -- assuming I have a complete disassembler for the bytecode:
Is there a good way to programmatically figure out that the code may be implementing a VM?
Is there a good way to programmatically figure out where the bytecode begins and where it ends? Like, looking for patterns in the code that implements the VM?
- Should I be trying to implement it with Ghidra scripts? Or write a Ghidra analyser? Or something else?