I've seen quite a few crackmes on reverse engineering VM's. I've also come across "Unpacking Virtualization Obfuscators". Are there any other writeups that you could point me to that discuss the RE'ing of a VM based obfuscation from scratch?
Also, is the source code for any VM based obfuscator available online?
Here is a pretty old challenge with, among other stuff, a VM in the end. You have both the write-ups of many candidates, and the official write-up of the author, with explanations as how to do such VM-based obfuscation.
There's a discussion of reverse engineering virtualization-based obfuscation here. A forerunner of this work, which in retrospect I'm not all that happy with, was published in ACM CCS 11 [link].
I don't know of source-code availability of any VM-based obfuscators. There's a free system called Tigress, available (in binary form) from http://tigress.cs.arizona.edu/, that performs virtualization at the C source-code level. This system gives you a lot of control over a lot of the parameters of the obfuscator so you can experiment with different combinations of settings. The FAQ explains the rationale for not distributing the source code.
There is a good article wrote by Rolf Rolles about it.
