I'm just getting into the RE field, and I learned about virtualized packers (like VMProtect or Themida) in a class about a year ago. How often is malware in the wild really packed with virtualized packers, and what is the state of the art in unpacking them for static analysis?
virtualizers usage in the wild
They are rarely used, and even worse (or better), rarely used in a useful way.
how they're used
Typically, it was the use of a virtualizer of over only the main function, or another binary packer, and both cases don't prevent analysis: if you bypass the virtualized packer code, then you get the original unpacked code anyway.
why they're not used more often
- It makes the target bloated and slower
- they're not trivial to use correctly
- it's fairly common to detect them based on their (usually pirated) licence's watermark, so no matter what you'd virtualize, it would be detected by a specific fingerprint.
a meaningful example
AFAIK the only known smart use of a virtualizer (VMProtect here) in a malware is Trojan.Clampi, for which Nicolas Fallière wrote a white paper, but it's not so detailed. For this one, the whole viral body was virtualized.
papers on de-virtualization
- Rolf Rolles' Unpacking Virtualization Obfuscators, OpenRCE blog entries
- sherzo's inside code virtualizer
I couldn't find a public download link for these (otherwise good) papers:
- Boris Lau's Dealing with Virtualization packers
- Zhenxiang Jim Wan's How to recover virtualized x86 instructions by Themida
I can support the presented view of the other responders. You will rarely encounter code virtualization when looking at in the wild samples.
Just to add, here is a recent case-study by Tora looking at the custom virtualization used in FinFisher (sorry, direct link to PDF, have no other source).
The VM used here has only 11 opcodes, thus this example can be easily understood and used to get an impression of some common design principles behind custom VMs.
In the last year or so, I think I have only encountered a single malware sample that was using a virtualized packer (VMProtect in that case). Most of the samples I look at use stupid packers that allow easy dumping of the original PE from memory. I don't spend all of my time looking at malware, but I typically look at a couple potentially malicious samples a week, just to give some perspective on the volume I'm talking about. Also, I seem to remember that the SpyEye author used VMProtect to protect the malware builder, I'm not sure if other's who sell those kind of kits have used it as well. I've heard the Citadel builder has some pretty gnarly protection, but I'm not certain of what it is.
Regarding the second part of your question, I am no expert, but a couple of sites came to mind to potentially check out if you are interested in trying to learn to unpack this stuff. This site has some posts about unpacking various things, including VMProtect and Themida. I haven't gone through those posts in particular, just noticed that they are on there. I think there are some tutorials dealing with some versions of these specific packers on tuts4you.com as well if you are interested to check some of those out.
To the first part of your question: It really depends on the domain. Malware samples that leverage virtualization-based packers are generally easy to detect, which is a drawback from the malware author's perspective. If avoiding detection is crucial, which is especially the case in targeted attacks where your sample is probably custom anyway, virtualized packers are not a good idea. I assume that's why only very few samples make use of them. I have seen only one (themida protected) in the last year or so.