Computer Malware: More complex than metamorphic?

Question

I have read the topic on oligo-, poly- and metamorphic malware. I have done a quick google search, and it seems like the concept of metamorphic malware was already known in the DOS age and in the 1990s at 32-bit Windows (according to Hunting For Metamorphic by Ször and Ferrie).

Are there more complex methods of self-modification in computer malware than metamorphism in modern times?

Edit

The answer by Jason Geffner is very interesting, dealing with problematic reverse-engineering techniques. As soon as you have the plain code, can you create a detection mechanism? I understand that for polymorphism (where there is an underlying unprotected layer) or for VMs. But what about codes that can change this underlying layer, such as metamorphic codes do (see the reference above)?

There, reversing gives you the underlying layer, which is not constant - right? I also wonder whether there are advanced techniques of metamorphism developed since the last 10 year or so, or did this type of self-modification become extinct?

Edit2 As I cannot comment due to too less friend-points, Jason: I intented to ask this as my original question. The info you gave is interesting, unfortunatly it doensn't answer fully my oringial question (which I have stated not well enough as it seems). With my edit above, I wanted to clarify what I wanted to know. I dont think a new question is required as this is the originally intented question, do you agree?

Answer 1

Metamorphism wasn't used for obfuscation as much as it was used to defeat static AV signatures.

In terms of modern* obfuscation:

• A custom virtual machine can be a pain to analyze, especially if it's a custom crafted one (as opposed to something like VMProtect which is more widely studied).
• Obfuscation that involves code running in the kernel can also be challenging for many reversers, since many common reverse engineering tools are designed for user-mode targets.
• Chunked packing, where not all code is deobfuscated at the same time, can be difficult to analyze.

There are plenty of other methods of complex obfuscation that I'm sure others will name on this thread :)

And although you didn't ask about it, a modern* polymorphic technique is server-side polymorphism, where the polymorphic engine is on a remote server and it is used to churn out millions of variations of a malware sample.

* Note that "modern" in this context means the past 10 years or so.