I have read the topic on oligo-, poly- and metamorphic malware. I have done a quick google search, and it seems like the concept of metamorphic malware was already known in the DOS age and in the 1990s at 32-bit Windows (according to Hunting For Metamorphic by Ször and Ferrie).
Are there more complex methods of self-modification in computer malware than metamorphism in modern times?
Edit
The answer by Jason Geffner is very interesting, dealing with problematic reverse-engineering techniques. As soon as you have the plain code, can you create a detection mechanism? I understand that for polymorphism (where there is an underlying unprotected layer) or for VMs. But what about codes that can change this underlying layer, such as metamorphic codes do (see the reference above)?
There, reversing gives you the underlying layer, which is not constant - right? I also wonder whether there are advanced techniques of metamorphism developed since the last 10 year or so, or did this type of self-modification become extinct?
Edit2 As I cannot comment due to too less friend-points, Jason: I intented to ask this as my original question. The info you gave is interesting, unfortunatly it doensn't answer fully my oringial question (which I have stated not well enough as it seems). With my edit above, I wanted to clarify what I wanted to know. I dont think a new question is required as this is the originally intented question, do you agree?