Is there any kind of software or research or paper which discusses replacement of frequent x86 instructions with ones which are less common and thus less understandable to the attacker (floating point/SSE/Virtualization/undocumented) while still maintaining the functionality?

For example, I wan to replace this

    PUSH 3
    CALL <check>
    JE <0xabcd>
    PUSH <text1>
    PUSH [EBP+5]
    CALL <MessageBox>
    PUSH <text2>
    PUSH [EBP+5]
    CALL <MessageBox>

with this

    etc etc

while still performing the same operation.


2 Answers 2


I do not think code morphism is the or an answer to this question.

What the question was about is obfuscating the algorithm implementation by using less common or undocumented assembly instructions. This can actually be done by some compilers when extensive optimizations are turned on. For example compilers like the Intel C Compiler, GCC, or PGI can autovectorize loops when matched to some internal patterns (reductions, matrix multiplications, ...) and when the target architecture supports vectorization. Other optimizations can lead to extremely tricky assembly code but still, it can always be reversed since the compiler performs no explicit obfuscation and because most of what the compiler does is pattern matching. Of course if you associate a high level pattern to a low level one, well, you lose the obfuscation and your code can easily be reversed. Thus techniques as the one you're looking for can only be performed by hand either by writing high level code using compiler intrinsics and alternative constructs or at the assembly level.

If you are really interested in obfuscation techniques I recommend you going over Jan CAPPAERT's PhD thesis : https://www.cosic.esat.kuleuven.be/publications/thesis-199.pdf, it covers some nice techniques used not only on malware but on industrial software too. The bibliography is quite rich. You can also check this talk given by Sean Taylor at Defcon on how to make the compiler do the obfuscation : https://www.defcon.org/images/defcon-17/dc-17-presentations/defcon-17-sean_taylor-binary_obfuscation.pdf.

About polymorphism, it is a nice obfuscation technique though it is rarely used in malware nowadays, and for many reasons. One of them is that few, if none, malware authors write code in assembly anymore, and most use frameworks and engines. You have to keep in mind that writing obfuscated assembly code is an art ... and that now it is used to harden the reverse engineering process for profit not for the challenge.

I've been working on a GCC plugin that adds an optimization pass which performs code obfuscation on the IR - internal representation (GIMPLE) - of a code before applying another obfuscation pass at the assembly level. The interesting thing about this approach is that you have the CFG (Control Flow Graph) of the program at compile time, and you can apply many obfuscation algorithms and techniques in order to break it into other equivalent CFGs and then assess which suites best and use it throughout the remaining compilation phases.

Hope my post helps.


These techniques of mutating code (and still keeping it semantically equivalent) are known as polymorphic code.

The software that can achieve a mutations of the code is usually called a polymorphic engine. It is a quite widely used technique in Malware design to evade pattern-matching detection of the anti-virus software.

With these key words in hand (and thanks to Google), you will be able to find tons of literature about the topic. But, here are a few pointers:

You also may find polymorphic engines ready to start. Just look for it.

  • 3
    FYI, polymorphism is extremely uncommon in malware these days. While many viruses back in the '90s had polymorphic engines, and several malware families in the aughts used server-side polymorphism, it is exceedingly rare to see polymorphic engines used by malware nowadays. Commented Feb 4, 2014 at 14:38
  • I have to admit that my experience with malware is from the 90's... :o) So, I trust your words about it.
    – perror
    Commented Feb 4, 2014 at 15:14
  • I notice more awesome tricks like metamorphing and time puzzles.
    – Stolas
    Commented Feb 4, 2014 at 15:30
  • 2
    @Jason, I think that it would be more correct to say that it is rare for malware to carry the engine these days. Polymorphism itself is still very much in use, particularly in the script space. Commented Feb 4, 2014 at 23:40
  • @peterferrie Are you grouping general obfuscation with polymorphism? Commented Feb 5, 2014 at 0:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.