Take the 2-minute tour ×
Reverse Engineering Stack Exchange is a question and answer site for researchers and developers who explore the principles of a system through analysis of its structure, function, and operation. It's 100% free, no registration required.

Is there any kind of software or research or paper which discusses replacement of frequent x86 instructions with ones which are less common and thus less understandable to the attacker (floating point/SSE/Virtualization/undocumented) while still maintaining the functionality?

For example, I wan to replace this

    PUSH EBP
    MOV EBP,ESP
    ...
    PUSH DWORD [0x0BEE]
    PUSH 3
    CALL <check>
    TEST EAX,EAX
    JE <0xabcd>
    PUSH <text1>
    PUSH [EBP+5]
    CALL <MessageBox>
0xabcd:
    PUSH <text2>
    PUSH [EBP+5]
    CALL <MessageBox>

with this

    AESKEYGENASSIST
    VFMSUBADDPD
    MOVLPS
    PMADDUBSW
    RET
    FLDL2T
    CMPXCHG8B
    AESKEYGENASSIST
    VFMSUBADDPD
    MOVLPS
    CMPXCHG8B
    STOSW
    VMLAUNCH
    etc etc

while still performing the same operation.

share|improve this question
add comment

2 Answers 2

up vote 6 down vote accepted

I do not think code morphism is the or an answer to this question.

What the question was about is obfuscating the algorithm implementation by using less common or undocumented assembly instructions. This can actually be done by some compilers when extensive optimizations are turned on. For example compilers like the Intel C Compiler, GCC, or PGI can autovectorize loops when matched to some internal patterns (reductions, matrix multiplications, ...) and when the target architecture supports vectorization. Other optimizations can lead to extremely tricky assembly code but still, it can always be reversed since the compiler performs no explicit obfuscation and because most of what the compiler does is pattern matching. Of course if you associate a high level pattern to a low level one, well, you lose the obfuscation and your code can easily be reversed. Thus techniques as the one you're looking for can only be performed by hand either by writing high level code using compiler intrinsics and alternative constructs or at the assembly level.

If you are really interested in obfuscation techniques I recommend you going over Jan CAPPAERT's PhD thesis : https://www.cosic.esat.kuleuven.be/publications/thesis-199.pdf, it covers some nice techniques used not only on malware but on industrial software too. The bibliography is quite rich. You can also check this talk given by Sean Taylor at Defcon on how to make the compiler do the obfuscation : https://www.defcon.org/images/defcon-17/dc-17-presentations/defcon-17-sean_taylor-binary_obfuscation.pdf.

About polymorphism, it is a nice obfuscation technique though it is rarely used in malware nowadays, and for many reasons. One of them is that few, if none, malware authors write code in assembly anymore, and most use frameworks and engines. You have to keep in mind that writing obfuscated assembly code is an art ... and that now it is used to harden the reverse engineering process for profit not for the challenge.

I've been working on a GCC plugin that adds an optimization pass which performs code obfuscation on the IR - internal representation (GIMPLE) - of a code before applying another obfuscation pass at the assembly level. The interesting thing about this approach is that you have the CFG (Control Flow Graph) of the program at compile time, and you can apply many obfuscation algorithms and techniques in order to break it into other equivalent CFGs and then assess which suites best and use it throughout the remaining compilation phases.

Hope my post helps.

share|improve this answer
add comment

These techniques of mutating code (and still keeping it semantically equivalent) are known as polymorphic code.

The software that can achieve a mutations of the code is usually called a polymorphic engine. It is a quite widely used technique in Malware design to evade pattern-matching detection of the anti-virus software.

With these key words in hand (and thanks to Google), you will be able to find tons of literature about the topic. But, here are a few pointers:

You also may find polymorphic engines ready to start. Just look for it.

share|improve this answer
2  
FYI, polymorphism is extremely uncommon in malware these days. While many viruses back in the '90s had polymorphic engines, and several malware families in the aughts used server-side polymorphism, it is exceedingly rare to see polymorphic engines used by malware nowadays. –  Jason Geffner Feb 4 at 14:38
    
I have to admit that my experience with malware is from the 90's... :o) So, I trust your words about it. –  perror Feb 4 at 15:14
    
I notice more awesome tricks like metamorphing and time puzzles. –  Stolas Feb 4 at 15:30
2  
@Jason, I think that it would be more correct to say that it is rare for malware to carry the engine these days. Polymorphism itself is still very much in use, particularly in the script space. –  peter ferrie Feb 4 at 23:40
    
@peterferrie Are you grouping general obfuscation with polymorphism? –  Jason Geffner Feb 5 at 0:08
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.