Take the 2-minute tour ×
Reverse Engineering Stack Exchange is a question and answer site for researchers and developers who explore the principles of a system through analysis of its structure, function, and operation. It's 100% free, no registration required.

Is there any kind of software or research or paper which discusses replacement of frequent x86 instructions with ones which are less common and thus less understandable to the attacker (floating point/SSE/Virtualization/undocumented) while still maintaining the functionality?

For example, I wan to replace this

    PUSH EBP
    MOV EBP,ESP
    ...
    PUSH DWORD [0x0BEE]
    PUSH 3
    CALL <check>
    TEST EAX,EAX
    JE <0xabcd>
    PUSH <text1>
    PUSH [EBP+5]
    CALL <MessageBox>
0xabcd:
    PUSH <text2>
    PUSH [EBP+5]
    CALL <MessageBox>

with this

    AESKEYGENASSIST
    VFMSUBADDPD
    MOVLPS
    PMADDUBSW
    RET
    FLDL2T
    CMPXCHG8B
    AESKEYGENASSIST
    VFMSUBADDPD
    MOVLPS
    CMPXCHG8B
    STOSW
    VMLAUNCH
    etc etc

while still performing the same operation.

share|improve this question
add comment

1 Answer

up vote 13 down vote accepted

These techniques of mutating code (and still keeping it semantically equivalent) are known as polymorphic code.

The software that can achieve a mutations of the code is usually called a polymorphic engine. It is a quite widely used technique in Malware design to evade pattern-matching detection of the anti-virus software.

With these key words in hand (and thanks to Google), you will be able to find tons of literature about the topic. But, here are a few pointers:

You also may find polymorphic engines ready to start. Just look for it.

share|improve this answer
2  
FYI, polymorphism is extremely uncommon in malware these days. While many viruses back in the '90s had polymorphic engines, and several malware families in the aughts used server-side polymorphism, it is exceedingly rare to see polymorphic engines used by malware nowadays. –  Jason Geffner Feb 4 at 14:38
    
I have to admit that my experience with malware is from the 90's... :o) So, I trust your words about it. –  perror Feb 4 at 15:14
    
I notice more awesome tricks like metamorphing and time puzzles. –  Stolas Feb 4 at 15:30
2  
@Jason, I think that it would be more correct to say that it is rare for malware to carry the engine these days. Polymorphism itself is still very much in use, particularly in the script space. –  peter ferrie Feb 4 at 23:40
    
@peterferrie Are you grouping general obfuscation with polymorphism? –  Jason Geffner Feb 5 at 0:08
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.