Is there any kind of software or research or paper which discusses replacement of frequent x86 instructions with ones which are less common and thus less understandable to the attacker (floating point/SSE/Virtualization/undocumented) while still maintaining the functionality?
For example, I wan to replace this
PUSH EBP MOV EBP,ESP ... PUSH DWORD [0x0BEE] PUSH 3 CALL <check> TEST EAX,EAX JE <0xabcd> PUSH <text1> PUSH [EBP+5] CALL <MessageBox> 0xabcd: PUSH <text2> PUSH [EBP+5] CALL <MessageBox>
AESKEYGENASSIST VFMSUBADDPD MOVLPS PMADDUBSW RET FLDL2T CMPXCHG8B AESKEYGENASSIST VFMSUBADDPD MOVLPS CMPXCHG8B STOSW VMLAUNCH etc etc
while still performing the same operation.