I'm reverse-engineering a particular dylib on Mac OS X. The dylib is highly obfuscated, but I suspect it's from some well-known technique that I'm not aware of.
I'd like to describe it in the hope that someone here could identify it:
There are junk instructions inserted everywhere, and every basic block is split by a lot of unconditional jump (but no opaque predicate is used).
At its core, it looks like a obfuscated VM. It behaves like this:
At entry, it push a starting value on the stack, then
call an entry point:
000000010013E070 68 5C 98 42 11 push 1142985Ch 000000010013E075 E8 B5 B4 0E 00 call sub_10022952F
The entry (e.g.
sub_10022952F) isn't a usual function. It will save all registers and point
rsi to an embedded data location that is determined by the
push (i.e. the
push 1142985Ch above), then starts reading
rsi array and jumping accordingly:
000000010013EED7 8B 06 mov eax, [rsi] 000000010013EED9 F5 ;; cmc 000000010013EEDA 45 84 FD ;; test r13b, r15b 000000010013EEDD 48 81 C6 04 00 00 00 add rsi, 4 000000010013EEE4 66 41 81 FA 1C 3B ;; cmp r10w, 3B1Ch 000000010013EEEA 33 C3 xor eax, ebx 000000010013EEEC D1 C0 rol eax, 1 000000010013EEEE E9 FD 8E 15 00 jmp loc_100297DF0 0000000100297DF0 FF C0 inc eax 0000000100297DF2 0F C8 bswap eax 0000000100297DF4 F8 ;; clc 0000000100297DF5 E9 F2 D6 E4 FF jmp loc_1000E54EC 00000001000E54EC C1 C0 03 rol eax, 3 00000001000E54EF 0F C8 bswap eax 00000001000E54F1 53 push rbx 00000001000E54F2 31 04 24 xor [rsp], eax 00000001000E54F5 0F B7 D8 ;; movzx ebx, ax 00000001000E54F8 0F BA F3 82 ;; btr ebx, 82h 00000001000E54FC 5B pop rbx 00000001000E54FD F5 ;; cmc 00000001000E54FE 49 F7 C7 E5 0F 9B 74 ;; test r15, 749B0FE5h 00000001000E5505 F9 ;; stc 00000001000E5506 48 63 C0 movsxd rax, eax 00000001000E5509 48 03 F8 add rdi, rax 00000001000E550C E9 D0 49 08 00 jmp loc_100169EE1 0000000100169EE1 FF E7 jmp rdi
I've commented out junk code. The code will fetch value from
rax, and do some bit-level-operation with
rbx; then advance
rdi <- rdi + rax; then jump to
This structure is everywhere in the dylib, and in a chain connected by
jmp rdi, the bit-level-operations are also the same. But chains from different entry point may have different bit operation for
The VM doesn't have any central structure (like a dispatcher or something), and the jumps go everywhere, not restricted to some location.
To call external function like
pthread_mutex_lock, it will go out of the VM loop first; then call the external function; then go to another entry point. There are many entry points (
push xxxx / call xxxx) to enter VM loop.
I believe this is some well-known technique, because when the dylib is modified, it will prompt message:
File corrupted! This program has been manipulated and maybe it's infected by a Virus or cracked. This file won't work anymore.
It's not a standard Mac OS X message, and Google search for this exact message gives a lot of result, but none of them explains the technique itself.
Some additional information:
It has sections named UPX0/UPX1, but I think they are disguise and the technique has nothing to do with UPX. Because the code only does self-checking, no self-modifying. As a result, it's still obfuscated just as the original file after fully loaded.
I don't know how the it works exactly by now. But I think it's some sort of a VM, because I've traced the invoke of a function of the dylib, I only find the following normal function calls:
pthread_mutex_lock --> 3
new operator -->
And everything else is done inside by those
jump rdi structures mentioned above and switches among many entries points. Therefore the rest of the code logic resides in the VM loop.