1

I'm reverse-engineering a particular dylib on Mac OS X. The dylib is highly obfuscated, but I suspect it's from some well-known technique that I'm not aware of.

I'd like to describe it in the hope that someone here could identify it:

  1. There are junk instructions inserted everywhere, and every basic block is split by a lot of unconditional jump (but no opaque predicate is used).

  2. At its core, it looks like a obfuscated VM. It behaves like this:

At entry, it push a starting value on the stack, then call an entry point:

000000010013E070 68 5C 98 42 11                          push    1142985Ch
000000010013E075 E8 B5 B4 0E 00                          call    sub_10022952F

The entry (e.g. sub_10022952F) isn't a usual function. It will save all registers and point rsi to an embedded data location that is determined by the push (i.e. the push 1142985Ch above), then starts reading rsi array and jumping accordingly:

000000010013EED7 8B 06                                   mov     eax, [rsi]
000000010013EED9 F5                                      ;; cmc
000000010013EEDA 45 84 FD                                ;; test    r13b, r15b
000000010013EEDD 48 81 C6 04 00 00 00                    add     rsi, 4
000000010013EEE4 66 41 81 FA 1C 3B                       ;; cmp     r10w, 3B1Ch
000000010013EEEA 33 C3                                   xor     eax, ebx
000000010013EEEC D1 C0                                   rol     eax, 1
000000010013EEEE E9 FD 8E 15 00                          jmp     loc_100297DF0

0000000100297DF0 FF C0                                   inc     eax
0000000100297DF2 0F C8                                   bswap   eax
0000000100297DF4 F8                                      ;; clc
0000000100297DF5 E9 F2 D6 E4 FF                          jmp     loc_1000E54EC

00000001000E54EC C1 C0 03                                rol     eax, 3
00000001000E54EF 0F C8                                   bswap   eax
00000001000E54F1 53                                      push    rbx
00000001000E54F2 31 04 24                                xor     [rsp], eax
00000001000E54F5 0F B7 D8                                ;; movzx   ebx, ax
00000001000E54F8 0F BA F3 82                             ;; btr     ebx, 82h
00000001000E54FC 5B                                      pop     rbx
00000001000E54FD F5                                      ;; cmc
00000001000E54FE 49 F7 C7 E5 0F 9B 74                    ;; test    r15, 749B0FE5h
00000001000E5505 F9                                      ;; stc
00000001000E5506 48 63 C0                                movsxd  rax, eax
00000001000E5509 48 03 F8                                add     rdi, rax
00000001000E550C E9 D0 49 08 00                          jmp     loc_100169EE1

0000000100169EE1 FF E7                                   jmp     rdi

I've commented out junk code. The code will fetch value from [rsi] into rax, and do some bit-level-operation with rbx; then advance rdi <- rdi + rax; then jump to rdi.

This structure is everywhere in the dylib, and in a chain connected by jmp rdi, the bit-level-operations are also the same. But chains from different entry point may have different bit operation for rax and rbx.

The VM doesn't have any central structure (like a dispatcher or something), and the jumps go everywhere, not restricted to some location.

To call external function like pthread_mutex_lock, it will go out of the VM loop first; then call the external function; then go to another entry point. There are many entry points (push xxxx / call xxxx) to enter VM loop.

I believe this is some well-known technique, because when the dylib is modified, it will prompt message:

File corrupted! This program has been manipulated and maybe it's infected by a Virus or cracked. This file won't work anymore.

It's not a standard Mac OS X message, and Google search for this exact message gives a lot of result, but none of them explains the technique itself.

Some additional information:

  • It has sections named UPX0/UPX1, but I think they are disguise and the technique has nothing to do with UPX. Because the code only does self-checking, no self-modifying. As a result, it's still obfuscated just as the original file after fully loaded.

  • I don't know how the it works exactly by now. But I think it's some sort of a VM, because I've traced the invoke of a function of the dylib, I only find the following normal function calls:

pthread_mutex_lock --> 3 new operator --> pthread_mutex_unlock.

And everything else is done inside by those jump rdi structures mentioned above and switches among many entries points. Therefore the rest of the code logic resides in the VM loop.

  • @perror I'm aware of "control-flow flattening", it's the technique for jmp rdi. But the actual difficulty is how the VM codes (pointed by rsi) behaves, and junk code and unconditional jump isn't essential here. I believe this obfuscation is generated by some well-known specific technique as the "file corrupted" message suggests. So that I don't have to deobfuscate it again by myself. – user27283 Jan 29 at 16:54
2

I could be wrong, but it looks like VMProtect v3. For this version, this obfuscator inlines all handlers, so that's normal you don't find any dispatcher.

  • +1 -- looks like VMProtect to me also, though I'm not sure about the version. – Rolf Rolles Feb 2 at 21:52

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.