1

I'm trying to reverse-engineer a file format, and I got a buddy of mine (who works in the security field) to decompile the program via Hex-Rays and give me the resulting .C file. I've spent the past several days analyzing this file. However there is still a lot that I am trying to figure out, in particular two parts:

// v42 is a FILE handler, Dst is a 0x40 byte buffer
if ( !ReadBuffer(&v42, &Dst, 0x40u, 0) ||
 !(unsigned __int8)(*(int (__stdcall **)(char *))(*(_DWORD *)v29 + 20))(&Dst) )

I understand the first part, it reads 0x40 bytes into the buffer which is the file header. The second line is very confusing. I know C++ well, but I haven't had to deal with how much is going on with this. My guess is that this is fancy magic for saying "If the byte at 0x20 is not 1", but that doesn't make sense.

There's also a password associated with this file. The password itself appears to be a SHA256 hash, but I'm not sure exactly how it's used. Digging into this .C file, I see the function related to the password. It has many lines that look near identical:

    v3 = a3;
      v4 = __ROL4__(
         (unsigned __int8)byte_441348[(*(_DWORD *)a1 + *(_DWORD *)a3) &         0xFF] | (((unsigned __int8)byte_441248[((unsigned int)(*(_DWORD *)a1 + *(_DWORD *)a3) >> 8) & 0xFF] | ((((unsigned __int8)byte_441048[(unsigned int)(*(_DWORD *)a1 + *(_DWORD *)a3) >> 24] << 8) | (unsigned __int8)byte_441148[((unsigned int)(*(_DWORD *)a1 + *(_DWORD *)a3) >> 16) & 0xFF]) << 8)) << 8),
     11);
      v5 = *(_DWORD *)(a1 + 4) ^ v4;

Is there a way that I could figure out what algorithm is being used here without manually converting the code to something usable?

On top of all of this, I know that the file is compressed with zlib (both 1.2.5 and 1.2.1 are being used by two related programs). I believe that there is some transformation from this password on the file before the stream is being inflated by zlib.

Any ideas?

Improve this question
7
  • Keep in mind decompiling is always somewhat iffy. You may want to verify if this is actually what the original assembly does.
    – Jongware
    Jun 16, 2015 at 17:31
  • Why you trying to reverse-engineer a file format in static? In my opinion for this case better use dynamic approach (ollydbg for 32 and x64dbg - for 64)
    – DOD
    Jun 17, 2015 at 8:35
  • Your many mostly identical lines seem to do table lookups at the addresses 441348, 441248, 441048, 441148. I'd look at the bytes at those addresses and google for them. For example, if the buffer at 441348 starts with C1 05 9E D8, googling for C1059ED8 will quickly reveal that this is the SHA-224 variant of SHA-2.
    – Guntram Blohm
    Jun 17, 2015 at 10:35
  • Your first code example seems to call a class method through the vtable pointer, as Jason said. This would make the code something like if (!ReadBuffer(...Dst...) || !class->method(Dst)). I have a hunch that the method decrypts the buffer - decompression seems lees likely since the length is fixed to 64 bytes. This is where @DOD's suggestion kicks in - dynamic analysis will help you much faster to (a) get the real function address and (b) verify if and how the function modifies the buffer. It might just verify a checksum and leave the buffer unchanged, as well.
    – Guntram Blohm
    Jun 17, 2015 at 10:42
  • @DOD there seems to be some protection against ollydbg. Running it with the exact same parameters in the same working directory, it spits out a security error.
    – Thebluefish
    Jun 17, 2015 at 13:19

1 Answer 1

Reset to default
4

For your first question, (*(int (__stdcall **)(char *))(*(_DWORD *)v29 + 20)) is a function call (v29 is an object (perhaps a vtable) that contains a function pointer).

Please create separate posts for each of your other questions (one question per post).

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.