I'm trying to reverse-engineer a file format, and I got a buddy of mine (who works in the security field) to decompile the program via Hex-Rays and give me the resulting .C file. I've spent the past several days analyzing this file. However there is still a lot that I am trying to figure out, in particular two parts:
// v42 is a FILE handler, Dst is a 0x40 byte buffer if ( !ReadBuffer(&v42, &Dst, 0x40u, 0) || !(unsigned __int8)(*(int (__stdcall **)(char *))(*(_DWORD *)v29 + 20))(&Dst) )
I understand the first part, it reads 0x40 bytes into the buffer which is the file header. The second line is very confusing. I know C++ well, but I haven't had to deal with how much is going on with this. My guess is that this is fancy magic for saying "If the byte at 0x20 is not 1", but that doesn't make sense.
There's also a password associated with this file. The password itself appears to be a SHA256 hash, but I'm not sure exactly how it's used. Digging into this .C file, I see the function related to the password. It has many lines that look near identical:
v3 = a3; v4 = __ROL4__( (unsigned __int8)byte_441348[(*(_DWORD *)a1 + *(_DWORD *)a3) & 0xFF] | (((unsigned __int8)byte_441248[((unsigned int)(*(_DWORD *)a1 + *(_DWORD *)a3) >> 8) & 0xFF] | ((((unsigned __int8)byte_441048[(unsigned int)(*(_DWORD *)a1 + *(_DWORD *)a3) >> 24] << 8) | (unsigned __int8)byte_441148[((unsigned int)(*(_DWORD *)a1 + *(_DWORD *)a3) >> 16) & 0xFF]) << 8)) << 8), 11); v5 = *(_DWORD *)(a1 + 4) ^ v4;
Is there a way that I could figure out what algorithm is being used here without manually converting the code to something usable?
On top of all of this, I know that the file is compressed with zlib (both 1.2.5 and 1.2.1 are being used by two related programs). I believe that there is some transformation from this password on the file before the stream is being inflated by zlib.
441148. I'd look at the bytes at those addresses and google for them. For example, if the buffer at
C1 05 9E D8, googling for
C1059ED8will quickly reveal that this is the SHA-224 variant of SHA-2.
if (!ReadBuffer(...Dst...) || !class->method(Dst)). I have a hunch that the method decrypts the buffer - decompression seems lees likely since the length is fixed to 64 bytes. This is where @DOD's suggestion kicks in - dynamic analysis will help you much faster to (a) get the real function address and (b) verify if and how the function modifies the buffer. It might just verify a checksum and leave the buffer unchanged, as well.