I'm trying to reverse-engineer a file format, and I got a buddy of mine (who works in the security field) to decompile the program via Hex-Rays and give me the resulting .C file. I've spent the past several days analyzing this file. However there is still a lot that I am trying to figure out, in particular two parts:
// v42 is a FILE handler, Dst is a 0x40 byte buffer if ( !ReadBuffer(&v42, &Dst, 0x40u, 0) || !(unsigned __int8)(*(int (__stdcall **)(char *))(*(_DWORD *)v29 + 20))(&Dst) )
I understand the first part, it reads 0x40 bytes into the buffer which is the file header. The second line is very confusing. I know C++ well, but I haven't had to deal with how much is going on with this. My guess is that this is fancy magic for saying "If the byte at 0x20 is not 1", but that doesn't make sense.
There's also a password associated with this file. The password itself appears to be a SHA256 hash, but I'm not sure exactly how it's used. Digging into this .C file, I see the function related to the password. It has many lines that look near identical:
v3 = a3; v4 = __ROL4__( (unsigned __int8)byte_441348[(*(_DWORD *)a1 + *(_DWORD *)a3) & 0xFF] | (((unsigned __int8)byte_441248[((unsigned int)(*(_DWORD *)a1 + *(_DWORD *)a3) >> 8) & 0xFF] | ((((unsigned __int8)byte_441048[(unsigned int)(*(_DWORD *)a1 + *(_DWORD *)a3) >> 24] << 8) | (unsigned __int8)byte_441148[((unsigned int)(*(_DWORD *)a1 + *(_DWORD *)a3) >> 16) & 0xFF]) << 8)) << 8), 11); v5 = *(_DWORD *)(a1 + 4) ^ v4;
Is there a way that I could figure out what algorithm is being used here without manually converting the code to something usable?
On top of all of this, I know that the file is compressed with zlib (both 1.2.5 and 1.2.1 are being used by two related programs). I believe that there is some transformation from this password on the file before the stream is being inflated by zlib.