Reverse Engineering Crypto Methods

Question

Since few days, Ive started to Reverse Engineer a little game. That game includes some data files, where the 3d files are stored. I want to write a little Software, where I can add own 3d models, to improve (cutomize) that game.

Luckily the game is compiled in debug mode, so I can clearly see all method names, c++ classes, etc., so I clearly know what happens in particular places in the code. Since I am new on reverse engineering (but not new in development), Ive figured out a lot of the code base and reimplemented many things related to the "Data management of the game".

I see, in the disassembled code, there is a class which decrypts a file. Two methods are used. Initializing the decryptor and reading a character. Since I am not good at "reverse engineering" and crypto algorithms (I don´t even know whether the used crypto is a know algorithm or an own one), I hoped you guys could help me out to figure out what happens exactly on those methods, so I can re implement that in c++.

I am going to include the asm sections and the pseudo c code (which my decompiler has produced). Notice that these methods are part of a class. So I will give you some additional information of what Ive figured out so far. I hope you guys could help me.

Decryptor init method: (edi + 0x100 + offset) seems to be a char array of size 128 byte, but I couldn´t figure out the rest (It seems to be worked with char array, but like mentioned, I dont have that much experience with RE)

ASM Code: http://pastebin.com/LiM10Cr5

Pseudo-C: http://pastebin.com/1ewfxAh1

Decryptor decrypt next char method:

(edi + 0x200) seems to be a char (byte) where the next char or something similar is stored.

ASM Code: https://pastebin.com/aWp8LYua

Pseudo-C: https://pastebin.com/u4w5yGYS

If you need some more information, like mentioned, the application is fully compiled in debug mode, so its like a dream... I have all labels, all vtables, all information...

I also have a read method, where these both methods are gonna called, but I guess I have reverse engineered that method properly (I hope so)...

Thanks :)

Answer 1

The crypto definitely isn't a "good" one. Check what happens in your nextDecrypt function.

The nextDecrypt function

There's a variable at ebx+0x200 which gets loaded into edx. This variable gets incremented by one, then written back to ebx+0x200, and this variable is also used to be xored with some byte (low byte of ecx, i.e. cl) before that byte is returned. So we have an algorithm that, very trivially, xors every byte, with the value that's xored being incremented every time.

There's a bit of special casing; the xor value avoids the value 0; if the value is 0xff, it's set to 1 instead of being incremented. Also, the value that's returned gets negated, and your decompiler made an error here; the ! operator should have been a ~ as that's what the not assembly instruction does. (Another reason why you should use decompiled C as a first glance, but always look at assembly code to understand what's really going on).

Initialization

The initialization function seems to be a bit more complicated, until you realize that there are 8 identical blocks in the do..while loop; probably an inner loop that the compiler unrolled. It seems to shuffle around the bytes a bit at [edx+0x100], and create an index map at [ecx]. The last thing it does seems to initialize the xor value from the nextDecrypt to 0x7f. But something seems to be wrong in the first part; your code accesses arg_4 but there is just an arg_0. And this arg_4 seems to influence the step with of the byte shuffler.

This is about as far as it makes sense to statically analyze the code; what a reverser should do at this point is run the thing in a debugger, and single-step through those 2 functions, checking how the data buffers change, to verify those assumptions. For example, i'd verify the "initializing the xor value to 0x7f" assumption can't be checked from your code alone, but in a debugger, you could check if the addresses are indeed the same. Also, it would be interesting to know what some real-world values of these arguments are, and where they come from.

Summary

So, to sum it up: This seems to be a "crypto" mechanism the author invented, not a standard one; it's not a complicated one; and you'll need some dynamic analysis of the software to get the details right of what it does.

Additional note

In cases like this, it makes sense to give as much information as possible, for example, the name of the game you're hacking. For example, loading up the binary in IDA makes analysis much easier than plaintext sourcecode, and people might actually be able to run the code and test a few assumptions on it.