Encrypted image format

Question

I am trying to reverse a simple image format for bitmap data with an indexed colour palette. The structure of the file is generally clear and I am able to extract the size of the picture and the palette. The place of where the run-length encoded data is stored is also pretty clear. However, the software for editing these images encrypts the run-length encoded data in a simple way by XORing it.

Through manipulation of the data and viewing it in the software, I am able to extract the pixel dependent key for the XOR function, i.e. if there are 9 pixels totally, there are 9 keys. The problem is, however, that everytime the file is saved in the software, the keys change. I have identified two bytes in the file that change together with the encrypted data everytime it is saved and I therefore expect that it must be some sort of seed for generating the keys.

Nevertheless, after saving the same file many times, examining the seed and the xor keys, no clear pattern emerges. It is only obvious that the first pixel has always the same key, the second key can only change by 1 and the rest looks currently random to me. Shifting the seed and combining it with the first key or using modulo operations don't work.

The software itself is packed/encrypted and I am unable to unpack it, i.e. I think I have to deduce the encryption solely based on the files I can generate. Is there some general advice on how to proceed in such a situation?

Answer 1

From what you describe, there might be somewhere in the SW an obscure algorithm calculating the seed. If there is no pattern which can be found, you must find this algo in the software. Here are some hints how you could proceed:

• Start it from within a debugger like Ida and set a random breakpoint during the run. Try to analyze the structure of the code loaded in memory.

• If this does not work, start it and try to attach a debugger during the run and proceed like above.

• If this not work either, try to investigate the unpacker/decrypter. There must be an unencrypted stub where everything starts. Do not analyze line by line, but in blocks and try to identify the point where the whole thing is unencrypted loaded in memory. If this is hampered by anti-debug measures, these should be removed. In this stage much could be done statically. You could try a decompiler like Ghidra to make faster progress than with pure static disassembler.

• If you have more advanced software which decrypts in chunks and after having done its work encrypts again you will have a rather hard time to procedd. It is possible though.

• For your strategy of "guessing" the seeds, no general rule can be given here. If you are able to define a roadmap how to proceed this might work. If there is some kind of weird app specific algo I would much prefer the other - tedious but rather straightforward - possibilities.