I am trying to reverse a simple image format for bitmap data with an indexed colour palette. The structure of the file is generally clear and I am able to extract the size of the picture and the palette. The place of where the run-length encoded data is stored is also pretty clear. However, the software for editing these images encrypts the run-length encoded data in a simple way by XORing it.
Through manipulation of the data and viewing it in the software, I am able to extract the pixel dependent key for the XOR function, i.e. if there are 9 pixels totally, there are 9 keys. The problem is, however, that everytime the file is saved in the software, the keys change. I have identified two bytes in the file that change together with the encrypted data everytime it is saved and I therefore expect that it must be some sort of seed for generating the keys.
Nevertheless, after saving the same file many times, examining the seed and the xor keys, no clear pattern emerges. It is only obvious that the first pixel has always the same key, the second key can only change by 1 and the rest looks currently random to me. Shifting the seed and combining it with the first key or using modulo operations don't work.
The software itself is packed/encrypted and I am unable to unpack it, i.e. I think I have to deduce the encryption solely based on the files I can generate. Is there some general advice on how to proceed in such a situation?