Techniques for reverse engineering an unknown or protected file format

Question

I'm working on converting a game originally developed to run on Windows for the web. I have a bunch of artwork for the game received from the artist in .tga image files which open fine in Photoshop. However I don't have the complete set of artwork and it cannot be found.

I also have a distribution of the Windows game which consists of .dlls, .lua files and a lot of .tga files. The .tga (Targa) files in the game distribution are named the same as the .tga files in the artwork I have received so I think it's safe to assume they are the same images. Even the images that I'm missing from the supplied artwork are present in the game distribution. Bingo! Well actually no…

The .tga files from the game won't open in Photoshop or Preview (mac) or Pixelmator (mac). So I think something has been done to the game .tga files to protect them and I want to reverse engineer them back to normal .tga files.

First I need to discover what the new files are and how they've been modified. Here's what I've tried so far:

---

Using linux file command:

• rockExplosion.tga (from artwork).
  • Output: rockExplosion.tga: Targa image data - RGB 100 x 43
• rockExplosion.tga (from game).
  • Output: rockExplosion.tga: data

---

Inspecting using hex viewer and diff

The two files are exactly the same byte length. The last ~15% of bytes are identical in each file and the first ~85% are completely different.

---

Binwalk didn't tell me anything for either file.

---

• What other techniques exist for reverse engineering unknown file formats?
• Is there a standard way of protecting images in .NET applications that could have been used?
• What other tools could I use to probe further?

Answer 1

If you can, you should post links to 2-3 of your real targa images, and the corresponding game targa images (beware of license issues though). Sometimes, looking at the files rings a bell, and i could have checked much of what i'm going to say if it works for your files or not.

That said:

• Are some of the tga files referenced from Lua? There is Lua source code out there to load targa files, maybe your lua has a variation on that that does the decryption.
• Because the files have the same length, i'd assume they're not converted to a different format, but encrypted somehow instead.
• It's unlikely to be a kind of encyption that's used elsewhere and that's well known, since that would defeat the point of encryption.
• If this is a simple way of encryption, then the first few bytes (the header of the tga file) should be the same for all of the encrypted files, since they're the same in the original files. Check the targa file format for details; for example, image width/height should be different, most other header fields should be the same.
• It could be interesting to know where exactly the different bytes stop and the identical bytes start. Is that always on a block boundary, i.e. a multiple of 512 bytes, with only the last block (that's smaller than 512 bytes) unencrypted? (Replace 512 with 1024, 2048, 4096 for likely block sizes.) Or is it always on a certain logical position in the file, for example start of the extension area, or start of the developer area?
• Do all of the files have a developer area? If so, it might contain an encryption key. (Unlikely since the file size doesn't change)
• What happens if you xor the bytes from the original file with the bytes from the encrypted file? Do different pairs of files yield the same xored pattern? Does this pattern repeat, and after how many bytes?

If all of this fails, you'll have to reverse engineer the game code, learn how the executable code loads the tga files, and implement the algorithm yourself.

• Are you sure this is a .net application? If so, reversing it might be quite trivial; "Is there any way to decompile a .NET assembly or program?" should give you some hints.
• If it's a compiled application (C, C++, Delphi), it's harder, since you probably need to dig through some assembly, but my answer to "Reversing encryption by analysing executable" has some pointers.

Answer 2

Don't be fooled by the .tga extention. Some developers add an extention to their files to throw people like us off.

Your files could easily be either, a different image type, an archive (zlib as an example) or an encrypted data. (In your case they just might be actual tga files.)

One method you could use to get the images that you are looking for is to go around the encryption. If theses files are indeed enceypted, then you could use its original engine to open the files as usual (boot the game). Then you could open the applications memory using a memory editor (such as CE). If you can locate the game data in memory, (at which point it would be decrypted) then you could ripp it out and rebuild it as the correct file type. You may have to do research on the tga file structure first.

Moreover, you could try poking around in the dll files (look for plain text in the header), and try to research their origin or functions, for all you know, those dlls could actually be compression or encryption libraries.

Lastly, I recomend a debugger like ProcessExplorer, pay attention to the offsets when the .tgas are loaded. That way you can determine it's structure.

You might wanna explore ollydbg, that way you could see the steps the application goes through before decryption.

Hope this helps.