2

I've been trying to reverse engineer the format of data files from the 1997 video game Helicops. I have experience reverse engineering binary files from games of this period but limited exposure to compression schemes.

I've been able to figure out the header format (which I've documented here; it uses an XOR "cipher" with the key 0xAA), but the individual data blocks within each file use a form of compression I haven't been able to figure out.

I'm certain the data are compressed: the header in each DAT file contains a table listing each data block's name, starting offset, and two values: one is the length of the data block in the DAT file and the other appears to be the data block's uncompressed length (notably, it's 768 for multiple data blocks that appear to be colour tables; 768 = 256 colours * 3 bytes/colour and these data blocks have names that end with "ACT", i.e., Adobe Color Table, and some include obvious grayscale ramps).

My full notes on the compression scheme are here, but I haven't been able to figure out much. Here's a summary:

  • In multiple data files, 0xFF is followed by 8 uncompressed bytes. This is consistent across multiple files containing different types of recognizable data (ASCII text and colour palettes), so it seems likely that all the game's data files use the same type of compression.
  • Currently, I have no idea how 0xFF translates to "the next 8 bytes are uncompressed data". Since the header uses an XOR cipher, I've tried XORing it with various values (0xAA since that's the key for the header data, 0xF7 since that gives 0x08, etc.) but I haven't come up with any that make sense for 0xFF and some of the other values that are clearly compression-related. My guess is that ranges of bits within each byte contain different compression-related info, but that the bytes need to be altered in some way first.
  • Based on bytes that are evidently uncompressed data, it's clear that there are many sequences of bytes related to data compression (see examples in the sample data below), but I haven't been able to figure out how to decode them.
  • I suspect the compression scheme uses some form of run-length encoding and/or something along the lines of LZ77 where references to sequences elsewhere in the data are used to reduce overall size.

I've uploaded a sample DAT file, PIC.DAT, here. It contains substantial quantities of ASCII text that make identifying compression-related bytes fairly easy. A list of its data blocks (parsed from its header) is here.

Here are some sample data from the data block starting at offset 184456 in PIC.DAT (PIC.DAT's header identifies this block as being associated with the third mission set, "Data Space Demon", and the first mission in that set, "Tower Attack"; note that these strings appear in the data below):

Bytes Notes/ASCII
FF
52 45 4D 20 4D 69 73 73 "REM Miss"
FF
33 31 2E 50 49 43 20 2D "31.PIC -"
FE FD F4
20 54 6F 77 65 72 20 " Tower "
BF 28 BF = 191/-65, 28 = 40/40
54 6F 6B 79 6F "Tokyo" (not part of level name)
05 03 29 FB 0D 0A EE F1
44 61 74 61 20 "Data "
FF
53 70 61 63 65 20 44 65 "Space De"
EF 239/-17
6D 6F 6E "mon"
2C 05 04
41 74 74 "Att"
F7 247/-9
61 63 6B "ack"
19 03
43 6F 70 69 "Copi"
FF
65 64 20 66 72 6F 6D 20 "ed from "
7F
48 43 5F 57 6F 72 6B "HC_Work"
FC F6 FE 5E 0E
20 20 31 2D 32 33 2D " 1-23-"
7F
39 37 20 44 42 0D 0A "97 DB\r\n"
19 03 F1 2A 84 0F 8A 02 19 03
[remaining bytes follow]

There are additional sample data here, including colour palettes (some of which appear to be minimally-compressed).

I'd appreciate any suggestions regarding how this compression format might work! I'll credit any assistance in the file format documentation I'm writing.

2 Answers 2

2

It's a very similar compression method to that used in this question.

Specifically, the compressed data begins with a flag byte. Each bit of this in turn (from bit 0 to bit 7) indicates that the next decompressed byte(s) are generated by -

  • (when the flag bit is 1) copying a single literal byte from the compressed data stream or
  • (when the flag bit is 0) copying between N bytes from elsewhere in the decompressed data stream. The length and location of these bytes are encoded in the following 2 bytes in the compressed data stream.

Hence 0xFF is 8x 1s and represents 8 uncompressed bytes.

Your example file begins with 2 of these, as you identified.

These are then followed by an FE = 0b11111110 So this is followed by -

  • 1 compressed byte-pair FD F4, and
  • 7 uncompressed literal bytes 20 54 6F 77 65 72 20

I think the length part of the compressed byte pair is likely given by 3 + the last nibble (in this case 3+4=7.) I haven't quite found an interpretation for the offset/location part of these 2 bytes that works consistently yet though.

1
  • Thanks, this is very helpful! Knowing this I'm going to take another look at the color palettes (some of which are here); given the relative simplicity of their data, they should help to narrow down the meaning of the compressed byte pairs. Jan 11, 2023 at 22:14
0

The mix of readable text fragments and high byte values makes me suspect LZSS but it may also be some simple RLE-like algorithm. To be completely sure you’d need to find the code which decompresses this data in the executable code.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.