1

Currently I'm in reversing an old MS-DOS Neuromancer game (Interplay Productions, 1988. Based on William Gibson's novel). For now I have already written an utility that parses game resources and extracts sprites. Sprites are bitmaps that compressed with some fancy algorithm. I have restored that algorithm by porting it's 16-bit disassembly to 64-bit assembly (I'm working on Windows 10 and using MASM in MSVS 2017). It results in ~800 lines of pretty obfuscated assembly (which became even more obfuscated after porting it to 64-bit) that works and I'm about rewriting it in plain old C. However it makes no sense because i can't identify an actual algorithm.

And here is the problem. I assume that the game uses some widly known data compression algorithm. I will provide an example by posting chunks of data that are observable during the decompression process. I hope that there are some compression experts who will recognize the algorithm (if my theory about "widly known algorithm" is correct). Thank you in advance!

Here we go, decompressing CURSORS.IMH file that contains a series of bitmaps that shows diffrent in-game cursors:

SRC: Initial data, 211 bytes stored in 512 byte buffer:
0x0000  3f 01 00 00 04 40 7e ec d0 c2 2d 0a 46 a3 3f f2
0x0010  71 11 62 30 64 0e 10 4c e0 c5 85 05 fc ff 50 01
0x0020  04 c0 26 0f e2 c0 85 c2 50 17 fa 05 54 b0 6c 2d
0x0030  74 13 e9 c7 45 1d d8 b6 2c 18 e7 14 7b 8b d7 9b
0x0040  bb e5 eb 60 4d 2f ef 70 3a 1e 42 fe d9 c0 5d c0
0x0050  eb cd ee 07 b9 bf 82 1c 01 41 bb 15 36 0b 74 3a
0x0060  4b c8 74 9e 45 d7 5b 26 63 f4 24 e9 da ea 5c c6
0x0070  e8 59 d7 93 41 f7 94 df b7 ac 4d b7 ef c2 cc 4f
0x0080  5d 5f 66 d1 e5 e3 3f 2b ac 42 7c 5e 3a f1 9f 95
0x0090  d7 d9 e8 e9 3d 75 62 de 9b 05 86 e6 0e bf 79 4c
0x00A0  1d 0b 3a 76 9a 97 31 ba 12 74 ed 75 26 63 fe 79
0x00B0  17 5f bc 87 fd 58 9b 6f dd 0b 12 f2 65 2f 12 fb
0x00C0  1e 99 5e 37 b2 4c 42 4c 4f 4c af 1b 6c c4 be c7
0x00D0  c9 9f c0

Processing starts from the fist byte of SRC and suspends on 44th byte. Here is the intermediate result of that processing stored in 2048 byte buffer located right after the SRC buffer (actually, there are 10 bytes between them, where among other things an address of 44th byte is stored):

0x0200  00 80 01 00 5c ea 3f 01 00 00
0x020A  02 00 02 00 0e 00 04 00 1f 00 05 00 04 00 04 00
0x021A  18 00 05 00 3a 00 07 00 0d 00 05 00 00 00 00 00
0x022A  05 00 04 00 df 00 08 00 0d 00 08 00 de 00 08 00
0x023A  05 00 07 00 00 00 00 00 00 00 00 00 00 00 00 00
0x024A  00 00 05 00 05 00 05 00 00 00 00 00 00 00 00 00
0x025A  00 00 00 00 00 00 00 00 0c 00 08 00 00 00 00 00
0x026A  00 00 00 00 0f 00 08 00 0e 00 08 00 00 00 00 00
... ZEROES ...
0x02CA  1a 00 05 00 00 00 00 00 00 00 00 00 0c 00 05 00
0x02DA  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x02EA  04 00 05 00 00 00 00 00 00 00 00 00 00 00 00 00
... ZEROES ...
0x038A  6e 00 07 00 1c 00 06 00 00 00 00 00 00 00 00 00
0x039A  00 00 00 00 00 00 00 00 09 00 08 00 00 00 00 00
... ZEROES ...
0x040A  19 00 05 00 00 00 00 00 00 00 00 00 07 00 05 00
0x041A  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x042A  06 00 05 00 00 00 00 00 00 00 00 00 00 00 00 00
... ZEROES ...
0x05EA  00 00 00 00 3b 00 07 00 00 00 00 00 08 00 08 00
0x05FA  36 00 06 00 0f 00 05 00 1e 00 05 00 01 00 04 00
0x060A  05 10 00 00 05 10 00 00 05 10 00 00 05 10 00 00
0x061A  05 10 00 00 05 10 00 00 05 10 00 00 05 10 00 00
0x062A  08 fb 00 00 08 66 00 00 07 0c 00 00 07 0c 00 00
0x063A  08 16 00 00 08 0a 00 00 08 1a 00 00 08 19 00 00
0x064A  04 ff 00 00 04 ff 00 00 04 ff 00 00 04 ff 00 00
0x065A  04 ff 00 00 04 ff 00 00 04 ff 00 00 04 ff 00 00
0x066A  04 ff 00 00 04 ff 00 00 04 ff 00 00 04 ff 00 00
0x067A  04 ff 00 00 04 ff 00 00 04 ff 00 00 04 ff 00 00
0x068A  05 38 00 00 05 38 00 00 05 38 00 00 05 38 00 00
0x069A  05 38 00 00 05 38 00 00 05 38 00 00 05 38 00 00
0x06AA  05 11 00 00 05 11 00 00 05 11 00 00 05 11 00 00
0x06BA  05 11 00 00 05 11 00 00 05 11 00 00 05 11 00 00
0x06CA  05 88 00 00 05 88 00 00 05 88 00 00 05 88 00 00
0x06DA  05 88 00 00 05 88 00 00 05 88 00 00 05 88 00 00
0x06EA  05 83 00 00 05 83 00 00 05 83 00 00 05 83 00 00
0x06FA  05 83 00 00 05 83 00 00 05 83 00 00 05 83 00 00
0x070A  04 03 00 00 04 03 00 00 04 03 00 00 04 03 00 00
0x071A  04 03 00 00 04 03 00 00 04 03 00 00 04 03 00 00
0x072A  04 03 00 00 04 03 00 00 04 03 00 00 04 03 00 00
0x073A  04 03 00 00 04 03 00 00 04 03 00 00 04 03 00 00
0x074A  04 08 00 00 04 08 00 00 04 08 00 00 04 08 00 00
0x075A  04 08 00 00 04 08 00 00 04 08 00 00 04 08 00 00
0x076A  04 08 00 00 04 08 00 00 04 08 00 00 04 08 00 00
0x077A  04 08 00 00 04 08 00 00 04 08 00 00 04 08 00 00
0x078A  05 33 00 00 05 33 00 00 05 33 00 00 05 33 00 00
0x079A  05 33 00 00 05 33 00 00 05 33 00 00 05 33 00 00
0x07AA  05 06 00 00 05 06 00 00 05 06 00 00 05 06 00 00
0x07BA  05 06 00 00 05 06 00 00 05 06 00 00 05 06 00 00
0x07CA  06 61 00 00 06 61 00 00 06 61 00 00 06 61 00 00
0x07DA  07 05 00 00 07 05 00 00 07 f9 00 00 07 f9 00 00
0x07EA  05 fd 00 00 05 fd 00 00 05 fd 00 00 05 fd 00 00
0x07FA  05 fd 00 00 05 fd 00 00 05 fd 00 00 05 fd 00 00
0x080A  02 00 00 00 02 00 00 00 02 00 00 00 02 00 00 00
0x081A  02 00 00 00 02 00 00 00 02 00 00 00 02 00 00 00
0x082A  02 00 00 00 02 00 00 00 02 00 00 00 02 00 00 00
0x083A  02 00 00 00 02 00 00 00 02 00 00 00 02 00 00 00
0x084A  02 00 00 00 02 00 00 00 02 00 00 00 02 00 00 00
0x085A  02 00 00 00 02 00 00 00 02 00 00 00 02 00 00 00
0x086A  02 00 00 00 02 00 00 00 02 00 00 00 02 00 00 00
0x087A  02 00 00 00 02 00 00 00 02 00 00 00 02 00 00 00
0x088A  02 00 00 00 02 00 00 00 02 00 00 00 02 00 00 00
0x089A  02 00 00 00 02 00 00 00 02 00 00 00 02 00 00 00
0x08AA  02 00 00 00 02 00 00 00 02 00 00 00 02 00 00 00
0x08BA  02 00 00 00 02 00 00 00 02 00 00 00 02 00 00 00
0x08CA  02 00 00 00 02 00 00 00 02 00 00 00 02 00 00 00
0x08DA  02 00 00 00 02 00 00 00 02 00 00 00 02 00 00 00
0x08EA  02 00 00 00 02 00 00 00 02 00 00 00 02 00 00 00
0x08FA  02 00 00 00 02 00 00 00 02 00 00 00 02 00 00 00
0x090A  05 04 00 00 05 04 00 00 05 04 00 00 05 04 00 00
0x091A  05 04 00 00 05 04 00 00 05 04 00 00 05 04 00 00
0x092A  05 80 00 00 05 80 00 00 05 80 00 00 05 80 00 00
0x093A  05 80 00 00 05 80 00 00 05 80 00 00 05 80 00 00
0x094A  05 30 00 00 05 30 00 00 05 30 00 00 05 30 00 00
0x095A  05 30 00 00 05 30 00 00 05 30 00 00 05 30 00 00
0x096A  06 fc 00 00 06 fc 00 00 06 fc 00 00 06 fc 00 00
0x097A  07 60 00 00 07 60 00 00 08 0b 00 00 08 09 00 00
0x098A  04 01 00 00 04 01 00 00 04 01 00 00 04 01 00 00
0x099A  04 01 00 00 04 01 00 00 04 01 00 00 04 01 00 00
0x09AA  04 01 00 00 04 01 00 00 04 01 00 00 04 01 00 00
0x09BA  04 01 00 00 04 01 00 00 04 01 00 00 04 01 00 00
0x09CA  05 fe 00 00 05 fe 00 00 05 fe 00 00 05 fe 00 00
0x09DA  05 fe 00 00 05 fe 00 00 05 fe 00 00 05 fe 00 00
0x09EA  05 02 00 00 05 02 00 00 05 02 00 00 05 02 00 00
0x09FA  05 02 00 00 05 02 00 00 05 02 00 00 05 02 00 00

Processing continues from 44th byte of SRC. The remaining bytes are processed with result stored in external intemediate buffer:

IMM: Intermediate data, 319 bytes:
0x0000  00 00 00 00 08 00 0a 00 ff 06 05 66 fe 00 61 05
0x0010  11 ff 60 04 00 fc 11 00 16 61 01 11 ff 01 01 11
0x0020  01 00 fe 06 60 02 11 01 00 fc 10 00 06 11 02 00
0x0030  fe 01 10 01 00 ff 01 03 11 02 00 fc 61 10 00 01
0x0040  01 10 01 00 fe 06 01 01 10 fe 01 06 02 00 fb 61
0x0050  10 11 10 60 04 00 00 00 06 00 0c 00 01 00 ff 30
0x0060  03 00 fe 03 83 03 00 fd 38 08 30 01 00 fc 03 80
0x0070  00 83 01 00 ff 38 01 00 f9 08 30 00 08 80 00 88
0x0080  01 00 ff 33 01 00 fe 03 30 19 00 fe 08 88 02 00
0x0090  0b 00 04 00 06 00 09 00 02 00 ff 33 04 00 fd 08
0x00A0  30 00 02 33 fc 00 83 00 08 01 88 fd 80 08 30 04
0x00B0  00 ff 83 04 00 fe 83 08 01 88 fd 80 08 30 02 33
0x00C0  fe 00 83 03 00 fd 08 30 00 04 00 0b 00 06 00 0c
0x00D0  00 ff 00 01 33 ff 30 02 00 fe 08 88 1a 00 ff 33
0x00E0  01 00 f9 03 30 00 08 80 00 88 01 00 ff 38 01 00
0x00F0  f9 08 30 00 03 80 00 83 02 00 fd 38 08 30 02 00
0x0100  fe 03 83 02 00 00 00 04 00 06 00 09 00 01 00 ff
0x0110  33 03 00 fe 03 80 03 00 fe 38 00 02 33 fd 03 80
0x0120  08 01 88 fe 80 38 04 00 ff 38 04 00 fd 03 80 08
0x0130  01 88 fc 80 00 38 00 02 33 fd 00 03 80 02 00

Finally, some processing performed on IMM and we have the result:

DST: The Result, 372 bytes:
0x0000  00 00 00 00 08 00 0A 00 06 66 66 66 66 66 66 00 
0x0010  67 77 77 77 77 77 77 60 67 77 77 77 77 66 77 76
0x0020  06 66 66 76 66 77 77 76 00 06 77 67 77 77 77 66
0x0030  00 00 66 67 77 77 76 76 00 00 67 76 66 66 67 76
0x0040  00 00 06 66 66 67 77 66 00 00 00 67 76 77 76 60
0x0050  00 00 00 06 66 66 66 00 04 00 00 00 06 00 0C 00
0x0060  00 00 30 00 00 00 00 03 B3 00 00 00 00 3B BB 30
0x0070  00 00 03 BB BB B3 00 00 3B BB BB BB 30 00 33 3B
0x0080  BB 33 30 00 00 3B BB 30 00 00 00 3B BB 30 00 00
0x0090  00 3B BB 30 00 00 00 3B BB 30 00 00 00 3B BB 30
0x00A0  00 00 00 33 33 30 00 00 0B 00 04 00 06 00 09 00
0x00B0  00 00 00 33 00 00 00 00 00 3B 30 00 33 33 33 3B
0x00C0  B3 00 3B BB BB BB BB 30 3B BB BB BB BB B3 3B BB
0x00D0  BB BB BB 30 33 33 33 3B B3 00 00 00 00 3B 30 00
0x00E0  00 00 00 33 00 00 04 00 0B 00 06 00 0C 00 00 33
0x00F0  33 30 00 00 00 3B BB 30 00 00 00 3B BB 30 00 00
0x0100  00 3B BB 30 00 00 00 3B BB 30 00 00 00 3B BB 30
0x0110  00 00 33 3B BB 33 30 00 3B BB BB BB 30 00 03 BB
0x0120  BB B3 00 00 00 3B BB 30 00 00 00 03 B3 00 00 00
0x0130  00 00 30 00 00 00 00 00 04 00 06 00 09 00 00 00
0x0140  33 00 00 00 00 03 B3 00 00 00 00 3B B3 33 33 33
0x0150  03 BB BB BB BB B3 3B BB BB BB BB B3 03 BB BB BB
0x0160  BB B3 00 3B B3 33 33 33 00 03 B3 00 00 00 00 00
0x0170  33 00 00 00

Decompression completed and it's easy to derive bitmaps like this:

  1. Get first 8 bytes of DST : 00 00 00 00 08 00 0A 00
  2. Multiply last 2 words : 08 * 0A = 50 (80)

  3. Get next 80 bytes of DST :

    06 66 66 66 66 66 66 00 67 77 77 77 77 77 77 60
67 77 77 77 77 66 77 76 06 66 66 76 66 77 77 76
00 06 77 67 77 77 77 66 00 00 66 67 77 77 76 76
00 00 67 76 66 66 67 76 00 00 06 66 66 67 77 66
00 00 00 67 76 77 76 60 00 00 00 06 66 66 66 00

  4. Arrange those bytes assuming that 08 and 0A are width and height respectively:

    06 66 66 66 66 66 66 00
67 77 77 77 77 77 77 60
67 77 77 77 77 66 77 76
06 66 66 76 66 77 77 76
00 06 77 67 77 77 77 66
00 00 66 67 77 77 76 76
00 00 67 76 66 66 67 76
00 00 06 66 66 67 77 66
00 00 00 67 76 77 76 60
00 00 00 06 66 66 66 00

  5. Extend this with zeroes:

    00 06 06 06 06 06 06 06 06 06 06 06 06 06 00 00
06 07 07 07 07 07 07 07 07 07 07 07 07 07 06 00
06 07 07 07 07 07 07 07 07 07 06 06 07 07 07 06
00 06 06 06 06 06 07 06 06 06 07 07 07 07 07 06
00 00 00 06 07 07 06 07 07 07 07 07 07 07 06 06
00 00 00 00 06 06 06 07 07 07 07 07 07 06 07 06
00 00 00 00 06 07 07 06 06 06 06 06 06 07 07 06
00 00 00 00 00 06 06 06 06 06 06 07 07 07 06 06
00 00 00 00 00 00 06 07 07 06 07 07 07 06 06 00
00 00 00 00 00 00 00 06 06 06 06 06 06 06 00 00

Thats it! Wrapping it in BMP header gives us a cute cursor image: enter image description here.

And here are the others: enter image description here enter image description here enter image description here enter image description here.

UPDATE

Thanks to @IanCook I have figured out how the IMM is decoded to the DST.

Improve this question
3
  • A quick inspection suggests that the compressed data in IMM is some form of run length encoding, based on the (xor) difference in pixel values compared to the previous row.
    – Ian Cook
    Commented Mar 22, 2018 at 22:12
  • @IanCook, I have checked this and you absolutly right! Thx.
    – Henadzi Matuts
    Commented Mar 23, 2018 at 7:34
  • 1
    @HenadziMatuts please provide the solution as an answer so we can a) close this question and b) share the knowledge
    – Remko
    Commented Mar 23, 2018 at 14:49

1 Answer 1

Reset to default
1

Well then, here is some results of my researching:

  1. The SRC is a compressed data. The first WORD of the SRC contains the lenght of the decompressed data (as it equals to the lenght of the IMM). The following WORD is zero-padding. Actual compression method is still unknown.

  2. The IMM is not really "compressed" but run-lengh encoded. It contains blocks of bytes prepended by 8-byte header. Product of the last two words of that header equals to the lenth of the decoded block. In that kind of RLE bytes with a value greater than 0x7F indicate the amount of following bytes that should go unchanged to the DST. The other bytes indicate how much times the following byte should be copied to DST. I goes like this:

    1. SRC: 00 00 00 00 08 00 0A 00 ff 06 05 66 fe 00 61 05 11 ...
    2. DST: 00 00 00 00 08 00 0A 00 06 66 66 66 66 66 66 00 61 11 11 11 11 11 11 ...
  3. After such decoding, assuming that the last two words of the header are amounts of columns and rows, the actual pixel values are obtained by consecutive XOR'ing of adjacent rows like this:
    1. ROW[1] = ROW[0] XOR ROW[1]: [0] 06 66 66 66 66 66 66 00 [1] 61 11 11 11 11 11 11 60 [2] 00 00 00 00 00 11 00 16 ...
    2. ROW[2] = ROW[1] XOR ROW[2]: [0] 06 66 66 66 66 66 66 00 [1] 67 77 77 77 77 77 77 60 [2] 00 00 00 00 00 11 00 16 ...
    3. ROW[3] = ROW[2] XOR ROW[3]: [0] 06 66 66 66 66 66 66 00 [1] 67 77 77 77 77 77 77 60 [2] 67 77 77 77 77 66 77 76 ...

That's all for now. Implementation of that RL decoding with row XORing takes ~50 lines of C code against ~400 lines were in disassembly. Not sure that I should mark this answer as correct as actual compression method still unknown.

Improve this answer
1
  • 1
    Well done! As for the type of compression: this is an old game, it would be rather unusual to find any "real world" compression applied to it; they tend to use something cobbled up by the programmers at the spur of the moment, "whatever decreases the size". It is clearly based on RLE, and I suppose the XOR thing was an attempt to increase compression on slightly different data, comparable to PNG's row filters.
    – Jongware
    Commented Mar 24, 2018 at 18:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.