9

I'm recovering data from old tape cartridges from circa 1994, where the user had forgotten what software was used to write them, and it doesn't seem to be any format I recognize. Fortunately the file structure within the backup is fairly straightforward (it's just literally one file followed by another, aligned on a block boundary), so I was able to extract the file contents, but the files appear to be compressed using what appears to be a primitive compression scheme that I feel should be easy enough to reverse-engineer to a trained eye, so I thought I'd post it here.

Fortunately the backup contains the user's DOS directory, with compressed versions of files from the standard DOS distribution (available on the web), so I'm able to cross-reference the compressed files against their known uncompressed versions.

For example, here's the first few lines of the compressed version of DOSHELP.HLP, which is a simple plaintext file:

00000000  00 21 10 15 40 20 43 6F 70 79 72 69 67 68 74 20  .!..@ Copyright 
00000010  28 43 29 20 31 39 39 30 2D 13 21 10 0B 39 34 20  (C) 1990-.!..94 
00000020  4D 69 63 72 6F 73 6F 66 0D 51 10 02 91 72 70 2E  Microsof.Q..‘rp.
00000030  20 20 41 6C 6C 20 09 B1 10 74 30 07 2E 21 10 0A    All .±.t0..!..
00000040  65 73 65 72 76 65 64 2E 0D 0A 03 41 05 68 90 06  eserved....A.h..
00000050  32 11 14 20 40 07 68 50 06 1B 21 10 09 53 2D 44  2.. @.hP..!..S-D
00000060  4F 53 20 67 65 6E 36 11 06 2D 61 14 02 71 6C 70  OS gen6..-a..qlp
00000070  20 66 69 6C 65 28 01 02 49 D0 10 02 71 63 6F 6E   file(..IÐ..qcon
00000080  74 61 69 6E 32 11 06 20 20 06 09 51 06 66 00 02  tain2..  ..Q.f..
00000090  3C 31 10 64 40 13 1E 91 06 70 40 07 69 20 16 20  <1.d@..‘[email protected] . 
000000A0  20 12 02 61 20 65 61 63 68 20 61 21 10 0D 6D 6D   ..a each a!..mm

and the reference uncompressed version:

00000000  40 20 43 6F 70 79 72 69 67 68 74 20 28 43 29 20  @ Copyright (C) 
00000010  31 39 39 30 2D 31 39 39 34 20 4D 69 63 72 6F 73  1990-1994 Micros
00000020  6F 66 74 20 43 6F 72 70 2E 20 20 41 6C 6C 20 72  oft Corp.  All r
00000030  69 67 68 74 73 20 72 65 73 65 72 76 65 64 2E 0D  ights reserved..
00000040  0A 40 20 54 68 69 73 20 69 73 20 74 68 65 20 4D  .@ This is the M
00000050  53 2D 44 4F 53 20 67 65 6E 65 72 61 6C 20 68 65  S-DOS general he
00000060  6C 70 20 66 69 6C 65 2E 20 20 49 74 20 63 6F 6E  lp file.  It con
00000070  74 61 69 6E 73 20 61 20 62 72 69 65 66 20 0D 0A  tains a brief ..
00000080  40 20 64 65 73 63 72 69 70 74 69 6F 6E 20 6F 66  @ description of
00000090  20 65 61 63 68 20 63 6F 6D 6D 61 6E 64 20 73 75   each command su
000000A0  70 70 6F 72 74 65 64 20 62 79 20 74 68 65 20 4D  pported by the M

It's very clearly a tantalizingly simple algorithm, but I'm hoping someone with more experience with compression can unravel it with greater ease than I can.

So far I'm only able to discern basic "run-length" portions, for example, in the first two lines:

00000000  00 21 10 15 40 20 43 6F 70 79 72 69 67 68 74 20  .!..@ Copyright 
00000010  28 43 29 20 31 39 39 30 2D 13 21 10 0B 39 34 20  (C) 1990-.!..94

...the 15 at offset 3 clearly indicates the next 0x15 bytes are uncompressed, and we see similar run lengths throughout, but beyond that is proving a challenge.

Here are links to download a few full example files:

Improve this question
6
  • 1
    Generally the more data there is available the easier these are to work out. Could you share both files ?
    – Ian Cook
    Commented Jan 26, 2023 at 23:00
  • 1
    @IanCook Sure, I linked to both at the end of the question.
    – Dmitry Brant
    Commented Jan 26, 2023 at 23:39
  • 1
    In the original the word fasthelp occurs multiple times but in the compressed one it occurs only once. This might mean that strings are written once and on next occurence an offset is stored. I think this mechanism is referred to as Run Length Encoding for Strings.
    – Remko
    Commented Feb 2, 2023 at 18:12
  • 1
    Based on the data given, the compression methodology employed appears to be a rudimentary run-length encoding technique, where a single byte serves as the indicator of the number of uncompressed bytes that follow. However, the absence of comprehensive information regarding the exact compression algorithm utilized makes it difficult to reverse-engineer the compression in its entirety. Further analysis of the compressed data may enable the creation of a decompression algorithm.
    – r0xdeadbeef
    Commented Feb 3, 2023 at 13:27
  • 1
    Nonetheless, it is also possible that the compression algorithm is proprietary and specific to the backup software, which would make the complete reverse-engineering of the algorithm a more extensive and complex endeavor
    – r0xdeadbeef
    Commented Feb 3, 2023 at 13:27

1 Answer 1

Reset to default
8

Just to close the loop on this: I ended up successfully reverse-engineering this compression format, which turned out to be written by an extremely obscure tape backup tool from the early 90s called TXPLUS.

Here is some code and documentation for uncompressing this format. It's a variant of LZ78, but it's unnecessarily convoluted and quite inefficient, which is probably why it's now in the dustbin of history.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.