5

I got data which is compressed but I fail to find the compression algorithm. The data is part of a larger file from which I know the layout, So I managed to find out few things. What I know:

  • I don't have the binary executable that load the data, I only have the updated version which no longer support the old copression algo. I tortured it in many way and it just doesn't contain the corresponding code
  • it is compressed (100% sure of it)
  • it can be home made as it was replaced later (see below)
  • no magic numbers so far
  • it is not plain:
    • deflate (wrong header)
    • lzma (wrong header)
    • gzip (wrong header)
    • Quantum (wrong header)
    • Microsoft CAB (wrong header)
    • Bzip2 (wrong header)
    • Zip (wrong header)
  • the uncompressed size is given in the file containing the data, this file layout is fully reversed and does not contain any clue
  • it might be encrypted but is unlikely because of speed requirements
  • if it is encrypted, it gives the same output given the same data input at the beginning of the sequence (by guess on some uncompressed data located nearby)
  • it is from 2001 and has been replaced by deflate since
  • Some of those data only output ASCII and nothing else (I know it from the layout of the container file) and have a compression ratio of about 0.30 (compressedSize/uncompressedSize) everytime
  • I don't have any before/after data sadly

EDIT: There are the 32 first bytes in hex: b9daed36cb64bedb61b9dd2cb72afd8ee565b0dd2ea00f0afda2c36eb25b0016

I made histograms of several of those data and they all match a specific pattern. Something is going on with the powers of 2 obviously but I fail to see what.

Anyone has a idea what it could be? What can I do to gather further information? Does it look Lempel-Ziv based? If yes how could I reverse it?

Improve this question
8
  • If you don't have the executable how did you get this file? Also, show a dump of the 16-32 bytes of the header.
    – Igor Skochinsky ♦
    Aug 19, 2014 at 13:32
  • Updated to answer + hex dump of the header
    – search4everNever
    Aug 19, 2014 at 13:46
  • You may find this question and answers interesting to look at: > Are there any tools or scripts for identifying compression algorithms in executables?
    – perror
    Aug 19, 2014 at 13:49
  • "The output is ASCII and nothing else" - that, coupled with a ratio of "about 0.30" actually suggests a simple compression scheme, rather than a complicated: a dedicated text-only compression scheme. Can we see this file?
    – Jongware
    Aug 19, 2014 at 14:20
  • 1
    Have you tried contacting the company that made the software?
    – Jason Geffner
    Aug 20, 2014 at 14:51

1 Answer 1

Reset to default
3
  • First of all, you forgot lzo. AFAIR it was invented in 1996.
  • Second of all, it is possible that the compressed data has no corresponding standard header (which makes all "incorrect header" mistakes not necessarily true).
  • Third of all, the histogram doesn't look like histogram of data compressed as one piece, it is possible that it has some internal structure (may be blobs with sizes before them ?).

I'd suggest to write script that runs different decompression algorithms on parts of the data and to observe results.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.