Decoding an ancient format

Question

It's a very interesting file format may greatly satisfies binary archaeologists. sample file

Background

This file format was created in 90s to archive customer created AutoCAD components library like window, door and column. The first line is in plain text xxxx DWG library 1.0, it has been removed from the sample file to avoid any problem. The company xxx already gone, not sure if anyone still hold the trademark though. Lots of my father's design documents depends on it.

This file format has a companion index/reference file which is a simple dBase III. The working platform is DOS/Windows.

Findings

I spent 2 days tried to understand the format. While still far from the full picture here are what I've got:

• It consists of two parts, a file list section and a section of blob array
• The file list section is a simple array of file name + a mythical 3 bytes file codes (let's call it m1) m1 points to the position of its binary, 2 bytes. the left 1 byte seems related to the file/record name. a 4 bytes int points to the location of its corresponding binary.
  • the filename is 0x20 in length, empty spaces filled with 0x00.
  • m1 is not position offset as it usually would be.
  • file records are separated by 0x00
• The blob array section is fun.
  • each blob starts with "AS LZW 1.0 " (don't miss the tailing space)
  • then 2 bytes of unknown usage. Not random. examples: 0x6323, 0x5b23, 0xfa22, 0x0223. (call it m2) the size of the decoded binary.
  • 6 bytes of 0x00
  • 2 bytes of unknown usage. Not random. mostly 0x9ccc (call it m3)
  • then 0xaa08 + 12 bytes of 0x00
  • LZW old-style codes encoded binary: 0c00 0000 0083 0c89 0123 460d 0008 113e 08c0 ...

Questions

1. Are the blobs in the blob array sections really compressed with LZW? I tried to decode with python lzw from m2, m3 positions but both failed.
2. Is m1 holding the key to decode blobs?
3. What's the usage of m2, m3?
4. Is there a general methodology to decode file format like this? (well, I have only google :( and file signature list) 010 Editor (decent software, thanks @0xC0000022L)

Update

• m1 are 3 bytes. therefore ruled out the possibility of being CRC16 for the blob entry.
• Thanks @0xC0000022L for his comment. The first line of the file is: ArchStar DWG Library 1.0 (line ends with 0x0d)

• tried Trid on the blob (after chopping off "AS LWS 1.0"), it reports

  100.0% (.CEL) Autodesk FLIC Image File (extensions: flc, fli, cel) (7/3)

  makes no sense.

Update 21/11/13

Progress The python lzw seems to be relatively recent. So I looked into older LZW implementation in last century. So I found the lzw module of libtiff, and it's python implementation. (have to use the bitarray module inside of it: ver 0.35 instead of the most recent ver). I tried this python library with the brute force method (thanks to @Attila), the blob is indeed decode-able from 28h (based: "AS LZW"). And most importantly, the decoded starts with "AC10", AutoCAD DWG format! However, it only decoded around 30% of the blob.

Progress 2 Tried lzw from golang, with the decoding configuration order:LSB and litWid: 2, decoded around 3x more data. Will report more result tomorrow.

Progress 3 The dwg decoded with python LibTiff LZW is a valid AutoCAD DWG. AutoCAD 2014 reported the one decoded from golang LZW invalid. Now I guess the rest part of the blob is image thumbnail of the vector. Will decode all 50 blobs and report back.

Update 22/11/13

The blobs decoded with the libtiff lzw (lzw.py) are not valid although they have correct header "AC10".

I looked carefully into the lzw.py inside of pylibtiff, and found the first two lines:

""" Encoder and decoder of Lempel-Ziv-Welch algorithm for TIFF.

This module is obsolete, use tif_lzw extension module instead. """ (lesson learned: read every line!)

So I modified my codes to use tif_lzw to decode the blob. No luck. This time, I read the code carefully, line by line.

At the first glance, the file has two more Python wrapper methods Py_decode and Py_encode. Then I noticed the disabled methods LZWDecodeCompat for Old-style LZW codes. It's kind of interesting. So I modified the module, enabled LZWDecodeCompat and use it in the Py_decode method. the tif_lzw.decode method requires an extra parameter size. Naturally I used m1 (for the first blob, i used 0x4c07). The decoded result have "AC10" header, and opened successfully in AutoCAD(mac ver) but show nothing. AutoCAD says it's a Trusted DWG and contains 1 block which I couldn't find. So I spent a good few hours installed Windows and AutoCAD. bingo! there is a column drawing! There are still remaining questions. e.g. still not able to find the right size for tif_lzw.decode method call.

progress decoded all 50 binaries with:

• tif_lzw LZWDecodeCompat
• use m3 as the size paramter

Template (work in progress)

struct FILE {

    char banner[26];
    char f1[2];
    FSkip(4); //char sp[4];

    local int count = 0;
    local int pos;
    local char sp[36];
    local char SP[36];
    Memset(SP, 0, 36);

    while(true){
        pos = FTell();
        ReadBytes(sp, pos, 36);

        if(Memcmp(sp, SP, 36) == 0){
            break;
        }
        count ++;
        FSeek(pos);

        struct {
            char name[32];
            DWORD loc;
        } index;
    }


    FSkip(36); //char sp2[36];
    struct Binary(int len){
            char banner[11];
            WORD size;
            FSkip(7);//char sp[7];
            char m3[2]; // CRC16?
            char m4[2]; // always AA 08
            FSkip(16);
            char lzw[len];
    };

    local int i;

    for(i = 1; i < count; i++){
        Binary b(index[i].loc - index[i-1].loc - 40);
    }

    if(count>0){
        Binary b(FileSize() - index[i-1].loc - 40);
    }
} file;

Answer 1

Consider changing your Python script to run LZW decompression from various offsets, that is in a brute force manner.

Answer 2

Feel free to edit this community-wiki "answer".

NB: Only writing this as answer because it's too long for a comment. I think given the bits and pieces of information there is no answer unless you get lucky and someone happens to know this particular file format.

If we assume that the file format has indeed something to do with DWG, we can learn a lot by looking into old or new specifications (Google-search for OpenDesign_Specification_for_.dwg_files.pdf) for DWG and finding out about their used file format. Of particular interest are the data types (see "BIT CODES AND DATA DEFINITIONS" and "Compression" of the linked paper) and algorithms. For example in the mentioned section we find:

The AutoCAD DWG file format uses a modification of a standard cyclic redundancy check as an error detecting mechanism. The CRC ends up being 2 bytes long due to a lookup in a table containing 256 16-bit values, and are not stored in any sort of bit code form. They also always appear on byte boundaries; they are not embedded within the stream of bits.

This is pretty valuable information and helps you confirm findings.

Also interesting, assuming that this holds for your case:

Autodesk also uses a method whereby the result of the CRC is XORed with a "magic number". This method is used extensively in pre-R13 files [...]

BricsCAD and others provide SDKs and libraries (and products) which they claim understand also old DWG formats.

But back to investigating the format yourself.

You'll want to get handy with a tool such as 010 Editor that allows you to create a template and literally develop your understanding of the file format this way.

Now, you have deprived us of vital information by stating that the first line is text but not giving a length or anyway an idea how you come to the conclusion that it's plain text.

There are for example 50 occurrences of AS LZW 1.0 in the file. Still, I count more occurrences of the mysterious ASCII numbers at the beginning. Possible that this counts some markers as well that aren't in need of the AS LZW 1.0 bit.

Also those decimals start out at 2. Why?

I'm also unable to follow your description, to be honest. If the first part are file names and they are separated by 00, then why are there so many occurrences of 00 in that part?

ArchStar explains the AS LZW 1.0 of course. It's probably a variation on the "normal" LZW then. LZW could of course also stand for something unrelated, like the author's initials or their name for something.

---

Concerning the update about CEL/FLIC files. If we assume that the information here is correct, we should see something useful here. But it looks like gibberish indeed.

So I agree, likely no FLIC file. Still, keep in mind that DWG comes from the same creator. This may not be a coincidence.

---

Update concerning the findings about the LZW data. From Wikipedia (here):

In this way the decoder builds up a dictionary which is identical to that used by the encoder, and uses it to decode subsequent input values. Thus the full dictionary does not need be sent with the encoded data; just the initial dictionary containing the single-character strings is sufficient (and is typically defined beforehand within the encoder and decoder rather than being explicitly sent with the encoded data.)

(Emphasis added.)