It's a very interesting file format may greatly satisfies binary archaeologists. sample file
Background
This file format was created in 90s to archive customer created AutoCAD components library like window, door and column. The first line is in plain text xxxx DWG library 1.0
, it has been removed from the sample file to avoid any problem. The company xxx already gone, not sure if anyone still hold the trademark though. Lots of my father's design documents depends on it.
This file format has a companion index/reference file which is a simple dBase III. The working platform is DOS/Windows.
Findings
I spent 2 days tried to understand the format. While still far from the full picture here are what I've got:
- It consists of two parts, a file list section and a section of blob array
- The file list section is a simple array of file name +
a 4 bytes int points to the location of its corresponding binary.a mythical 3 bytes file codes (let's call it m1)m1 points to the position of its binary, 2 bytes. the left 1 byte seems related to the file/record name.- the filename is 0x20 in length, empty spaces filled with 0x00.
m1 is not position offset as it usually would be.file records are separated by 0x00
- The blob array section is fun.
- each blob starts with "AS LZW 1.0 " (don't miss the tailing space)
then 2 bytes of unknown usage. Not random. examples: 0x6323, 0x5b23, 0xfa22, 0x0223. (call it m2)the size of the decoded binary.- 6 bytes of 0x00
- 2 bytes of unknown usage. Not random. mostly 0x9ccc (call it m3)
- then 0xaa08 + 12 bytes of 0x00
- LZW old-style codes encoded binary: 0c00 0000 0083 0c89 0123 460d 0008 113e 08c0 ...
Questions
Are the blobs in the blob array sections really compressed with LZW? I tried to decode with python lzw from m2, m3 positions but both failed.Is m1 holding the key to decode blobs?- What's the usage of
m2,m3? - Is there a general methodology to decode file format like this? (well, I have only google :( and file signature list) 010 Editor (decent software, thanks @0xC0000022L)
Update
m1 are 3 bytes. therefore ruled out the possibility of being CRC16 for the blob entry.- Thanks @0xC0000022L for his comment. The first line of the file is: ArchStar DWG Library 1.0 (line ends with 0x0d)
tried Trid on the blob (after chopping off "AS LWS 1.0"), it reports
100.0% (.CEL) Autodesk FLIC Image File (extensions: flc, fli, cel) (7/3)
makes no sense.
Update 21/11/13
Progress The python lzw seems to be relatively recent. So I looked into older LZW implementation in last century. So I found the lzw module of libtiff, and it's python implementation. (have to use the bitarray module inside of it: ver 0.35 instead of the most recent ver). I tried this python library with the brute force method (thanks to @Attila), the blob is indeed decode-able from 28h (based: "AS LZW"). And most importantly, the decoded starts with "AC10", AutoCAD DWG format! However, it only decoded around 30% of the blob.
Progress 2 Tried lzw from golang, with the decoding configuration order:LSB and litWid: 2, decoded around 3x more data. Will report more result tomorrow.
Progress 3 The dwg decoded with python LibTiff LZW is a valid AutoCAD DWG. AutoCAD 2014 reported the one decoded from golang LZW invalid. Now I guess the rest part of the blob is image thumbnail of the vector. Will decode all 50 blobs and report back.
Update 22/11/13
The blobs decoded with the libtiff lzw (lzw.py) are not valid although they have correct header "AC10".
I looked carefully into the lzw.py inside of pylibtiff, and found the first two lines:
""" Encoder and decoder of Lempel-Ziv-Welch algorithm for TIFF.
This module is obsolete, use tif_lzw extension module instead. """ (lesson learned: read every line!)
So I modified my codes to use tif_lzw to decode the blob. No luck. This time, I read the code carefully, line by line.
At the first glance, the file has two more Python wrapper methods Py_decode and Py_encode. Then I noticed the disabled methods LZWDecodeCompat for Old-style LZW codes. It's kind of interesting. So I modified the module, enabled LZWDecodeCompat and use it in the Py_decode method. the tif_lzw.decode method requires an extra parameter size. Naturally I used m1 (for the first blob, i used 0x4c07). The decoded result have "AC10" header, and opened successfully in AutoCAD(mac ver) but show nothing. AutoCAD says it's a Trusted DWG and contains 1 block which I couldn't find. So I spent a good few hours installed Windows and AutoCAD. bingo! there is a column drawing! There are still remaining questions. e.g. still not able to find the right size for tif_lzw.decode method call.
progress decoded all 50 binaries with:
- tif_lzw LZWDecodeCompat
- use m3 as the size paramter
Template (work in progress)
struct FILE {
char banner[26];
char f1[2];
FSkip(4); //char sp[4];
local int count = 0;
local int pos;
local char sp[36];
local char SP[36];
Memset(SP, 0, 36);
while(true){
pos = FTell();
ReadBytes(sp, pos, 36);
if(Memcmp(sp, SP, 36) == 0){
break;
}
count ++;
FSeek(pos);
struct {
char name[32];
DWORD loc;
} index;
}
FSkip(36); //char sp2[36];
struct Binary(int len){
char banner[11];
WORD size;
FSkip(7);//char sp[7];
char m3[2]; // CRC16?
char m4[2]; // always AA 08
FSkip(16);
char lzw[len];
};
local int i;
for(i = 1; i < count; i++){
Binary b(index[i].loc - index[i-1].loc - 40);
}
if(count>0){
Binary b(FileSize() - index[i-1].loc - 40);
}
} file;
x
(upper/lowercase as it was) in place of the original would have been better. For all we know the key could be part of the first line.