Non standard LZ77 compression header

Question

I'm working on reversing the SJ4000 camera firmware but I found a problem unpacking it.

This is the header I found on the image:

00000000  42 43 4c 31 81 66 00 09  00 54 68 e0 00 2f 2b bf  |BCL1.f...Th../+.|

As you can see BCL1 is the header for 'Basic Compression Library' ( http://bcl.comli.eu/home-en.html ) using LZ77 algorithm but I can't decompress the image with it.

I built some files using BCL, compared to the firmware header and found this:

00000000  42 43 4c 31 81 66 00 09  00 54 68 e0 00 2f 2b bf  |BCL1.f...Th../+.| < FIRMWARE
00000000  42 43 4c 31 00 00 00 09  00 00 4f 88 99 7f 45 4c  |BCL1......O...EL| < LZ
00000000  42 43 4c 31 00 00 00 02  00 00 4f 88 20 03 06 90  |BCL1......O. ...| < HUFFMAN
00000000  42 43 4c 31 00 00 00 01  00 00 4f 88 99 7f 45 4c  |BCL1......O...EL| < RLE
00000000  42 43 4c 31 00 00 00 0a  00 00 4f 88 56 01 64 9f  |BCL1......O.V.d.| < SF
00000000  42 43 4c 31 00 00 00 03  00 00 4f 88 00 7f 45 4c  |BCL1......O...EL| < RICE8
00000000  42 43 4c 31 00 00 00 04  00 00 4f 88 00 45 7f 46  |BCL1......O..E.F| < RICE16
00000000  42 43 4c 31 00 00 00 05  00 00 4f 88 00 46 4c 45  |BCL1......O..FLE| < RICE32

According to this the compression algorithm is LZ77 and it follows the same structure except for 2 bytes.

42 43 4c 31 < Magic Number
81 66 00 09 < unknown 2 bytes + 2 standard bytes 
00 54 68 e0 < Original Size
00 2f 2b bf < Compressed Size

Any idea what these 2 bytes mean?

EDIT: I tried to edit that 2 bytes and override them with 00 00 so the header matches the standard. After that tried to uncompress it with BCL LZ77 and it prompts a segmentation fault:

LZ77 decompress FW96655A_ZERO.bin to test...
Input file: 3091395 bytes
Output file: 5531872 bytes
Segmentation fault

Checking the lenght bytes I got the following result:

0x005468E0 > Big Endian Long: 5531872
0x002F2BBF > Big Endian Long: 3091391

As you can see for the compressed data length there is a difference of 4 bytes that might be causing the Seg. fault.

Answer 1

I don't know for sure but can think of two possibilities.

1. One is that it's simply a coding error and only the low half of the 32-bit register was initialized when the file was created. If that's the case, then simply zeroing those two bytes should result in a successful decompression using the standard tool.

2. The other is that it's a (proprietary) modification to the normal LZ77 decompression, in which case, it's likely to be a minor enhancement rather than a completely new scheme.

All of the samples of that camera's firmware that I was easily able to locate on the internet use the standard 00 00 00 09 tag, which may indicate that the first possibility is the more likely.

After checking with the binary you mentioned, it became clear that there was simply an error in the bfc.c file. Specifically, around line 192 of bfc.c it says:

if( command == 'd' )
{
    /* Read header */
    algo = ReadWord32( f );  /* Dummy */
    algo = ReadWord32( f );
    outsize = ReadWord32( f );
    insize -= 12;
}

However, this is incorrect because it fails to read the infile size as written in the header. To quickly fix this, just change those lines to these and recompile:

if( command == 'd' )
{
    /* Read header */
    algo = ReadWord32( f );  /* Dummy */
    algo = ReadWord32( f );
    outsize = ReadWord32( f );
    ReadWord32( f );
    insize -= 16;
}

When I do that, there is no problem decompressing the code. To make the corresponding change to the output file (that is, compressing), change the code around line 364 of bfc.c from this:

/* Write output file */
WriteWord32( outsize, f );
fwrite( out, outsize, 1, f );
fclose( f );

to this:

/* Write output file */
fwrite( out, outsize, 1, f );
fclose( f );

I should probably mention the likely reason this change was done. In the original program, the file size can be used to determine the size, but in an undifferentiated array of bytes (i.e. no file system) as is often the case in ROM, it is useful to encode both input and output files sizes within the header.