2

I'm reading gzip format specification, trying to understand byte-to-byte the following minimal example (generated using echo -n | gzip > /tmp/a.gz):

00000000  1f 8b 08 00 70 3c b4 5d  00 03 03 00 00 00 00 00  |....p<.]........|
00000010  00 00 00 00                                       |....|
00000014

I managed to map most of the bytes, but the "03 00" is something I cannot interpret. I wrote a Python script to fuzz different values for the "03" byte, but nothing other than that byte is allowed:

[15:13:05]>>> import subprocess
[15:13:08]>>> results = {}                  
[15:13:10]>>> for i in range(256): results[i] = subprocess.Popen(f'''echo '1f8b0800703cb45d0003{hex(i)[2:]:0>2}000000000000000000'  | xxd -r -p |  zcat''', shell=True, stderr=subprocess.PIPE).stderr.read()
... 
Pp0�` ��@�X▒�x8�h(�H�[15:13:25]>>> 
[15:13:26]>>> results2 = {value: list(k for k in results if results[k] == value) for value in results.values()}
[15:13:31]>>> results2
{b'\ngzip: stdin: invalid compressed data--format violated\n': [0, 1, 2, 6, 7, 8, 9, 10, 14, 15, 16, 17, 18, 22, 23, 24, 25, 26, 30, 31, 32, 33, 34, 38, 39, 40, 41, 42, 46, 47, 48, 49, 50, 54, 55, 56, 57, 58, 62, 63, 64, 65, 66, 70, 71, 72, 73, 74, 78, 79, 80, 81, 82, 86, 87, 88, 89, 90, 94, 95, 96, 97, 98, 102, 103, 104, 105, 106, 110, 111, 112, 113, 114, 118, 119, 120, 121, 122, 126, 127, 128, 129, 130, 134, 135, 136, 137, 138, 142, 143, 144, 145, 146, 150, 151, 152, 153, 154, 158, 159, 160, 161, 162, 166, 167, 168, 169, 170, 174, 175, 176, 177, 178, 182, 183, 184, 185, 186, 190, 191, 192, 193, 194, 198, 199, 200, 201, 202, 206, 207, 208, 209, 210, 214, 215, 216, 217, 218, 222, 223, 224, 225, 226, 230, 231, 232, 233, 234, 238, 239, 240, 241, 242, 244, 245, 246, 247, 248, 249, 250, 252, 253, 254, 255], b'': [3], b'\ngzip: stdin: unexpected end of file\n': [4, 5, 11, 12, 13, 19, 20, 21, 27, 28, 29, 35, 36, 37, 43, 44, 45, 51, 52, 53, 59, 60, 61, 67, 68, 69, 75, 76, 77, 83, 84, 85, 91, 92, 93, 99, 100, 101, 107, 108, 109, 115, 116, 117, 123, 124, 125, 131, 132, 133, 139, 140, 141, 147, 148, 149, 155, 156, 157, 163, 164, 165, 171, 172, 173, 179, 180, 181, 187, 188, 189, 195, 196, 197, 203, 204, 205, 211, 212, 213, 219, 220, 221, 227, 228, 229, 235, 236, 237, 243, 251]}

What's this 0x03 0x00 and where in the gzip (or DEFLATE) documentation can I find it?

3

If you read zlib source code alongside the DEFLATE Compressed Data Format Specification you can find where they come from.

The bits in those bytes represent the start and end of the compressed stream.

Specifically they come from 2 relevant places in the source code -

// trees.c
// line 978

// _tr_flush_block 

send_bits(s, (STATIC_TREES<<1)+last, 3);

Where here last=1 and STATIC_TREES=1 (these correspond to BFINAL and BTYPE in the specification) This outputs 3 bits 1, 1, 0 into the compressed file.

These indicate that the compressed block uses the 'fixed tree' and is also the last block.

Then it indicates that it's reached the end of the block -

// trees.c
// line 1108

// compress_block

send_code( c, END_BLOCK, ltree )

The end-of-block code is 256 which for the fixed huffman tree corresponds to 7 zero bits. (See 3.2.6 in the DEFLATE specification.) Hence this outputs 7 bits 0, 0, 0, 0, 0, 0, 0 to the compressed file.

This gives us 10 bits in total -

1 1 0 0 0 0 0 0 0 0

Converting to bytes according the specification we get -

00000011 00000000, or
0x03, 0x00

These are the values that you are seeing.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.