What kind of compression/obfuscation algorithm is this?

Question

I have a couple of streams that look pretty odd within a DICOM file. In a few words, DICOM is very close to what binary XML would look like. Almost all of the information from those DICOM files are straighforward and can be read and interpreted nicely with DCMTK and/or GDCM.

However there are two binary fields stored within the end of the file that looks like private encoded information. Since DICOM is mostly for interoperability in between system, vendors are actually storing there own internal file format within one of the field of the DICOM file (declared as private field, much like what would people in the TIFF world would do). In my past experience, the encoding was trivial (plain struct stored as binary), see here or here for example.

Now if I extract the binary blobs from the DICOM file (debian/jessie amd64), here is what I see :

$ gdcmraw -t 7101,1000 input.dcm file1000.gz
$ gdcmraw -t 7101,1002 input.dcm file1002.gz
$ file file1000.gz file1000.gz: gzip compressed data, max compression, from FAT filesystem (MS-DOS, OS/2, NT)

$ gunzip file1000.gz

gzip: file1000.gz: invalid compressed data--format violated

However gunzip is not capable of decompressing them. Could someone with more gzip knowledge please check if those files are actually gzip compressed ? It looks like it's possible to decompress them because in the medical industry we tend to re-use code whenever possible. For example a well known MRI vendor is also using gzip compressed stream to store its private file format, see here for example (full thread here).

The obfuscation should be pretty trivial too because it needs to pass medical industry clearance. From past experience, I've only seen byte reversing being used or simple incremental XOR.

I've uploaded come of the files here:

• https://github.com/malaterre/MRPicker

The image can be extracted nicely, so I suspect only a few extra private vendor information (metadata only) is stored within this field (MRI serial number...).

To be more specific, the fake gzip stream comes in pair eg (file1000.gz & file1002.gz were taken from the same DICOM file). From another DICOM file I found that the second fake gzip stream was (bitwise) identical to file1002.gz, so I only uploaded file1000_other1.gz (same goes for file1000_other2.gz, file1000_other3.gz and file1000_other4.gz). So maybe file1002.gz is a bit special here. Since I do not have physical access to the MRI workstation that produces those images, I can only do brute-force approach here.

---

Update: I did check that the files are not simply a deflate codestream with broken header using unz.py and runme (binwalk -X did not reveal anything either). So they are not direct simple gzip files.

Update2: I did try to read the stream backwards using this code but again this still does not look like a deflate stream.

Update3: So far, all streams I found have proper gzip header, and they all finish with 4 zeros (0) bytes, just like any valid gzip. I should be able to recover the file using the last 4 bytes since they are used to store a crc32 (as per gzip RFC).

Update4: Thanks to help here, I discover those private tags are actually slightly documented:

Table A.2.1.2.1.3-3 Private Elements for MR Scanner or MR Workstation Images
When exporting Marconi MR Scanner or MR Workstation images the following
private elements may be included.

Tag Name Value Representation
7101,0010 Private MR Creator Data element LO
7101,1000 MR Processing Field 1 OB
7101,1001 MR Processing Field 1 Length SL
7101,1002 MR Processing Field 2 OB
7101,1003 MR Processing Field 2 Length SL
7101,1004 Scan Duration SH
7101,1005 MR Processing Field 3 SH
7101,1006 MR Processing Field 4 SH

I did check that the length of the extracted fake-gzip actually match the value stored in the associated attribute (so length for attribute 7101,1000 match value stored in attribute 7101,1001, and length for attribute 7101,1002 matches value stored in attribute 7101,1003). For instance:

$ gdcmdump input3.dcm
[...]
(07a1,0010) ?? (LO) [ELSCINT1]                                    # 8,1 Private Creator
(07a1,1013) ?? (UL) 62940                                         # 4,1 ?
(7101,0000) ?? (UL) 24242                                         # 4,1 Generic Group Length
(7101,0010) ?? (LO) [Picker MR Private Group ]                    # 24,1 Private Creator
(7101,1000) ?? (OB) 1f\8b\08\00\00\00\00\00\02\00\14\5d\4b\8d\db\48\6e\3e\ec\53\4f\7b\28\c3\1e\ef\8c\d8\2d\86\e8\86\57\01\c9\d9\96\4a\bd\76\45\35\92\99\dc\33\e5\1b\08\78\04\25\94\93\04\f3\80\7a\03\fa\cd\34\02\40         # 10784,1 ?
(7101,1001) ?? (SL) 10784                                         # 4,1 ?
(7101,1002) ?? (OB) 1f\8b\08\00\00\00\00\00\02\00\14\3c\6d\8d\da\48\6d\9f\93\a1\31\e5\9c\2f\f6\6b\c1\48\44\d8\9e\26\67\ab\78\8d\1d\8a\6d\a0\80\6c\36\31\dd\95\b6\96\84\2f\13\90\a8\49\d8\0f\fe\fa\15\97\19\97\24\c0         # 13328,1 ?
(7101,1003) ?? (SL) 13328                                         # 4,1 ?
(7101,1004) ?? (SH) [00:48 ]                                      # 6,1 ?
(7101,1005) ?? (SH) [ECHO\CARDIAC]                                # 12,2 ?
(7101,1006) ?? (SH) [115204\4187\0\0 ]                            # 16,4 ?

Update5: DICOM can only stores even-bytes length as attribute. One fake-gzipped stream was actually padded to the next even length, but the actual length reported in 7101,1001 was odd (10765). I've updated file1000_other4.gz to have the proper length (the trailing bytes are not anymore 03 00 00 00, but 0B 03 00 00)

Answer 1

The gzip headers are valid, but the deflate compressed data format is violated almost immediately, within less than ten bytes in for all of the files.

For all of the example files provided, the first deflate block is a dynamic block which has an oversubscribed code lengths code. That means that a Huffman code required to decode the code lengths for that block is itself invalid. This immediately halts the decompression, since no further progress can be made.

The last four bytes may not be what you think they are. The last four bytes of a valid gzip file is the length of the uncompressed data, modulo 2³², in little-endian order. Those lengths do not seem correlated to the file sizes. For example file1010.gz's last four bytes are 0b 03 00 00. It's length is 10766. So 10766 bytes decompresses to 779 bytes? I don't think so. So the second-to-last four bytes are likely also not what you would expect there for a gzip stream, i.e. likely not a CRC.

The data after the header appears to be random in all cases (pretty flat histogram over 0..255), and is itself mostly incompressible, which is consistent with it being compressed data.

I tried decompressing from all bit offsets starting after the gzip header to the end, but no joy. This rules out some sort of header followed by valid deflate data.