2

I have a couple of streams that look pretty odd within a DICOM file. In a few words, DICOM is very close to what binary XML would look like. Almost all of the information from those DICOM files are straighforward and can be read and interpreted nicely with DCMTK and/or GDCM.

However there are two binary fields stored within the end of the file that looks like private encoded information. Since DICOM is mostly for interoperability in between system, vendors are actually storing there own internal file format within one of the field of the DICOM file (declared as private field, much like what would people in the TIFF world would do). In my past experience, the encoding was trivial (plain struct stored as binary), see here or here for example.

Now if I extract the binary blobs from the DICOM file (debian/jessie amd64), here is what I see :

$ gdcmraw -t 7101,1000 input.dcm file1000.gz
$ gdcmraw -t 7101,1002 input.dcm file1002.gz
$ file file1000.gz file1000.gz: gzip compressed data, max compression, from FAT filesystem (MS-DOS, OS/2, NT)

$ gunzip file1000.gz

gzip: file1000.gz: invalid compressed data--format violated

However gunzip is not capable of decompressing them. Could someone with more gzip knowledge please check if those files are actually gzip compressed ? It looks like it's possible to decompress them because in the medical industry we tend to re-use code whenever possible. For example a well known MRI vendor is also using gzip compressed stream to store its private file format, see here for example (full thread here).

The obfuscation should be pretty trivial too because it needs to pass medical industry clearance. From past experience, I've only seen byte reversing being used or simple incremental XOR.

I've uploaded come of the files here:

The image can be extracted nicely, so I suspect only a few extra private vendor information (metadata only) is stored within this field (MRI serial number...).

To be more specific, the fake gzip stream comes in pair eg (file1000.gz & file1002.gz were taken from the same DICOM file). From another DICOM file I found that the second fake gzip stream was (bitwise) identical to file1002.gz, so I only uploaded file1000_other1.gz (same goes for file1000_other2.gz, file1000_other3.gz and file1000_other4.gz). So maybe file1002.gz is a bit special here. Since I do not have physical access to the MRI workstation that produces those images, I can only do brute-force approach here.


Update: I did check that the files are not simply a deflate codestream with broken header using unz.py and runme (binwalk -X did not reveal anything either). So they are not direct simple gzip files.

Update2: I did try to read the stream backwards using this code but again this still does not look like a deflate stream.

Update3: So far, all streams I found have proper gzip header, and they all finish with 4 zeros (0) bytes, just like any valid gzip. I should be able to recover the file using the last 4 bytes since they are used to store a crc32 (as per gzip RFC).

Update4: Thanks to help here, I discover those private tags are actually slightly documented:

Table A.2.1.2.1.3-3 Private Elements for MR Scanner or MR Workstation Images
When exporting Marconi MR Scanner or MR Workstation images the following
private elements may be included.

Tag Name Value Representation
7101,0010 Private MR Creator Data element LO
7101,1000 MR Processing Field 1 OB
7101,1001 MR Processing Field 1 Length SL
7101,1002 MR Processing Field 2 OB
7101,1003 MR Processing Field 2 Length SL
7101,1004 Scan Duration SH
7101,1005 MR Processing Field 3 SH
7101,1006 MR Processing Field 4 SH

I did check that the length of the extracted fake-gzip actually match the value stored in the associated attribute (so length for attribute 7101,1000 match value stored in attribute 7101,1001, and length for attribute 7101,1002 matches value stored in attribute 7101,1003). For instance:

$ gdcmdump input3.dcm
[...]
(07a1,0010) ?? (LO) [ELSCINT1]                                    # 8,1 Private Creator
(07a1,1013) ?? (UL) 62940                                         # 4,1 ?
(7101,0000) ?? (UL) 24242                                         # 4,1 Generic Group Length
(7101,0010) ?? (LO) [Picker MR Private Group ]                    # 24,1 Private Creator
(7101,1000) ?? (OB) 1f\8b\08\00\00\00\00\00\02\00\14\5d\4b\8d\db\48\6e\3e\ec\53\4f\7b\28\c3\1e\ef\8c\d8\2d\86\e8\86\57\01\c9\d9\96\4a\bd\76\45\35\92\99\dc\33\e5\1b\08\78\04\25\94\93\04\f3\80\7a\03\fa\cd\34\02\40         # 10784,1 ?
(7101,1001) ?? (SL) 10784                                         # 4,1 ?
(7101,1002) ?? (OB) 1f\8b\08\00\00\00\00\00\02\00\14\3c\6d\8d\da\48\6d\9f\93\a1\31\e5\9c\2f\f6\6b\c1\48\44\d8\9e\26\67\ab\78\8d\1d\8a\6d\a0\80\6c\36\31\dd\95\b6\96\84\2f\13\90\a8\49\d8\0f\fe\fa\15\97\19\97\24\c0         # 13328,1 ?
(7101,1003) ?? (SL) 13328                                         # 4,1 ?
(7101,1004) ?? (SH) [00:48 ]                                      # 6,1 ?
(7101,1005) ?? (SH) [ECHO\CARDIAC]                                # 12,2 ?
(7101,1006) ?? (SH) [115204\4187\0\0 ]                            # 16,4 ?

Update5: DICOM can only stores even-bytes length as attribute. One fake-gzipped stream was actually padded to the next even length, but the actual length reported in 7101,1001 was odd (10765). I've updated file1000_other4.gz to have the proper length (the trailing bytes are not anymore 03 00 00 00, but 0B 03 00 00)

7
  • Various other tools all say the file is "damaged" or "broken". So: are you sure these files are complete? (Be aware that if the answer is "no" or "I don't know", then that is most likely the problem: your files are truncated.)
    – Jongware
    Jul 6, 2015 at 15:55
  • @Jongware no the files are not truncated, simply obfuscated.
    – tibar
    Jul 15, 2015 at 9:02
  • May I ask how did you get the gz files and on which OS ?!
    – yaspr
    Jul 16, 2015 at 21:33
  • 2
    you should upload complete files; there's a remote possibility something went wrong when extracting the streams. Also, do you have the sotware which produced them?
    – Igor Skochinsky
    Jul 17, 2015 at 18:48
  • 1
    Where do your filenames come from? Dicom Tag, or something else? There seems to be a specification (well, kind of) here on page 20, but 1004 and 1006 shouldn't be byte blobs, and 1008/1010 shouldn't exist. Jul 19, 2015 at 11:42

1 Answer 1

6
+25

The gzip headers are valid, but the deflate compressed data format is violated almost immediately, within less than ten bytes in for all of the files.

For all of the example files provided, the first deflate block is a dynamic block which has an oversubscribed code lengths code. That means that a Huffman code required to decode the code lengths for that block is itself invalid. This immediately halts the decompression, since no further progress can be made.

The last four bytes may not be what you think they are. The last four bytes of a valid gzip file is the length of the uncompressed data, modulo 232, in little-endian order. Those lengths do not seem correlated to the file sizes. For example file1010.gz's last four bytes are 0b 03 00 00. It's length is 10766. So 10766 bytes decompresses to 779 bytes? I don't think so. So the second-to-last four bytes are likely also not what you would expect there for a gzip stream, i.e. likely not a CRC.

The data after the header appears to be random in all cases (pretty flat histogram over 0..255), and is itself mostly incompressible, which is consistent with it being compressed data.

I tried decompressing from all bit offsets starting after the gzip header to the end, but no joy. This rules out some sort of header followed by valid deflate data.

6
  • I know the files are not directly handled by gzip (see orig post). I was hoping someone would do a finer analysis (probability distribution, entropy analysis...) about how close those files are from gzip compressed ones (esp. coming from one of the author of gzip).
    – tibar
    Jul 18, 2015 at 19:06
  • 1
    The data after the header appears to be random in all cases (pretty flat histogram over 0..255), which is consistent with it being compressed. Beyond that, there is no other useful analysis I can think of.
    – Mark Adler
    Jul 18, 2015 at 19:54
  • 1
    I checked what happens if you compress those .gz files again, with gzip. For 4 of them, the size gets increased slightly, just as you would expect (no more compression possible and headers added). For the other 2, the size decreases. This might mean it's not just gzip + obfuscation, but a different algorithm, which is slightly better than gzip in some and slightly worse in other cases. Jul 19, 2015 at 9:00
  • @MarkAdler: Maybe I'm misinterpreting the specification (ietf.org/rfc/rfc1951.txt), but as far as i can tell, byte 10 in each file is the first byte of the first compressed data block - 0x14 in all 6 files. Starting with the least significant bit (as specified in 3.1.1), this is a bit pattern of 00101000. I'd translate this to (3.2.3) BFINAL=0, BTYPE=01 (compressed with fixed huffman code). How do you conclude "the first deflate block is a dynamic block" ? Jul 19, 2015 at 10:40
  • Yes, you're misinterpreting the specification. You leave the byte as is, 00010100, but start at the bottom. So the lowest bit is 0, indicating that this is not the last block. Then the next two bits are 10 indicating a dynamic block.
    – Mark Adler
    Jul 19, 2015 at 14:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.