I have a binary file and a text file of the corresponding data, and I know the location in the binary file where the data are contained. However, I am unable to determine in what manner the data are encoded. The data do not appear to be stored as float16, float32, float64, signed/unsigned int of various length, or char, based on analysis of the hexdump. Perhaps the section containing the data has been compressed by some algorithm, or perhaps the numbers are stored in a representation with which I am unfamiliar.
The human-readable data are as follows:
20.0,0.001 21.0,0.001 22.0,0.001 23.0,0.001 24.0,0.002 25.0,0.002 26.1,0.002 27.0,0.004 28.0,0.002 29.0,0.002 30.0,0.003 31.0,0.004 (etc)
I have 70 such lines, each containing two numbers of the precision shown in the table. The corresponding data section comprises 2312 hexadecimal digits, or 1156 bytes. There is a repeating pattern of characters ascending the ASCII table followed by @ signs (e.g., 4@ = '0x3440', 5@ = '0x3540', 6@ ='3640', etc.). This "motif" occurs every 16 bytes and there are 70 occurrences. One challenge is that although there is a monotonic progression in the ASCII value of the byte preceding the @ ('0x40'), it does not always increase (i.e., there are stationary points, sometimes followed by a jump that skips over one of the ASCII characters; for instance
4@ ... 5@ ... 6@ ...6@ ... 8@ ... 9@ ...). Based on the 16-byte periodicity of this motif, I assume each row in the human-readable table is represented in the binary file by 16 consecutive bytes. Thus, the total size of the binary data table would be 16*70 = 1120 bytes. The complete data table is 1156 bytes, so I further assume the unexplained 36 bytes contain header information. Indeed, I can account for most of the header: 15 bytes containing a data descriptor in plain text, 16 bytes of zeros which appears to serve as an offset, and one byte ('0x46' = 70 in decimal) which appears to encode the number of lines in the table.
My problems are currently the following:
- I do not know the "phase" of the 16 bytes of data (i.e., where one 16 byte segment ends and the next begins)
- I do not know how the human-readable numbers are encoded. They do not appear to be encoded in various float or integer representations, nor as ASCII characters.
- I do not know whether the data are compressed and if so how to determine the method of compression. If they are compressed, then the compression would be restricted to the data table itself since I can find plain text elsewhere in the binary file by running
stringson it. Running
fileon the binary file simply reports that it contains "data".
An example of the hexadecimal contents (
xxd dump) corresponding very nearly to the contents above and aligned to show the progression of '0x40' = @ that I mentioned previously is shown below. This table begins with the '0x46' (decimal 70), to which I alluded previously as being a part of the header and representing the number of lines in the data table. The hex digits before the colon simply give the offset of the line from the beginning of the file; the middle section shows eight bytes (16 hex digits) of data; the right part of the table shows the ASCII-printable values for the hex data (
. signifies ASCII-unprintable values, mostly
00000180: 4600 0000 0000 0000 0000 3440 0000 0000 F.........4@.... 00000190: 0000 583f 0000 0000 0000 3540 0000 0000 ..X?......5@.... 000001a0: 0000 503f c3f5 285c 8f02 3640 0000 0000 ..P?..(\..6@.... 000001b0: 0000 553f 3e0a d7a3 70fd 3640 0000 0000 ..U?>...p.6@.... 000001c0: 0000 453f 0000 0000 0000 3840 0000 0000 ..E?......8@.... 000001d0: 0040 5d3f 85eb 51b8 1e05 3940 0000 0000 .@]?..Q...9@.... 000001e0: 0000 5e3f cdcc cccc cc0c 3a40 0000 0000 ..^?......:@.... 000001f0: 0000 633f f628 5c8f c2f5 3a40 0000 0000 ..c?.(\...:@.... 00000200: 00c0 703f 3e0a d7a3 70fd 3b40 0000 0000 ..p?>...p.;@.... 00000210: 0000 5a3f 0000 0000 0000 3d40 0000 0000 ..Z?......=@.... 00000220: 0020 613f 48e1 7a14 ae07 3e40 0000 0000 . a?H.z...>@.... 00000230: 00c0 643f c3f5 285c 8f02 3f40 0000 0000 ..d?..(\..?@....
I would like advice on how to proceed with this particular problem, mainly determining whether compression is a factor and how the numbers are represented in the binary file, since I can thus far detect no obvious correspondence between the rows of human-readable data and the repeating 16-byte motifs that I described.