extracting data from binary file

Question

I have a binary file and a text file of the corresponding data, and I know the location in the binary file where the data are contained. However, I am unable to determine in what manner the data are encoded. The data do not appear to be stored as float16, float32, float64, signed/unsigned int of various length, or char, based on analysis of the hexdump. Perhaps the section containing the data has been compressed by some algorithm, or perhaps the numbers are stored in a representation with which I am unfamiliar.

The human-readable data are as follows:

20.0,0.001
21.0,0.001
22.0,0.001
23.0,0.001
24.0,0.002
25.0,0.002
26.1,0.002
27.0,0.004
28.0,0.002
29.0,0.002
30.0,0.003
31.0,0.004
(etc)

I have 70 such lines, each containing two numbers of the precision shown in the table. The corresponding data section comprises 2312 hexadecimal digits, or 1156 bytes. There is a repeating pattern of characters ascending the ASCII table followed by @ signs (e.g., 4@ = '0x3440', 5@ = '0x3540', 6@ ='3640', etc.). This "motif" occurs every 16 bytes and there are 70 occurrences. One challenge is that although there is a monotonic progression in the ASCII value of the byte preceding the @ ('0x40'), it does not always increase (i.e., there are stationary points, sometimes followed by a jump that skips over one of the ASCII characters; for instance 4@ ... 5@ ... 6@ ...6@ ... 8@ ... 9@ ...). Based on the 16-byte periodicity of this motif, I assume each row in the human-readable table is represented in the binary file by 16 consecutive bytes. Thus, the total size of the binary data table would be 16*70 = 1120 bytes. The complete data table is 1156 bytes, so I further assume the unexplained 36 bytes contain header information. Indeed, I can account for most of the header: 15 bytes containing a data descriptor in plain text, 16 bytes of zeros which appears to serve as an offset, and one byte ('0x46' = 70 in decimal) which appears to encode the number of lines in the table.

My problems are currently the following:

• I do not know the "phase" of the 16 bytes of data (i.e., where one 16 byte segment ends and the next begins)
• I do not know how the human-readable numbers are encoded. They do not appear to be encoded in various float or integer representations, nor as ASCII characters.
• I do not know whether the data are compressed and if so how to determine the method of compression. If they are compressed, then the compression would be restricted to the data table itself since I can find plain text elsewhere in the binary file by running strings on it. Running file on the binary file simply reports that it contains "data".

An example of the hexadecimal contents (xxd dump) corresponding very nearly to the contents above and aligned to show the progression of '0x40' = @ that I mentioned previously is shown below. This table begins with the '0x46' (decimal 70), to which I alluded previously as being a part of the header and representing the number of lines in the data table. The hex digits before the colon simply give the offset of the line from the beginning of the file; the middle section shows eight bytes (16 hex digits) of data; the right part of the table shows the ASCII-printable values for the hex data (. signifies ASCII-unprintable values, mostly 0x00).

00000180: 4600 0000 0000 0000 0000 3440 0000 0000  F.........4@....
00000190: 0000 583f 0000 0000 0000 3540 0000 0000  ..X?......5@....
000001a0: 0000 503f c3f5 285c 8f02 3640 0000 0000  ..P?..(\..6@....
000001b0: 0000 553f 3e0a d7a3 70fd 3640 0000 0000  ..U?>...p.6@....
000001c0: 0000 453f 0000 0000 0000 3840 0000 0000  ..E?......8@....
000001d0: 0040 5d3f 85eb 51b8 1e05 3940 0000 0000  .@]?..Q...9@....
000001e0: 0000 5e3f cdcc cccc cc0c 3a40 0000 0000  ..^?......:@....
000001f0: 0000 633f f628 5c8f c2f5 3a40 0000 0000  ..c?.(\...:@....
00000200: 00c0 703f 3e0a d7a3 70fd 3b40 0000 0000  ..p?>...p.;@....
00000210: 0000 5a3f 0000 0000 0000 3d40 0000 0000  ..Z?......=@....
00000220: 0020 613f 48e1 7a14 ae07 3e40 0000 0000  . a?H.z...>@....
00000230: 00c0 643f c3f5 285c 8f02 3f40 0000 0000  ..d?..(\..?@....

I would like advice on how to proceed with this particular problem, mainly determining whether compression is a factor and how the numbers are represented in the binary file, since I can thus far detect no obvious correspondence between the rows of human-readable data and the repeating 16-byte motifs that I described.

Answer 1

This data actually is in standard x86 double (8 byte) representation, with the first 4 byte something else (46 00 00 00 probably being 70 as you said) and the data starting at offset 0x184 in the binary file.

What probably confused you is the fact that the human-readable data is rounded, and the binary file has a better precision. So, for example, your 23.0 is actually 22.99, which is why 3640 occurs twice, and the next value, 24.00, has 38 40.

When I convert your hex back to binary (edit out the offset and character dump, then xxd -r -p < x.hex > x.bin), then run the following perl program over it:

open(F, "<$ARGV[0]");
binmode F;

seek(F, 4, 0);

while (read(F, $dbl, 8)) {
    read(F, $dbl2, 8);
    printf("%15.10f  %15.10f   ", unpack("d", $dbl), unpack("d", $dbl2));
    printf("%4.1f  %5.3f\n", unpack("d", $dbl), unpack("d", $dbl2));
}

I get this:

20.0000000000     0.0014648438   20.0  0.001
21.0000000000     0.0009765625   21.0  0.001
22.0100000000     0.0012817383   22.0  0.001
22.9900000000     0.0006408691   23.0  0.001
24.0000000000     0.0017852783   24.0  0.002
25.0200000000     0.0018310547   25.0  0.002
26.0500000000     0.0023193359   26.1  0.002
26.9600000000     0.0040893555   27.0  0.004
27.9900000000     0.0015869141   28.0  0.002
29.0000000000     0.0020904541   29.0  0.002
30.0300000000     0.0025329590   30.0  0.003
31.0100000000     0.0000000000   31.0  0.000

As you see, the right 2 columns are the numbers in your precision and match (except the last 0.004 which isn't in your file), while the first 2 columns are in high precision and, sometimes, have a value that is slightly less than the next integer.

I found this by converting 20.0 to double and checking the hex:

perl -e "print pack('d', 20.0);" | xxd
0000000: 0000 0000 0000 3440                      ......4@

then just starting at the fitting offset in your file and converting stuff back.