I have files with binary data, the format description of them is very vague and incomplete. E.g., it states that records start with header byte, like (hex) FA, followed by datetime (accurate down to milliseconds) and other data fields, but no indication of field length, least significant bit (LSB) value, or even the byte endianness of record fields. Overall, the files should represent some sort of message log, and I need to decode them properly into meaningful data.
Given the vagueness, incompleteness and possible errors (see below) in format description, my only hope to achieve the goal is a table that I have. It's describing roughly what's in the binary files. E.g., I know that some field from a specific file must decode to a value near 2700, another field must be -8.77, etc. There's at most one record statement like that, per file.
I've first read this question, but I'm not sure which of those tools can help in my situation. So I've translated my input binary into text files, simply displaying the initial data in hex representation, all in one big string. Splitting it by header bytes yielded some weird picture where each record seemed to have different length in bytes. Further investigation has shown that there are more types of headers (I call them sub-headers) than stated in format description. Also the first 1-byte field seems to indicate how many internal 22-byte blocks of data a record additionally has. This first field is out of place - it should've been datetime, judging by the format description. So, it's not that accurate/trustworthy, but at least it pushed me (seemingly) in the right direction.
I'm totally new to reverse engineering, so my questions may be rather bad, but please bear with me:
Is my task even possible to do, given the described situation?
If it is, how should I try and find a decoding method? What tools could help find correct field length, LSB and semantic (i.e., which data field is which, as I don't trust that format description too much anymore)?
EDIT: Additional information on findings
Here are some examples of internal 22-byte blocks. One of the records has 7 blocks:
0018001E030825411C004303076D000D230000013802 0018002B020B56010C001C030011000D22065D011601 0018003103166A0052001803000A000D22065D011601 00187F7301197440390017030779000D22065D011701 0018002B02230540390019030779000D22065D011E01 00187F7E032578004A0024030009000D22065D012B01 00180038012B2501040028030010000D230000013101
Prefixed by 'FE070F600710', where '07' says that there are 7 of them, and '0F600710' seems to be repeated in such prefixes throughout the file. Example of a different, 8-blocks record:
00187F4C020614414E0030030767000D230000012001 00187F4E000669414E0031030767000D230000012301 00180014030E3B004A0028030009000D230000012601 0018002B0110694042001B030778000D230000011C01 00187F620321080052001203000A000D230000011601 0018000B00254440390028030779000D230000012E02 0018001601345C00420018030008000D230000012401 0018002B013923404A0010030777000D230000011E01
As we can see, they all start with '0018', so that may be another sub-header, not data. That leaves us with exactly five 4-byte floats, or two 8-byte doubles and extra 4 bytes.
Some columns of '00's can be seen, '0D' seems to also repeat in a column pattern. There's a '03' that is also always present. If we think of them as additional delimiters, fields of 7, 1, 2, and 6 bytes can be guessed, which mostly isn't like some standard single- or double-precision floats. That's why in the initial statement I thought real numbers were coded as integers, with some unknown LSB.