We are trying to extract sonar data from a proprietary sonar log file format, which we have working well in most cases. The data is stored in binary, with a variable length header containing information such as depth, geographic location etc, and a byte array of raw sonar returns. Most of the examples we have of this format have a simple byte array, where each byte is a raw sonar return level. Creating a bitmap using these values with a monochrome 8 bit palette will give you a visual depiction of the sonar, as you might see on an echo sounder screen. Recently we have been given some older files that do do not store the sonar ping data in the same way. These files were apparently recorded on a "low quality" setting, and some sort of compression has been applied. The headers are readable as before.
We know that the files we cannot read have a "chunk length" of 1600 bytes (this was configurable on the unit). Chunk length refers to the size of the ping array + the header. In this case, the header is 28 bytes long - but the gap until the next row is only 372 bytes. This suggests that the compression is fixed, and produces 4 final bytes per file encoded byte.
Here is an example of one chunk of this compressed byte array. The first two bytes of the example are not part of the byte array, but might be significant. On uncompressed versions of this file, this number is -1. On the compressed versions, it has a number that varies per ping.
Any advice or pointers how to proceed would be very warmly received. Just in case it helps at all, its very typical for ping to contain a cluster of non-zero values at the beginning (noise from the surface), then a lot of zero or very low values, then some more significant values from the return signal of the bottom.
67 7F 42 46 3D 35 3C 53 3B 40 80 40 36 41 3A
53 3F 3F 40 40 80 40 40 81 40 47 40 40 40 3D 51
3E 40 40 40 40 40 40 81 40 89 40 3B 43 3F 40 40
40 40 40 80 3E 46 3F 41 3E 41 40 40 40 40 80 40
40 40 40 40 80 40 81 40 80 40 95 3F 42 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 40 40 3F 46
40 40 40 41 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 40 40 80 40 40 40 40 40 40 40 40 84
40 90 40 9F 40 3F 40 40 40 40 40 40 40 40 40 40
40 40 40 40 40 40 81 40 40 40 40 40 40 40 80 40
40 40 40 81 40 81 40 80 40 44 40 40 40 40 80 40
40 40 80 40 40 80 40 80 40 40 40 81 40 40 40 82
40 40 40 80 40 81 40 83 40 80 40 40 83 40 80 40
40 80 40 80 40 80 40 40 80 40 88 40 84 40 E6 40
40 40 40 40 40 40 83 40 80 40 40 80 40 40 40 40
80 40 40 40 82 40 40 40 40 40 40 80 40 40 40 80
40 40 85 40 81 40 40 81 40 81 40 82 40 88 40 40
80 40 80 40 40 40 42 83 40 40 40 81 40 80 40 80
40 40 80 40 40 80 40 80 40 40 80 40 80 40 40 80
40 80 40 80 40 40 40 84 40 40 40 40 86 40 40 40
AC 40 93 40 FF E9 40 40 40 40 40 80 40 40 40 40
80 40 40 82 40 87 40 82 40 83 40 40 80 40 80 40
8B 40 40 8B 40 82 40 86 40 8D 40 91 40 93 40 81
40 86 40 88 40 42 00
Edit to add: "compressed" might not be the right term for this (the file snippet above certainly doesn't look compressed with all those similar values). I guess it might be just some type of encoding, and is probably lossy - which is why the option is referred to as "low quality". It's also possible that just two bytes per byte are required here - we are not 100% sure. It's nothing as obvious as nibbles though. Also, this would have been recorded on old hardware units without a lot of processing power, in real time. So I doubt its anything fancy.
Edit to add further information and clarification
Just to clear up some confusion (my fault) and answer sukminders questions:
- My bad - I did not remove the trailing 34 bytes from my original example - I have now done so, and those 34 bytes were the next header and first 6 bytes of the next ping. I have updated the original bytes.
A bit more about the file format:
The file starts with 8 bytes that are not repeated, contained within this header is a short that indicates the row length of the file.
The header for each row can be variable in length - there is a bit mask as one of the first fields in the header, which indicates which fields to read. In this particular file, the headers are 28 bytes - but can in fact vary row to row as data is included.
The ping data size (in file bytes) is the file-level row-size minus the header length.
We know we have the header sorted, as we get legitimate latitude, longitude, time (ms offset) and depth information. We can plot a trail using the data in this file.
I am going to take back the idea that we think these bytes expand to 1600 bytes - we are not sure enough, and that could be a red herring. We do "know" that the observed ping is larger that 372 bytes, and that the file was recorded at "low quality".
We have access to a viewing app that shows this file, but only as an image on screen. We cannot get a raw byte output for comparison. The ping in the example above is very typical - the big cluster of 0x40 and 0x80 values in the middle are likely to be zero or near zero values.
Files that exhibit this encoding have the value 1024 (decimal) in the 8 byte file header, where files that do not exhibit this encoding have a zero.
I think some sort of run-length encoding sounds very likely - but nothing obvious enough for me to work out! Happy to supply files and viewer app (freeware - not ours) to anyone interested.