5

We are trying to extract sonar data from a proprietary sonar log file format, which we have working well in most cases. The data is stored in binary, with a variable length header containing information such as depth, geographic location etc, and a byte array of raw sonar returns. Most of the examples we have of this format have a simple byte array, where each byte is a raw sonar return level. Creating a bitmap using these values with a monochrome 8 bit palette will give you a visual depiction of the sonar, as you might see on an echo sounder screen. Recently we have been given some older files that do do not store the sonar ping data in the same way. These files were apparently recorded on a "low quality" setting, and some sort of compression has been applied. The headers are readable as before.

We know that the files we cannot read have a "chunk length" of 1600 bytes (this was configurable on the unit). Chunk length refers to the size of the ping array + the header. In this case, the header is 28 bytes long - but the gap until the next row is only 372 bytes. This suggests that the compression is fixed, and produces 4 final bytes per file encoded byte.

Here is an example of one chunk of this compressed byte array. The first two bytes of the example are not part of the byte array, but might be significant. On uncompressed versions of this file, this number is -1. On the compressed versions, it has a number that varies per ping.

Any advice or pointers how to proceed would be very warmly received. Just in case it helps at all, its very typical for ping to contain a cluster of non-zero values at the beginning (noise from the surface), then a lot of zero or very low values, then some more significant values from the return signal of the bottom.

67 7F 42 46 3D 35 3C 53 3B 40 80 40 36 41 3A
53 3F 3F 40 40 80 40 40 81 40 47 40 40 40 3D 51
3E 40 40 40 40 40 40 81 40 89 40 3B 43 3F 40 40
40 40 40 80 3E 46 3F 41 3E 41 40 40 40 40 80 40
40 40 40 40 80 40 81 40 80 40 95 3F 42 40 40 40
40 40 40 40 40 40 40 40 40 40 40 40 40 40 3F 46
40 40 40 41 40 40 40 40 40 40 40 40 40 40 40 40
40 40 40 40 40 40 80 40 40 40 40 40 40 40 40 84
40 90 40 9F 40 3F 40 40 40 40 40 40 40 40 40 40
40 40 40 40 40 40 81 40 40 40 40 40 40 40 80 40
40 40 40 81 40 81 40 80 40 44 40 40 40 40 80 40
40 40 80 40 40 80 40 80 40 40 40 81 40 40 40 82
40 40 40 80 40 81 40 83 40 80 40 40 83 40 80 40
40 80 40 80 40 80 40 40 80 40 88 40 84 40 E6 40
40 40 40 40 40 40 83 40 80 40 40 80 40 40 40 40
80 40 40 40 82 40 40 40 40 40 40 80 40 40 40 80
40 40 85 40 81 40 40 81 40 81 40 82 40 88 40 40
80 40 80 40 40 40 42 83 40 40 40 81 40 80 40 80
40 40 80 40 40 80 40 80 40 40 80 40 80 40 40 80
40 80 40 80 40 40 40 84 40 40 40 40 86 40 40 40
AC 40 93 40 FF E9 40 40 40 40 40 80 40 40 40 40
80 40 40 82 40 87 40 82 40 83 40 40 80 40 80 40
8B 40 40 8B 40 82 40 86 40 8D 40 91 40 93 40 81
40 86 40 88 40 42 00 

Edit to add: "compressed" might not be the right term for this (the file snippet above certainly doesn't look compressed with all those similar values). I guess it might be just some type of encoding, and is probably lossy - which is why the option is referred to as "low quality". It's also possible that just two bytes per byte are required here - we are not 100% sure. It's nothing as obvious as nibbles though. Also, this would have been recorded on old hardware units without a lot of processing power, in real time. So I doubt its anything fancy.

Edit to add further information and clarification

Just to clear up some confusion (my fault) and answer sukminders questions:

  • My bad - I did not remove the trailing 34 bytes from my original example - I have now done so, and those 34 bytes were the next header and first 6 bytes of the next ping. I have updated the original bytes.

A bit more about the file format:

  • The file starts with 8 bytes that are not repeated, contained within this header is a short that indicates the row length of the file.

  • The header for each row can be variable in length - there is a bit mask as one of the first fields in the header, which indicates which fields to read. In this particular file, the headers are 28 bytes - but can in fact vary row to row as data is included.

  • The ping data size (in file bytes) is the file-level row-size minus the header length.

  • We know we have the header sorted, as we get legitimate latitude, longitude, time (ms offset) and depth information. We can plot a trail using the data in this file.

  • I am going to take back the idea that we think these bytes expand to 1600 bytes - we are not sure enough, and that could be a red herring. We do "know" that the observed ping is larger that 372 bytes, and that the file was recorded at "low quality".

  • We have access to a viewing app that shows this file, but only as an image on screen. We cannot get a raw byte output for comparison. The ping in the example above is very typical - the big cluster of 0x40 and 0x80 values in the middle are likely to be zero or near zero values.

  • Files that exhibit this encoding have the value 1024 (decimal) in the 8 byte file header, where files that do not exhibit this encoding have a zero.

I think some sort of run-length encoding sounds very likely - but nothing obvious enough for me to work out! Happy to supply files and viewer app (freeware - not ours) to anyone interested.

  • I'm trying to get this in my head; chunks are supposed to be 1600 bytes. You say 'the gap until the next row is 372'. How do can you tell when the next row starts? Does it have another header which you an identify? 372+28 = 400 bytes. Thus 1 chunk consists of four 400bytes sections? You posted a 'chunk' but it's 408 bytes which does not seem to match the 4 x 400 bytes. Any chance to elaborate a bit more. maybe in some sort of memory/byte map? Looking at the data (and plotting a quick histogram) it doesn't look anywhere close to random enough to be compressed. What makes you so sure it is? – ixje Sep 27 '13 at 15:05
  • OK I should clarify a bit. 1. I think you are right - as I mentioned in the edit to my OP, I think compression is the wrong term here. The bytes are encoded somehow, such that each byte in the file is containing data for 2 or more final bytes. The "1600" reference is what the recorder of the file told me he set the "ping size" to when he made the recording. For the files with normally encoded ping data, this "ping size" field actually refers to a whole row, which includes header + ping. Header length is variable, and the ping takes the rest. In this case, the whole row takes 400 bytes. – Matt Sep 27 '13 at 15:20
  • I have edited the OP to remove a few additional bytes from the end - included in error. There are 372 ping bytes, plus 2 bytes at the beginning that I left in from the header, because I thought they might be significant. Yes - I recognized the next row from the header. I have a feeling this might be a 2-1 encoding, so the 1600 thing is not set in stone. Hope that clarifies (a bit!). – Matt Sep 27 '13 at 15:22
  • Do not know if I have the time for it, but would be interesting to look at the files / app. – Runium Sep 29 '13 at 7:51
  • @Sukminder - how do I get the files to you? – Matt Sep 29 '13 at 8:04
4

Not an answer, but this became to messy as a comment.

From what you write I interpret it to be:

67 7F                                            < Last two bytes of header
                                                 +---
42 46 3D 35 3C 53 3B 40 80 40 36 41 3A 53 3F 3F  |
40 40 80 40 40 81 40 47 40 40 40 3D 51 3E 40 40  |
...                                              : 372 ping bytes
8B 40 82 40 86 40 8D 40 91 40 93 40 81 40 86 40  |
88 40 42 00                                      |
                                                 +---
                                                 +---
01 2C CB C9 97 41 B9 1E 55 41 05 EA CC 3F 0D 8E  |
53 41 73 00 00 00 ED 20 00 00 67 7F 42 46 3D 35  | What is this then? (34 bytes)
3C 53                                            |
                                                 +---

At first glimpse I also note that the sample data has this sequence repeated at top and end of data:

67 7F 42 46 3D 35 3C 53
  • How do you know the header is 28 bytes? Is the size hardcoded in the header?
  • Are the depth and location data sane throughout the file? Do you have approximately these values elsewhere?

It seems to me that before trying to do anything else one should make sure one decode the headers correctly. Knowing depth, from header, you should be able to recognize scatter in ping data from bottom as depth changes.

As there is a mayor use of bytes in the lower 0x40's and 0x80's it could indicate higher bits, e.g. 3, are some sort of repeat-count. If so the last 00 of the 372 could indicate some sort of end of data indicator. (00 are not elsewhere, but in the next chunk of bytes – the 34 at end).

As said, a bit hard when I'm not sure how the data should be mapped – and only having one sample …


If you respond to this comment with vital information it would be best to update question instead of commenting below this. (I might delete this as well, if I'm far off or doesn't look more at it …)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.