0

I'm in the process of decoding the semantics of an unknown binary stream1. I've come to a point where I understand the overall structure: A tag byte, followed by a size field, and a blob of bytes of that size, e.g. 80 02 42 0a (type: 0x80, size: 2 bytes, data: 66, 10).

To help me drill deeper I'm looking for a tool that allows me to extract the individual chunks of data, run queries against those, and arbitrarily transform the results. I'm looking for simple queries (like "all packets with a specific tag"), but also need more complex filters (e.g. "distribution of all size values for packets with a specific tag").

Are there any tools that allow me to easily perform those queries, given the following requirement?

  • Needs to understand the overall stream structure.
  • Needs to have basic decoding capabilities (e.g. to extract base128-encoded varints as used by Protocol Buffers).
  • Needs customizable filters (e.g. by tag, size, or data).
  • [Optional] Ability to control display of data, like 64-bit FILETIME values.
  • [Optional] Ability to render graphs.
  • Needs to run on Windows (preferably free).

1 The data is streamed off of an embedded device. I have access to fully self-contained chunks of that data as binary files.

Improve this question
6
  • 1
    You might be able to use WireShark if you write a custom dissector. Kaitai Struct might be helpful as well. Here's a basic script that allows converting from a Kaitai Struct to a Wireshark LUA dissector: github.com/joushx/kaitai-to-wireshark
    – Dvd848
    Jan 17 '20 at 20:33
  • @dvd: "Stream" was possibly a misleading term. I do have fully self-contained chunks of the stream available as files, so I'm not looking to analyze network traffic. I also played with Kaitai Struct today. It's a great tool, but it will only become useful to me, once I know the meaning of the data. I'm not there yet. Right now I'm looking at tag size <blob of data>. I don't understand <blob of data> yet, and I need help with that. To get there, I need to test hypotheses, by collecting statistical data.
    – IInspectable
    Jan 17 '20 at 20:56
  • WireShark isn't limited to network traffic, it can dissect any data given the correct dissector (try using it to open a PNG file for example). However, if all else fails and you can't find a better tool or strategy, it might be useful to think of your data as a network stream nonetheless. Write a simple script to combine all of the different binary files to a pcap file, where the data of each packet is the contents of a single binary file. Then, create a basic dissector that is able to dissect "tag, size, blob". You can then use WireShark's/tshark's filtering capabilities for analysis/stats.
    – Dvd848
    Jan 17 '20 at 21:31
  • @dvd: I got it eventually to work, writing a tool to convert the binary stream into a pcap file, and jotting up a Lua dissector. While helpful in finding patterns I wasn't even looking for, Wireshark ultimately turned out not to be suitable for the task. It really is meant for communication protocols, and that's really where it shines. I went on to write up a custom tool that does what I need. The advice was still helpful.
    – IInspectable
    Jan 19 '20 at 19:44
  • Impressive! Glad to hear you were able to solve your problem.
    – Dvd848
    Jan 21 '20 at 15:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.