0

I'm in the process of decoding the semantics of an unknown binary stream1. I've come to a point where I understand the overall structure: A tag byte, followed by a size field, and a blob of bytes of that size, e.g. 80 02 42 0a (type: 0x80, size: 2 bytes, data: 66, 10).

To help me drill deeper I'm looking for a tool that allows me to extract the individual chunks of data, run queries against those, and arbitrarily transform the results. I'm looking for simple queries (like "all packets with a specific tag"), but also need more complex filters (e.g. "distribution of all size values for packets with a specific tag").

Are there any tools that allow me to easily perform those queries, given the following requirement?

  • Needs to understand the overall stream structure.
  • Needs to have basic decoding capabilities (e.g. to extract base128-encoded varints as used by Protocol Buffers).
  • Needs customizable filters (e.g. by tag, size, or data).
  • [Optional] Ability to control display of data, like 64-bit FILETIME values.
  • [Optional] Ability to render graphs.
  • Needs to run on Windows (preferably free).

1 The data is streamed off of an embedded device. I have access to fully self-contained chunks of that data as binary files.

Improve this question
6
  • 1
    You might be able to use WireShark if you write a custom dissector. Kaitai Struct might be helpful as well. Here's a basic script that allows converting from a Kaitai Struct to a Wireshark LUA dissector: github.com/joushx/kaitai-to-wireshark
    – Dvd848
    Jan 17, 2020 at 20:33
  • @dvd: "Stream" was possibly a misleading term. I do have fully self-contained chunks of the stream available as files, so I'm not looking to analyze network traffic. I also played with Kaitai Struct today. It's a great tool, but it will only become useful to me, once I know the meaning of the data. I'm not there yet. Right now I'm looking at tag size <blob of data>. I don't understand <blob of data> yet, and I need help with that. To get there, I need to test hypotheses, by collecting statistical data.
    – IInspectable
    Jan 17, 2020 at 20:56
  • WireShark isn't limited to network traffic, it can dissect any data given the correct dissector (try using it to open a PNG file for example). However, if all else fails and you can't find a better tool or strategy, it might be useful to think of your data as a network stream nonetheless. Write a simple script to combine all of the different binary files to a pcap file, where the data of each packet is the contents of a single binary file. Then, create a basic dissector that is able to dissect "tag, size, blob". You can then use WireShark's/tshark's filtering capabilities for analysis/stats.
    – Dvd848
    Jan 17, 2020 at 21:31
  • @dvd: I got it eventually to work, writing a tool to convert the binary stream into a pcap file, and jotting up a Lua dissector. While helpful in finding patterns I wasn't even looking for, Wireshark ultimately turned out not to be suitable for the task. It really is meant for communication protocols, and that's really where it shines. I went on to write up a custom tool that does what I need. The advice was still helpful.
    – IInspectable
    Jan 19, 2020 at 19:44
  • Impressive! Glad to hear you were able to solve your problem.
    – Dvd848
    Jan 21, 2020 at 15:50

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.