What are some modern tools or techniques for attempting to reverse a binary protocol sent over UDP?

I've been trying to do research on this, and it seems like almost everything is obselete. Previous answers on this site and other resources mention a tool called Canape, which seems to no longer be maintained, or the tool netzob, which has not been updated for years. While I did manage to get it running, it doesn't seem to be terribly useful.

Are there any more modern tools which might show with visual aids, repeating patterns and similar things which would aid in reversing an unknown protocol?

  • I have some tools in the works which do much of what you're after. If you'd like to share some dumped messages (hex, 1 msg per line), I can give you a hand. – pythonpython Nov 16 '20 at 2:55

Check out Kaitai Struct.

Basically you define a binary format in YAML, and it will generate code to deserialize the format. There is a github repo with quite a few existing definitions, including one for UDP.

It also has a simple hex viewer that's nice for visualization - it will highlight the format sections/structures/fields as you move around. Or you can use the generated code to write your own programs for viewing, analysis, testing, etc.

Once you get comfortable with the definition language, it lets you focus more on figuring out the protocol or file format, and less on how to parse it.


It's may be worth checking out writing dissectors for Wireshark. This is a good tutorial using Lua.

  • However, keep in mind that Lua dissectors - all but the most simple ones - tend to slow down the whole capturing and display process. Been there, done that. But I like them for prototyping. – 0xC0000022L Dec 1 '20 at 8:49

Everyone would like a magic tool which makes it less tedious to reverse an unknown binary protocol. I hope we will see some things in the next 12 months which do this.

Difficult Problem

Previous static inference tools (Discoverer, Netzob) try to straddle both text and binary protocols. Text (think JSON payload) based protocols are generally much easier to reverse since as a human you can examine the data and understand at least what the types of the pieces are and the boundaries between fields. Binary protocols are ambiguous for both of types and field boundaries. Are these 4 bytes a IEEE Float or 2 unsigned shorts or 4 ASCII Chars?

Additionally, if you have access to the program / executable, you will eventually be able to reverse engineer the format from using IDA/Ghidra/Whatever. It may be tedious and take time, but you or a team of engineers can get it done.

So if you don't have access to the program, and all you have is some network traces or pcaps of the traffic, the odds are stacked against you. I think the reason you don't see modern tools (yet) is because it's hard and for less effort you can get more payoff either sticking to text protocol formats such as JSON / anything ASCII, or by working on binary program instrumentation and analysis to determine the format from the program itself.

So what can you try?

If you don't have access to the binary then hopefully you have multiple messages. The following assume you have a number of messages and have grouped them into pure formats meaning that the field layout for each message is exactly the same and only the values differ.

  1. Are all the messages the same length or different?

    • If the lengths are the same, then you have no variable length fields in the sample.

    • If the lengths are different, are they always in units of 2, or 4? or do you have both even and odd differences in length? This can tell you the unit size of the variable length portion of the message.

  2. Are there regions which take on a constant (same) value in all messages at a particular offset from the message start? These may help you identify where fields with changing values begin relative to message start.

  3. Are there any byte regions which correspond to message length?

    • If you have variable length messages in UDP and you find a length field, that's possibly an indication that the format is designed to work someplace besides UDP. UDP has a length field, so why send length twice?

    • If you don't see a length code within the body of message in UDP, that's interesting because whatever is consuming this message must need to know how many bytes to read, either from UDP length, or some other value in the message.

  4. Look for regions with known values such as a known IP address, or a known string.

  5. Split the messages into traffic sent and traffic received. Sometimes one format is used to send data, and another is used to acknowledge it.

  6. Is there some byte in all the messages which only takes on two or three values? Split your dataset into groups based on the value at that byte.

In terms of tools, for checksums here's something you can try: https://github.com/laurenlabell/checksum_finder

Background: https://dl.acm.org/doi/10.1145/3411506.3417599

  • 1
    While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review – sudhackar Nov 29 '20 at 13:02
  • I don't have enough points to make a comment. OP asked for specific modern tools, so a link is appropriate in this context. I hope some of the techniques I added are helpful to people working in this space. – pythonpython Nov 29 '20 at 20:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.