What is the best approach to reverse a custom TCP Application Layer?

Question

I have recorded a TCP Stream using Wireshark, so I have Client/Server Communication as a hex dump.

What is the best approach to reverse the custom application layer?

Some data seems to be ascii encoded, some data isn't.

There is no SSL encryption present.

What kind of structure can I expect/look out for in a custom protocoll.

What (free) tools can I use?

Answer 1

at first you need to gather all - and i mean ALL information you can get about everything around the paket producing software/device

some questions:

• how big are the pakets - are we talking about +-100 bytes or megabytes of unknown data
• what applications using that TCP data? automative, medicine, logging/tracing, control commanding,... what is the purpose of the communication?
• are multiple features inside those pakets? control-commands, logging data, reading data etc.
• have you the ability to change the paket content - with a vendors tool or something to invoke function A, B, C...
• what is the vendor/name of the device/software that produces the pakets
• example wireshark logs, at least a screenshot or text-dump of a part would be very helping

all this can give hints what is the content and how its stored/ciphered... there is no standard how the pakets should look like - its totaly open to the original developer

you need to be way more informative if you want to make progress

Answer 2

A useful tool for this is Scapy. You can import a pcap, and it's quite easy to add custom protocols.

https://scapy.readthedocs.io/en/latest/

https://scapy.readthedocs.io/en/latest/build_dissect.html

What you need to to is define the length of each field, and document their possible values (and their meaning).

Look up for any documentation that could help you in your quest. And send your own packets to see if it works !