Everyone would like a magic tool which makes it less tedious to reverse an unknown binary protocol. I hope we will see some things in the next 12 months which do this.
Difficult Problem
Previous static inference tools (Discoverer, Netzob) try to straddle both text and binary protocols. Text (think JSON payload) based protocols are generally much easier to reverse since as a human you can examine the data and understand at least what the types of the pieces are and the boundaries between fields. Binary protocols are ambiguous for both of types and field boundaries. Are these 4 bytes a IEEE Float or 2 unsigned shorts or 4 ASCII Chars?
Additionally, if you have access to the program / executable, you will eventually be able to reverse engineer the format from using IDA/Ghidra/Whatever. It may be tedious and take time, but you or a team of engineers can get it done.
So if you don't have access to the program, and all you have is some network traces or pcaps of the traffic, the odds are stacked against you. I think the reason you don't see modern tools (yet) is because it's hard and for less effort you can get more payoff either sticking to text protocol formats such as JSON / anything ASCII, or by working on binary program instrumentation and analysis to determine the format from the program itself.
So what can you try?
If you don't have access to the binary then hopefully you have multiple messages. The following assume you have a number of messages and have grouped them into pure formats meaning that the field layout for each message is exactly the same and only the values differ.
Are all the messages the same length or different?
If the lengths are the same, then you have no variable length fields in the sample.
If the lengths are different, are they always in units of 2, or 4? or do you have both even and odd differences in length? This can tell you the unit size of the variable length portion of the message.
Are there regions which take on a constant (same) value in all messages at a particular offset from the message start? These may help you identify where fields with changing values begin relative to message start.
Are there any byte regions which correspond to message length?
If you have variable length messages in UDP and you find a length field, that's possibly an indication that the format is designed to work someplace besides UDP. UDP has a length field, so why send length twice?
If you don't see a length code within the body of message in UDP, that's interesting because whatever is consuming this message must need to know how many bytes to read, either from UDP length, or some other value in the message.
Look for regions with known values such as a known IP address, or a known string.
Split the messages into traffic sent and traffic received. Sometimes one format is used to send data, and another is used to acknowledge it.
Is there some byte in all the messages which only takes on two or three values? Split your dataset into groups based on the value at that byte.
In terms of tools, for checksums here's something you can try: https://github.com/laurenlabell/checksum_finder
Background: https://dl.acm.org/doi/10.1145/3411506.3417599