Others have provided clues about this particular protocol, so I'll just address the general mechanism of decoding a protocol. You have specified "wire" but this generic method can and has been applied to protocols over fiber and RF as well.
Formulate a plan for investigation
The best start is often not mentioned, but is quite important in practice. That step is to formulate a plan. What do you want to know about the protocol? What do you intend to do with the knowledge? What you know could include, as with your case, the name of the protocol and its purpose. It might also be any clues you can glean or infer. For example, you know that it's low latency audio with multiple channels. Do you have equipment that you can control and observe, or are you just passively listening? Are there regulatory agencies or standards bodies which might have documentation on the protocol? Have you looked for patents or patent applications? Are there protocols with similar purpose that you can read about?
Investigate the signal visually
The human eye and brain are remarkably good at pattern recognition, so a useful first step is to convert the signal of interest to a visual representation. For a wired 2-wire connection, I would usually start by using a simple multimeter across the line. That gives some rough indication of what voltages might be present. That's important for both personal safety and the safety of any other equipment you might attach. In this case, since it's over Ethernet cabling, and given the purpose, it's likely to be compatible with Ethernet voltages, and possibly using Ethernet voltages +/- 2.5V. So the next step I'd use would be to attach an oscilloscope. The simplest and most typical 'scope setup is simply a graph of voltage over time. This can give a lot of insight. For instance, are there discrete voltage levels used? If so, how many (2, 4, more?)? Does the signal appear to be packetized or is it more or less continuous? What is the shortest duration between changes in the signal? More advanced 'scopes have other useful functions, such as spectrum analysis and/or FFT and time and voltage measurement.
Attempt to modify the signal in known ways
If you have equipment that you can control, try changing just one thing to see how it affects the output signal. For example, try sending a 1kHz sine wave audio signal over a single channel. Now send the same signal over 2 channels. Try changing the audio frequency. Try changing the audio amplitude. All of these can lead you to some insight about the protocol.
See if you can capture the signal in digital form
If you can faithfully digitize the signal and store it as a file in your computer, then you have many many more resources available to you for investigation. You can do the spectrum analysis on your computer. You can try out theories about modulation type, channelization, and so forth all on the computer by using off-the-shelf or your own custom software. This is the fun part for most of us!
Test your theory
If you think you have figured out the protocol and if you have either equipment you can control or at least more samples you can observe, then test your theory with that data. See if your computer implementation of the protocol matches what you observe. Ideally, you'd also be able to build your own interface and pretend to be one end or the other of the protocol. In the case of audio gear, you're not likely to hurt anything if you're not quite correct. Other realms (such as vehicle engine controls) may very well require much more caution. See if you can test corner cases and error conditions.
Share your results
If you can, and if it's responsible to do so, consider sharing your results. Chances are that somebody out there somewhere also has the same interest. By sharing notes, you'll both make more progress than if either works in isolation. Many's the time that I've had great luck finding even a half-baked partial implementation on github or sourceforge or the like, saving me a ton of time. I try to share what I've found, too, in the same spirit.
Good luck, and have fun!