Reversing network protocol

Question

I have a DVR that sends video over Ethernet using its own propriety TCP protocol. I want to write a VLC module to view the video, rather than the supplied DxClient.exe. I have captured traffic in wireshark and attempted to reverse engineer the client with IDA Pro, from what I can tell the client does some kind of handshake authentication, the DVR then sends 2 network packets (always 1514 bytes long), the client sends a TCP ACK and 2 more packets are transmitted, etc.etc... forever. From what I can tell the client uses Microsoft's AVIFIL32 library to decompress the packets to what essentially become AVI file frames.

The problem is I don't understand how these frames are encoded or if they even are AVI frames. Can anyone help me, here is the data payload from 2 packets:

http://pastebin.com/2VDu2Tc2

http://pastebin.com/L3Zi3VqU

Answer 1

You can try Netzob tool. This is a tool dedicated to reverse engineering protocols.

• You can download it here : http://www.netzob.org/
• A great example w/ ZeroAccess C&C protocol : http://www.netzob.org/documentations/presentations/netzob_29C3_2012.pdf

You can also take a look at CANAPE : http://www.contextis.com/research/tools/canape/

Answer 2

I can't give you a specific solution, though I can tell you a tool to make reverse engineering a protocol easier.

Scapy is a python packet manipulation tool. One of the problems you have is, that wireshark doesn't know those packets. With Scapy its very easy to build and dissect strange/own packets. This will definetly help when you start to reverse engineer with IDA how a packet is build.

Here is an example of a UDP Layer definition:

class UDP(Packet):
    name = "UDP"
    fields_desc = [ ShortEnumField("sport", 53, UDP_SERVICES),
                    ShortEnumField("dport", 53, UDP_SERVICES),
                    ShortField("len", None),
                    XShortField("chksum", None), ]

You can create your own layer now and try to make sense of the packet data by reverse engineering the packet build process with IDA and rebuild the protocol with scapy.

Answer 3

From what I have recently gathered DxClient is designed as a client for DVR Netview technology. Just by looking at functionality of the DxClient, it is clear that it is more then just binary transfer of AVI formatted stream. I think, it is safe to assume, that rather proprietary transfer and control protocol is used. 2 frames that you provided is just not enough to get you some help on it. I think you should try to to focus on reversing the client. With enough time spent, you should be able to drill down to how each frame is constructed.

I would recommend you getting familiar with general principals of network protocol reverse engineering. An article blog An Overview of Protocol Reverse-Engineering is a great place to start.