Introduction to problem: I have a binary executable with an unknown network packet protocol. I want to reverse engineer this packet protocol. My current way of doing it is to send some data and step through the instructions in disassembly to try to figure out what the application is doing with this data, and gradually construct the correct protocol. This solution is extremely inefficient. So I want to automate at least a part of this process.
Assuming that my network receive function is:
int recv(SOCKET s, char * buf, int len, int flags);
What I want to do is to automate instruction tracking for all instructions reading the chunk of memory pointed by
char * buf
mov eax, [globalRecvBufferPointer] mov dl, [eax] cmp dl, 20h jz somewhere
In the example above, I want my automated tool to detect
mov dl, [eax] and
cmp dl, 20h instructions.
Adding a hardware r/w breakpoint to
char * buf lets me detect
mov dl, [eax] but not the other.
Another problem I can think of at this stage is when memory pointed by
char * buf is copied to stack or other memory locations.
Are there ready-made tools for this kind of operation? If not, are there tools where I can implement this idea?