Introduction to problem: I have a binary executable with an unknown network packet protocol. I want to reverse engineer this packet protocol. My current way of doing it is to send some data and step through the instructions in disassembly to try to figure out what the application is doing with this data, and gradually construct the correct protocol. This solution is extremely inefficient. So I want to automate at least a part of this process.
Assuming that my network receive function is:
int recv(SOCKET s, char * buf, int len, int flags);
What I want to do is to automate instruction tracking for all instructions reading the chunk of memory pointed by char * buf
mov eax, [globalRecvBufferPointer]
mov dl, [eax]
cmp dl, 20h
jz somewhere
In the example above, I want my automated tool to detect mov dl, [eax] and cmp dl, 20h instructions.
Adding a hardware r/w breakpoint to char * buf lets me detect mov dl, [eax] but not the other.
Another problem I can think of at this stage is when memory pointed by char * buf is copied to stack or other memory locations.
Are there ready-made tools for this kind of operation? If not, are there tools where I can implement this idea?
add dl, 1andcmp dl, 21h— there's always the option of decompiling the program and just look at the source, what's the issue with that?