I am reverse engineering a program that uses a custom auth protocol. This is a basic outline of the process:
- Client takes input for username and password, then sends username to the auth server.
- Auth server calculates a public key and responds with it.
- Client uses that key to calculate a session key, which is sent to the server.
The communication is all done via HTTP. My goal is to locate and analyze the functions responsible for calculating the session key. My most recent approach was to set a breakpoint on
ws2_32.recv, begin the login process, then step through the execution from the recv breakpoint to analyze how the public key was being used to calculate the session key.
However, the breakpoint isn't triggered at the expected time. A breakpoint on
ws2_32.send is triggered as expected. I'm pretty stumped on this. Is there something I've overlooked? Is there a better approach to locating the functions?