Analyzing a Login Process

Question

I am reverse engineering a program that uses a custom auth protocol. This is a basic outline of the process:

1. Client takes input for username and password, then sends username to the auth server.
2. Auth server calculates a public key and responds with it.
3. Client uses that key to calculate a session key, which is sent to the server.

The communication is all done via HTTP. My goal is to locate and analyze the functions responsible for calculating the session key. My most recent approach was to set a breakpoint on ws2_32.recv, begin the login process, then step through the execution from the recv breakpoint to analyze how the public key was being used to calculate the session key.

However, the breakpoint isn't triggered at the expected time. A breakpoint on ws2_32.send is triggered as expected. I'm pretty stumped on this. Is there something I've overlooked? Is there a better approach to locating the functions?

Answer 1

Your program might be using ws2_32.recvfrom instead of ws2_32.recv. Or alternatively, it could be using one of the lower-level Winsock functions, such as ws2_32.WSARecv, ws2_32.WSARecvEx, ws2_32.WSARecvFrom, or ws2_32.WSARecvMsg.

You may want to try one of the following approaches to determine what the program is doing:

• Use an API monitor such as API Monitor to determine what networking functions are called.
• Use Process Monitor to capture the network-receive events and double-click on those events to see the callstack.