Tracing to a function during debugging?

Question

I am trying to find an encryption algorithm for a game client, but I need to find the actual register/stack in the client before I can decrypt/encrypt the packets correctly (on my emulated server side). The problem is that I have no idea how to find the function.

In OllyDbg, is there a way I can trace during my debugging in order to find the function right after user input? The user input is a string converted to a char array, which is then encrypted in the executable. In this case, it would mean sending a packet to the game client, and then the game client decrypts the packet.

Answer 1

There is clearly not enough of information to get you the exact answer. However, I will try my best to demonstrate logic behind what happens. That in a way might give you an idea how to go about locating the encryption function. I will assume you are using Windows, since it has not being stated otherwise.

The main logic of ordinary (and when I say ordinary, I mean there is no deliberate attempt to obfuscate or conceal normal execution flow) server/client TCP/IP network application boils down to the following:

It is obvious, Send and Receive on both client and server side is where all of the exchange happening. Usually, data will get encrypted by one side (client or server). Shortly thereafter, it will get transmitted to the other side. Other side will receive the data and decrypt it:

data -> encrypt -> send <----> receive -> decrypt -> data

In order to find a function that performs encryption, you need to locate send() function, determine location of the buffer being sent, go back and see what function sets(creates/modifies) that particular buffer location.

OllyDbg comes with some features that particularly useful in this situation. Firstly, we need to find all send() and receive() functions. We will search for intermodular calls. In disassembly window right click somewhere to bring up the following menu:

Select Search For -> All Intermodular Calls. It will bring up the window with all of the calls found. We specifically are looking for WS2_32.recv:

Set breakpoint on the function by pressing F2. Run the binary. Whenever WS2_32.recv breakpoint hits you can examine the stack window and determine location of the buffer:

We can see, that our buffer is at location 0x001CFBB0 with the size of the buffer at 0x80, which is 128 bytes. Let's go to to the buffer location by pressing Ctrl + G and typing location. We get our buffer:

Now, you need to step through and see whatever happens to that buffer, and data. I would assume one of the next function will try to decrypt it.

The logic is in reverse(encrypt -> send) for sending stuff out.

Best of luck.

Answer 2

Your question is too broad. How is your user input taken? Try answering a few questions to narrow down the scope

GetWindowText() 
SendMessage(.....WM_GETTEXT...) LB_GETTEXT 
RichEdit 
WM_CHAR in a Loop 
scanf in a console ??
sscanf from a stram ??
Directx SendInput ??
opengl ??
touch recognition?? biometrics??  smart-card ?? voice commands

Once you narrow down to specifics you should next concentrate on narrowing down the message processing loop. Is it a classic while (TRUE) loop with switch cases for each message (Windows Message Pump) etc etc

Once the scope is narrowed down to a manageable size, several options exist to trace them. For example with:

• Windbg you can use ( wt watch and trace when you are on a function prologue ) it will execute the function and provide you a list of subfunctions
• with ollydbg you can use run trace feature or hit trace feature. Several plugins exist that enhance this feature
• for ida look for Trace Replayer documentation

I hope you are on windows. If you are on another OS then GDB also has certain tracing features available.

Answer 3

If the data is ever assigned to a register then you can try OllyStepNSearch to locate it. You can find it and more info here: http://blog.didierstevens.com/programs/ollystepnsearch/

Answer 4

there are a lot of tutorials which could help you do that.

I would recommend you to go through Lena's reversing tutorial.

Advice: Please do not use your knowledge to crack any software. Use it constructively.