Debugger always paused in ntdll

Question

I'm a beginner at reverse engineering so any help would be appreciated. I apologise if a similar question has already been asked or if this is a dumb question, but I couldn't find anything.

I have tried using x32dbg, and ollydbg

I am trying to decrypt packets sent out from the game. I began by finding Winsock's SEND function. I sent messages in the games inbuilt chat functionality to send off packets, and traced my way backwards trying to find a point where the packet wasn't encrypted. I found myself in the ntdll.dll import.

I tried setting a breakpoint inside ntdll.dll, but the debugger instantly pauses on the breakpoint before I can send a packet. I try to press "run", but it just pauses itself again instantly. I am unsure if that means that ntdll.dll is being used all the time, or if it has anti-debugger.. or if i'm just doing things completely wrong.

Is there a way for me to only break on SEND packets? Any advice for this situation would be appreciated.

Answer 1

You started correctly by finding the Winsock SEND function, and breaking there. However, to trace back to the point where the packets are not encrypted requires tracing further back than ntdll.dll.

The debugger is pausing on RtlEnterCriticalSection because that function has many uses including those unrelated to sending packets, so it is called very frequently.

Instead of single-stepping from SEND, you might have more success if you examine the stack frames - the value at [EBP+4] is the return address for the calling function, and [EBP] is the previous value of EBP. If you examine that previous value of EBP, let's call it EBP', then that [EBP'+4] is the return address for the calling function, and [EBP'] is the previous value of EBP. If you follow the chain, you will end up in the program.

Once in the program, you can think about placing breakpoints around the memory locations where packets are stored before being sent. When you see them being accessed, you can see how the encryption is being performed.