There is clearly not enough of information to get you the exact answer. However, I will try my best to demonstrate logic behind what happens. That in a way might give you an idea how to go about locating the encryption function. I will assume you are using Windows, since it has not being stated otherwise.

The main logic of ordinary (and when I say ordinary, I mean there is no deliberate attempt to obfuscate or conceal normal execution flow) server/client TCP/IP network application boils down to the following:

It is obvious, Send and Receive on both client and server side is where all of the exchange happening. Usually, data will get encrypted by one side (client or server). Shortly thereafter, it will get transmitted to the other side. Other side will receive the data and decrypt it:

data -> encrypt -> send <----> receive -> decrypt -> data

In order to find a function that performs encryption, you need to locate send() function, determine location of the buffer being sent, go back and see what function sets(creates/modifies) that particular buffer location.

OllyDbg comes with some features that particularly useful in this situation. Firstly, we need to find all send() and receive() functions. We will search for intermodular calls. In disassembly window right click somewhere to bring up the following menu:

Select Search For -> All Intermodular Calls. It will bring up the window with all of the calls found. We specifically are looking for WS2_32.recv:

Set breakpoint on the function by pressing F2. Run the binary. Whenever WS2_32.recv breakpoint hits you can examine the stack window and determine location of the buffer:

Now, you need to step through and see whatever happens to that buffer, and data. I would assume one of the next function will try to decrypt it.

The logic is in reverse for sending stuff out.

Best of luck.

There is clearly not enough of information to get you the exact answer. However, I will try my best to demonstrate logic behind what happens. That in a way might give you an idea how to go about locating the encryption function. I will assume you are using Windows, since it has not being stated otherwise.

The main logic of ordinary (and when I say ordinary, I mean there is no deliberate attempt to obfuscate or conceal normal execution flow) server/client TCP/IP network application boils down to the following:

It is obvious, Send and Receive on both client and server side is where all of the exchange happening. Usually, data will get encrypted by one side (client or server). Shortly thereafter, it will get transmitted to the other side. Other side will receive the data and decrypt it:

data -> encrypt -> send <----> receive -> decrypt -> data

In order to find a function that performs encryption, you need to locate send() function, determine location of the buffer being sent, go back and see what function sets(creates/modifies) that particular buffer location.

OllyDbg comes with some features that particularly useful in this situation. Firstly, we need to find all send() and receive() functions. We will search for intermodular calls. In disassembly window right click somewhere to bring up the following menu:

Select Search For -> All Intermodular Calls. It will bring up the window with all of the calls found. We specifically are looking for WS2_32.recv:

Set breakpoint on the function by pressing F2. Run the binary. Whenever WS2_32.recv breakpoint hits you can examine the stack window and determine location of the buffer:

We can see, that our buffer is at location 0x001CFBB0 with the size of the buffer at 0x80, which is 128 bytes. Let's go to to the buffer location by pressing Ctrl + G and typing location. We get our buffer:

Now, you need to step through and see whatever happens to that buffer, and data. I would assume one of the next function will try to decrypt it.

The logic is in reverse(encrypt -> send) for sending stuff out.

Best of luck.