I have several applications that interact with the network, but the data is encrypted. I set breakpoints on send ws2_32.dll and found the correct buffer in which the encrypted data was located (then I checked the call stack to find the encryption function used), but this did not lead to anything good, for example, breakpoints for reading and entries in. If you have information describing the process of searching for packages in programs or in examples / books. I'll be happy
In some games I have found the encryption on, it has been before the send call e.g. a function that calls send sometimes 2-3 calls up.
For recv it is decrypting after but some first decrypted part of it for a header to look up the size/id.
Others had compressed packets (zlib) which took a little bit of figuring.
A trick to see if is possibly compressed or encrypted well is try to compress the data (if large enough) if it does not get any meaningful compression then it is already compressed or encrypted enough to look random.
So the application might call recv or some other func to get data in. Then it may, decrypt some identifier or length e.g. Packet ID lookup size or size in header.
Try sending 20 A characters the first time Restarting the app and sending 23 A characters
Do a byte by byte difference and see what stands out as different.
Sometimes encryption key could be sent by the server first, in which case look for a single byte, block of bytes etc, that is copied somewhere on first recv, find references to that or memory (add a read breakpoint on it) and you might find your encrypt and/or decrypt routine.
Do multiple connections with same input data have the same data sent on the wire?
Can you send some reliable information and packet sniff it to see what it looks like?
For example if it has a login as the first packet and it is sent first straight after connection.
Send a long string of AAAAAAAAAAAAAAAAAAAAAAA And observe the data.
Does it look like
10 00 01 00 C9 C9 C9 C9 C9 C9 C9 C9 C9 C9 C9 C9 C9 C9 C9 C9
For example that would indicate something like xor. If xor observe many packets and look for bytes that you see often, e.g. 00 xor by something is the value.
Could also look more jumbled up or blocks of 4 bytes or some length. If you see repeating sequence e.g. C9 EF AD CC C9 EF AD CC and you know the input data was a long string of "A" for example, then its doing a pattern.
I have seen some implementations where something like blowfish was used, initially seeded but they did not handle padding the packet out fully to a multiple of the block length, so odd number of bytes showed up as plain text.
If the data is different each time from the very first send with no other recv, it could be a time based encryption based on when you connected, or that the client just picks random key and tells the server it.
For example if the first packet sent from client to server is 32 bytes of gibberish, its probably a key or an initialization vector to seed a block cipher or something...
One software was like this. CustomSend(data, length)
Which copied the bytes to a buffer, offset by packet header. Filled in the header with the length. Encrypted the entire content using another function (Look for loops before send or a function called before send containing loops could be a few levels up or deep, but you should follow what happens to the buffer of data, you need to do this backwards.
Following the code backwards can be tricky sometimes I find it easier to go forward if you can find something you know will submit some known data to send to this socket action/packet. Sometimes this is after a connect or after a first recv. Or after hitting enter in a chat box.
It is also not guaranteed that the same encryption or key is used for both encryption and decryption.
It could also be some kind of Asymmetric encryption, e.g. one key to encrypt but another to decrypt with each end only knowing the key needed to send or recv data with the other.
Or they could get crafty and encrypt the data one way for the first few packets and switch it up after negotiating keys to use.
If you happen to use IDA or maybe there is an alternative on Ghidra https://github.com/d3v1l401/FindCrypt-Ghidra or other tools? You can use something like FindCrypt to find known crypto algorithms https://www.hex-rays.com/blog/findcrypt/ which by looking at what uses/references them you might find the encrypt/decrypt that way.
Sounds fun hope you can make some progress on it.
The encryption routine is not likely to be in your call stack because it is not actually going to call the send function directly.
I would suggest you set a watch point inside the buffer to catch whenever the buffer is written to. Routines that write to the buffer will probably be reading from some input buffer. There could well be a few steps of adding headers/checksums but if you keep going backwards you may find a function which reads unencrypted data and writes encrypted data.