2

I'm currently trying to reverse engineer the decryption algorithm for an old online game, using a chat message packet, as it contains text which is easily recognizable.

I used a packet sniffer to get the received packet (which is encrypted) from the server:

5e 00 09 32 3c c1 6e b5 ae be 90 4a 70 8f b7 3d
3a 81 c5 3a f9 11 a6 06 36 3d f3 68 a3 dd 72 a3
ff e1 c3 e9 12 28 d7 c5 a0 f1 ce 38 35 59 19 b6
85 90 76 23 42 af 50 6f 66 52 ec ad f9 da 61 3f
5d 09 ee 8d dd 9e ff ee 7d 27 0f 5c 1e df ba 30
de d0 c8 8c 1b 93 1d 53 66 13 98 ff 29 db

The first 4 bytes are known and unencrypted:

5E 00 is the size, here 94. 09 32 is the OPCODE as a big endian: here 2354. What follow is the encrypted payload.

Knowing the encrypted packet, I used Cheat Engine to set a breakpoint right after the winsocket receive function and then search the buffer where the packet is stored.

Important to note: Apparently, the buffer which stores the received (encrypted) packet will also store the decrypted one.

Checking what writes to this address results in exact one instruction. enter image description here

As the buffer which contains the encrypted packet also contains the decrypted one, I can say for sure that this has to be the function which decrypts the value.

The decrypted packet looks the following (last bits contain the chat message, somewhere in between the username is stored):

5e 00 09 32 00 00 00 00 4d 00 6e 60 00 00 00 00
06 0a 2c 00 00 00 74 00 65 00 73 00 74 00 63 00
68 00 61 00 72 00 31 00 00 00 91 12 ad 14 5e 75
00 00 cf 00 00 00 00 00 28 cd b8 69 fc f7 91 12
aa f1 07 4d 00 00 cf 00 00 00 00 00 10 74 65 73
74 5f 6d 65 73 73 61 67 65 5f 31 32 33 00

I then used Ghidra to inspect the specific instructions which handle the decryption part - this is where I am currently stuck and don't know how to proceed, as I don't have a deep understanding of this topic.

    undefined4 decrypt_package(int type,int param_2,int param_3,int param_4,int param_5)
    {
      undefined4 success;
      int counter;
      
      if ((param_3 == 0) || (param_4 == 0)) {
        s = 0;
      }
      else {
          // Old encryption function which only used byte ^ 255
          // I assume it's still in here, because the code wasn't cleaned up.
        if (type == 0) {
          counter = 0;
          while (counter < param_5) {
            *(byte *)(param_3 + counter) = *(byte *)(param_4 + counter) ^ 0xff;
            counter = counter + 1;
          }
        }
        else {  // Don't know when this is used - at least not for decryption
          if (type == 1) {
            param_2 = 0x48473c;
            counter = 0;
            while (counter < param_5) {
              *(byte *)(param_3 + counter) = *(byte *)(param_4 + counter) ^ (byte)((uint)param_2 >> 8);
              param_2 = ((uint)*(byte *)(param_3 + counter) + param_2) * 0x2ba339 + 0x2cad2b5;
              counter = counter + 1;
            }
          }
          else {
              // This block of code is called for the decryption part.
            if (type == 2) {
              counter = 0;
              while (counter < param_5) {
                *(byte *)(param_3 + counter) = *(byte *)(param_4 + counter) ^ (byte)((uint)param_2 >> 8)
                ;
                param_2 = ((uint)*(byte *)(param_3 + counter) + param_2) * 0x8e9a99 + 0x685b24;
                counter = counter + 1;
              }
            }
          }
        }
        success = 1;
      }
      return success;
    }

(Note for the type == 0: When the game first released, it only used a simple XOR with 255 as an encryption, this was apparently later discontinued)

From my understanding, the code does the following (assuming, param_2 is the buffer for the packet?):

  1. Loop over the buffer, as long as counter < param_5 (which I assume is the length of the payload to decrypt?)
  2. XOR the byte with a value (first byte that needs to be decrypted would be at position 5, so I assume that param_3 would be a kind of offset to skip the first 4 bytes.)
  3. Return a bool (don't know why it is declared as undefined) whether the decryption worked or not.

Below I also posted the assembly code for this block of code (starting at the counter = 0 in the while loop for type == 2) enter image description here


As said in the beginning, I am not experienced when it comes to this kind of things.

I therefore would like some additional information on how I would continue from here on, so that I could implement the decryption in my programming language of choice - what exactly is the code block above doing?

I hope I included all information needed.

Thanks for reading (and helping)! :)

1
  • This is unrelated to the reversal of the function itself but if your final goal is being able to decrypt packets, it's more practical to inject a dll that calls hooks recv() & send() and calls the decryption function.
    – Luppo
    Apr 13 at 3:21
2

From type == 0 you see that it writes to param_3 and reads from param_4. So those parameters must be pointers to output and input buffers respectively. In fact, if you change them to correct type (byte *), then I believe Ghidra should be able to simplify the rest of the code to easily understandable form.

The part that modifies param_2 is a simple pseudo-RNG. Note the classic trick of getting rid of the not-very-random lower bits of its output by shifting right.

Also, I didn't check this myself, but I bet types 1 and 2 do the opposite of each other. That is, one of them encrypts and the other decrypts.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.