I have been battling this infection I got that encrypts my files in 512 byte chunks with a friend. We have managed to find the Decryption function we think in IDA (the code is heavily obfuscated) from a user who paid for the decrypter. Below is the C dump of the encryption function:

int __stdcall sub_40C78E(int a1, int a2, int a3, int a4)
  int result;
  char v5; 
  int v6; 
  int v7; 
  int v8; 

  v7 = a1;
  v6 = a2;
  v5 = 0;
  result = 0;
  if ( a2 )
    v8 = a3;
      LOBYTE(v8) = v5 + v8;
      *(_BYTE *)v7 ^= v8;
      v5 = *(_BYTE *)v7++;
      v8 = __ROL__(a4 + v8, 8);
    while ( v6 );
    result = v8;
  return result;

My friend tried to simplify it or make sense of it and this is what he came up with:

int __stdcall sub_40C78E((_BYTE *)buffer, int nonce1, int nonce2)
  char v5; 
  int n; 
  int v8; 

  n = 0x400;    // It is a little bit confusing, because the length of block is 0x200 (rest of buffer is filled by 0).
                        // Only first 0x200 bytes are saved to a file for block CT0A.
  v5 = 0;
  v8 = nonce1;

      LOBYTE(v8) = v5 + LOBYTE(v8);
      *buffer ^= LOBYTE(v8);
      v5 = *buffer;
      v8 = __ROL__(nonce2 + v8, 8);
  while ( n );

return v8;       

Where nonce1 and nonce2 is suppose to represent some kind of key.

What we have found with this infection is that if you XOR the first byte of the cipher text with the plain text, you get a key byte you can use to get the first byte of every file back. Which makes sense with this function because the first time in the loop the key is added to 0, which means it is simply the key. But then this guy used some type of weird CFB type xor encryption where it uses the previous xor'ed byte with the key next.

I just don't quite understand the function and I was hoping somewhere here could perhaps simplify it more for me or explain it. I also know C#, and VB if anyone would know how to explain the function in these languages.

1 Answer 1


As you expected, it's doing a very simple XOR. The equivalent code looks something like this:

int mystery(char *buff, int bufsize, int nonce1, int nonce2)
  int result = 0;
  // ch is the next byte (character) in the buffer
  char ch = 0; 
  int count = bufsize; 
  char *ptr = buff; 
  int x;

  for (x = nonce1; count; --count)
      // this bit of trickery just replaces the low 8 bits
      // of x with the low 8 bits of (x+ch) neglecting carry, if any
      x = (x & ~0xff) | ((x+ch) & 0xff);
      // XOR the buffer with the calculated x value
      *ptr ^= x;
      // read in the next character into ch
      ch = *ptr++;
      // obfuscate by adding nonce2 
      x += nonce2;
      // if x = 0x12345678, this would make it 0x34567812
      // for 32-bit ints.  Just a rotate left of 8 bits.
      x = (x<<8) | ((x >>((sizeof(int)-1)*8) ) & 0xff);  
  result = x;
  // return the last calculated x which may be used to chain all
  // of the blocks together.  That is, the return value x is 
  // probably passed as nonce1 to encode the next block.
  return result;
  • Thank you so much for your answer. I had almost given up hope. But i do have another question if you don't mind. At first i was confused with this function. If it was a function called in a loop or if it did the loop encryption itself. Basically there is a 20 byte key in each decrypter, 4 bytes of it is used to generate a header, and the other bytes used for encryption. In this function above, do you think Nonce1 and nonce2 and more than 1 byte (as in a full key), or one byte (one byte from key). And do you think this encryption scheme is bruteforceable/Crackable like XOR? Thank you so much. May 2, 2014 at 15:28
  • It should be easy to reverse. In fact, you can use almost the exact same code, except change the line ch = *ptr++; to ch = *ptr++ ^ x; and call it with the same nonce values and that will effectively unscramble a previously scrambled buffer. As for your other question, please ask it as a separate question -- comments are too short to explain it, I think.
    – Edward
    May 2, 2014 at 16:23
  • Well as for the cracking, the nonce's are generated from the key, Of which i do not know as this code came from another victims decrypter. So do you think there would be anyway without knowing the key (nonce)? May 2, 2014 at 17:03
  • And as for the first question, i was wondering if you could tell from the function if the nonce's passed to the function seem like they are a single byte or more than one byte? i think i worded that right. Btw, thanks again May 2, 2014 at 17:04
  • @user3546043: each nonce is declared as an int so I presume that's 32 bits on your machine. As for the other question, I'll think about it, but that's definitely worthy of another question.
    – Edward
    May 2, 2014 at 17:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.