IDA Pro C dump for Decryption function

Question

I have been battling this infection I got that encrypts my files in 512 byte chunks with a friend. We have managed to find the Decryption function we think in IDA (the code is heavily obfuscated) from a user who paid for the decrypter. Below is the C dump of the encryption function:

int __stdcall sub_40C78E(int a1, int a2, int a3, int a4)
{
  int result;
  char v5; 
  int v6; 
  int v7; 
  int v8; 

  v7 = a1;
  v6 = a2;
  v5 = 0;
  result = 0;
  if ( a2 )
  {
    v8 = a3;
    do
    {
      LOBYTE(v8) = v5 + v8;
      *(_BYTE *)v7 ^= v8;
      v5 = *(_BYTE *)v7++;
      v8 = __ROL__(a4 + v8, 8);
      --v6;
    }
    while ( v6 );
    result = v8;
  }
  return result;
}

My friend tried to simplify it or make sense of it and this is what he came up with:

int __stdcall sub_40C78E((_BYTE *)buffer, int nonce1, int nonce2)
{
  char v5; 
  int n; 
  int v8; 

  n = 0x400;    // It is a little bit confusing, because the length of block is 0x200 (rest of buffer is filled by 0).
                        // Only first 0x200 bytes are saved to a file for block CT0A.
  v5 = 0;
  v8 = nonce1;

  do
  {
      LOBYTE(v8) = v5 + LOBYTE(v8);
      *buffer ^= LOBYTE(v8);
      v5 = *buffer;
      buffer++;
      v8 = __ROL__(nonce2 + v8, 8);
      --n;
  }
  while ( n );

return v8;       
}

Where nonce1 and nonce2 is suppose to represent some kind of key.

What we have found with this infection is that if you XOR the first byte of the cipher text with the plain text, you get a key byte you can use to get the first byte of every file back. Which makes sense with this function because the first time in the loop the key is added to 0, which means it is simply the key. But then this guy used some type of weird CFB type xor encryption where it uses the previous xor'ed byte with the key next.

I just don't quite understand the function and I was hoping somewhere here could perhaps simplify it more for me or explain it. I also know C#, and VB if anyone would know how to explain the function in these languages.

Answer 1

As you expected, it's doing a very simple XOR. The equivalent code looks something like this:

int mystery(char *buff, int bufsize, int nonce1, int nonce2)
{
  int result = 0;
  // ch is the next byte (character) in the buffer
  char ch = 0; 
  int count = bufsize; 
  char *ptr = buff; 
  int x;

  for (x = nonce1; count; --count)
  {
      // this bit of trickery just replaces the low 8 bits
      // of x with the low 8 bits of (x+ch) neglecting carry, if any
      x = (x & ~0xff) | ((x+ch) & 0xff);
      // XOR the buffer with the calculated x value
      *ptr ^= x;
      // read in the next character into ch
      ch = *ptr++;
      // obfuscate by adding nonce2 
      x += nonce2;
      // if x = 0x12345678, this would make it 0x34567812
      // for 32-bit ints.  Just a rotate left of 8 bits.
      x = (x<<8) | ((x >>((sizeof(int)-1)*8) ) & 0xff);  
  }
  result = x;
  // return the last calculated x which may be used to chain all
  // of the blocks together.  That is, the return value x is 
  // probably passed as nonce1 to encode the next block.
  return result;
}