1

I am trying to understand an algorithm from a malware I found. This algorithm encodes http request to C&C. The decompiled version is taken from IDA. There are some comments I wrote.

The problem is: I dont see the malware sends result from QueryPerformanceCounter() to its C&C. So without this value how C&C decodes this request?

int __cdecl sub_A2AA0(int a1, unsigned int a2, int a3, _BYTE *a4, int a5)
{
// a5: length of buffer
// a4: buffer to encode
// a3: always 0
// a2: always 4
// a1: addr contains result from QueryPerformanceCounter()

  signed int v5; // eax@1
  char v6; // si@3
  unsigned int v7; // eax@3
  signed int v8; // edi@3
  char v9; // cl@4
  int v10; // edx@4
  int v11; // edi@7
  int v12; // esi@7
  int result; // eax@7
  char v14; // cl@8
  int v15; // ebp@9
  _BYTE *i; // edi@9
  char v17; // cl@10
  char v18[256]; // [sp+0h] [bp-100h]@2

 // buffer with 0 to 255
  v5 = 0;
  do
  {
    v18[v5] = v5;
    ++v5;
  }
  while ( v5 < 256 );


  // exchange content of a random index starting with index 0
  v6 = 0;
  v7 = 0;
  v8 = 0;
  do
  {
    v9 = v18[v8];
    v10 = (unsigned __int8)(v6 + v18[v8] + *(_BYTE *)(v7++ + a1)); // generate index
    v6 = v10;
    if ( v7 >= a2 )
      v7 = 0;
    v18[v8++] = v18[v10];
    v18[v10] = v9;
  }
  while ( v8 < 256 );

// iterate local buffer again, exchange content of cells, starting with index 0
  v11 = a3; // init with 0
  LOBYTE(v12) = 0;
  for ( result = 0; v11; v18[v12] = v14 )
  {
    result = (unsigned __int8)(result + 1);
    v14 = v18[result];
    --v11;
    v12 = (unsigned __int8)(v12 + v18[result]);
    v18[result] = v18[v12];
  }

// mask the request
  v15 = a5;
  for ( i = a4; v15; --v15 )
  {
    result = (unsigned __int8)(result + 1);
    v17 = v18[result];
    v12 = (unsigned __int8)(v12 + v18[result]);
    v18[result] = v18[v12];
    v18[v12] = v17;
    *i++ ^= v18[(unsigned __int8)(v17 + v18[result])]; // simple xor
  }
  return result;


}
  • Generally speaking it uses a1 in line marked with "// generate index" comment. *(_BYTE *)(v7++ + a1) makes dereference from this address, so the content of v18 depends on the counter content. – w s Oct 6 '16 at 9:44
2

This seems to me like RC4, a stream cipher (here is a somewhat similar implementation, and here's another). It does not encode data, it encrypts it (with a key). Being a stream cipher, it generates a sequence of bytes derived from the key, and then XORs the data with those. These two steps are usually separated in two functions, which is why you don't see any reference to the key in your function: because it is the part which XORs with an already generated stream somewhere before that function gets called.

a1 seems to be a pointer to the state structure, so it is not a pointer to a 4 byte sequence, but rather, a structure which holds the current state. From what I have seen, most binaries do something like this:

RC4_Init(state, key, ...)
RC4_Process(...) // this is the function you're looking at now

Find where the function is called, look a bit above (try cross-referencing a1), and you should find the initialization function, which has the key.


To further expand on why I think this is RC4

Your code:

 // buffer with 0 to 255
  v5 = 0;
  do
  {
    v18[v5] = v5;
    ++v5;
  }
  while ( v5 < 256 );

rc4_init:

void rc4_init(unsigned char *key, unsigned int key_length) {
    for (i = 0; i < 256; i++)
        S[i] = i;

    /* ... more code ... */
}

Your code:

// exchange content of a random index starting with index 0
  v6 = 0;
  v7 = 0;
  v8 = 0;
  do
  {
    v9 = v18[v8];
    v10 = (unsigned __int8)(v6 + v18[v8] + *(_BYTE *)(v7++ + a1)); // generate index
    v6 = v10;
    if ( v7 >= a2 )
      v7 = 0;
    v18[v8++] = v18[v10];
    v18[v10] = v9;
  }
  while ( v8 < 256 );

Chunk of rc4_init:

for (i = j = 0; i < 256; i++) {
        j = (j + key[i % key_length] + S[i]) & 255;
        swap(S, i, j);
    }

    i = j = 0;

Inlined swap in your code:

    v9 = v18[v8];
    v18[v8++] = v18[v10];
    v18[v10] = v9;

RC4 swap:

void swap(unsigned char *s, unsigned int i, unsigned int j) {
    unsigned char temp = s[i];
    s[i] = s[j];
    s[j] = temp;
}

Do you see the similarities? I definitely think this is RC4, so your best bet is to hope that the malware uses a static/predictable/weak key, so you can decrypt the communication (if sniffing from outside).


TL;DR You can't decrypt this without the proper key, which may or may not be predictable. Look at how it is generated to find out if it is.

  • 1
    Although you're probably right about your guess, it's important to note the actual code pieces you compared are only the state initialization, which could be replaced with any other similar algorithm, as the core of RC4 is in the sequence update which you skipped (and is indeed RC4) – NirIzr Oct 9 '16 at 19:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.