2

RC4 encryption is quite basic and its Pseudo random generation algorithm is given at Wikipedia

When Looking for a hashing algorithm like MD5 or SHAx it is pretty simple to look for the Hashing constants in the code, for example to find the MD5 code most times looking for the byte sequense of 0x01234567 is enough and you'd probably hit the function iterating the text section.

Now that example gives the MD5 function. but what if i would like to find an implementation of the symmetric RC4 encryption? would there be any obvious opcode sequense that would repeat itself?

I'm asking that question because im working on some heavily anti-debugging obfuscated piece of code with no symbols. I know for a high chance (like 90% sure) it is using a RC4 encryption. now i want to be able to find that RC4 function just like i was able to find the MD5 hashing functions.

Just in case i was wrong, and its not hard-coded inside the main executable itself (Because md5 is, so i can assume it should be too), i have already tried to set a BreakPoint at CRYPTSP!CryptEncrypt with no success, are there any alternative Api functions to Encrypt using RC4?

So to sum my questions up:

  1. Is there any static opcode usage or number in the assembly of an RC4 encryption?
  2. Are there any alternative WinAPI calls for encrypting using RC4 encryption beside CRYPTSP!CryptEncrypt?
  3. Any other way to look for the RC4 encryption algorithm you can help me think about is blessed.
2
  1. Is there any static opcode usage or number in the assembly of an RC4 encryption?
  • Opcode: Search the disassembly for xor <x>, <y> where <x> != <y>.
  • Number: Search for 0x100

Obviously both of these searches will yield many false-positives, so you'd have to look at each match manually to see if it's part of an RC4 function.

  1. Are there any alternative WinAPI calls for encrypting using RC4 encryption beside CRYPTSP!CryptEncrypt?

Every time I see software using RC4, the RC4 code is always written inline, not imported from a dynamic library.

  1. Any other way to look for the RC4 encryption algorithm you can help me think about is blessed.

You could try Aligot.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.