finding a RC4 Encryption function in obfuscated assembly code with no symbols

Question

RC4 encryption is quite basic and its Pseudo random generation algorithm is given at Wikipedia

When Looking for a hashing algorithm like MD5 or SHAx it is pretty simple to look for the Hashing constants in the code, for example to find the MD5 code most times looking for the byte sequense of 0x01234567 is enough and you'd probably hit the function iterating the text section.

Now that example gives the MD5 function. but what if i would like to find an implementation of the symmetric RC4 encryption? would there be any obvious opcode sequense that would repeat itself?

I'm asking that question because im working on some heavily anti-debugging obfuscated piece of code with no symbols. I know for a high chance (like 90% sure) it is using a RC4 encryption. now i want to be able to find that RC4 function just like i was able to find the MD5 hashing functions.

Just in case i was wrong, and its not hard-coded inside the main executable itself (Because md5 is, so i can assume it should be too), i have already tried to set a BreakPoint at CRYPTSP!CryptEncrypt with no success, are there any alternative Api functions to Encrypt using RC4?

So to sum my questions up:

1. Is there any static opcode usage or number in the assembly of an RC4 encryption?
2. Are there any alternative WinAPI calls for encrypting using RC4 encryption beside CRYPTSP!CryptEncrypt?
3. Any other way to look for the RC4 encryption algorithm you can help me think about is blessed.

Answer 1

1. Is there any static opcode usage or number in the assembly of an RC4 encryption?

• Opcode: Search the disassembly for xor <x>, <y> where <x> != <y>.
• Number: Search for 0x100

Obviously both of these searches will yield many false-positives, so you'd have to look at each match manually to see if it's part of an RC4 function.

2. Are there any alternative WinAPI calls for encrypting using RC4 encryption beside CRYPTSP!CryptEncrypt?

Every time I see software using RC4, the RC4 code is always written inline, not imported from a dynamic library.

3. Any other way to look for the RC4 encryption algorithm you can help me think about is blessed.

You could try Aligot.

Answer 2

I would first start by reading the following article https://blog.talosintelligence.com/2014/06/an-introduction-to-recognizing-and.html

For some other encryption you could also try to use https://hex-rays.com/blog/findcrypt/ this plugin that identifies crypto constants in the IDA database, then find xrefs to those constants usually will help you find the encryption/decryption loop.

The previous answer also had some good thumb rules.