RC4 encryption is quite basic and its Pseudo random generation algorithm is given at Wikipedia
When Looking for a hashing algorithm like MD5 or SHAx it is pretty simple to look for the Hashing constants in the code, for example to find the MD5 code most times looking for the byte sequense of 0x01234567 is enough and you'd probably hit the function iterating the text section.
Now that example gives the MD5 function. but what if i would like to find an implementation of the symmetric RC4 encryption? would there be any obvious opcode sequense that would repeat itself?
I'm asking that question because im working on some heavily anti-debugging obfuscated piece of code with no symbols. I know for a high chance (like 90% sure) it is using a RC4 encryption. now i want to be able to find that RC4 function just like i was able to find the MD5 hashing functions.
Just in case i was wrong, and its not hard-coded inside the main executable itself (Because md5 is, so i can assume it should be too), i have already tried to set a BreakPoint at CRYPTSP!CryptEncrypt with no success, are there any alternative Api functions to Encrypt using RC4?
So to sum my questions up:
- Is there any static opcode usage or number in the assembly of an RC4 encryption?
- Are there any alternative WinAPI calls for encrypting using RC4 encryption beside CRYPTSP!CryptEncrypt?
- Any other way to look for the RC4 encryption algorithm you can help me think about is blessed.