I'm analyzing some software that appears to encrypt its communications over the network, but it does not appear to be SSL. How can I easily determine what encryption algorithm its using, and maybe find the key?
Maybe check out this IDA plugin.
After you locate the crypto functions, doing a cross-reference in IDA should allow you to see where the functions are called and likely the key is nearby. If you can set a break-point on those functions and see what is being passed in for the key, this, of course, would be the easiest way.
For a bit more advanced way of automatic crypto identification see Felix Gröbert's work on Automatic Identification of Cryptographic Primitives in Software . He uses a pintool to dynamically instrument the code which can allow to even recover keys. The code is also available. The repository contains other tools used in comparison , such as PeID and OllyDBG plugins.
I have not used it but there is an open source tool called Aligot that may help when the encryption algorithms have been obfuscated. According to its authors, Aligot can idenfity TEA, MD5, RC4 and AES.
Aligot does have an important disclaimer:
Aligot was build as a proof-of-concept to illustrate the principles described in the associated paper. In particular it is not currently suitable to automatically analyze large programs. If you are interested in such project, please contact the author ;)
Despite the disclaimer, the results indicated in the paper suggest that Aligot is worth looking into.