Reversing encryption by analysing executable

Question

I have a program that reads data from a file, parses/organizes it, and spits it out as an encrypted XML. The application can also take an encrypted XML and spit out the original file. My objective at this point would be to access the clear-text XML (I'm not interested in the clear text original file as it's not XML organized)

I have no idea what the encryption is yet although one guy on a forum said it was AES-128 (not sure how he got to that conclusion).

I ran PEiD with the KANAL plugin on the application, it doesn't detect any encryption signatures.

Because I have access to the program and some past some experience with exploiting BO on WinXP with some knowledge of ASM, I figured I could give it a try using a debugger.

In a nutshell, what are the general steps I should be following to figure this out? In this situation would it be best to start looking for the encryption key itself, or find a way to use the application's encrypt/decrypt functions to my advantage?

Answer 1

There are typically many ways to start, and which one you want to use depends on your experience. Also, what works for one target might fail on another, and vice versa. What i'd start with is:

• Use signsrch to check if the executable has a standard encryption algorithm linked in. Note that this might yield false positives (if the application links openssl, for example, you might find signatures of many algorithms even if only one or two of them get used)
• Check for strings in the file that relate to encryption, and google for them. Maybe this helps to find which encryption library was used. For example, the string SHA-%d test #%d: quickly leads to the polarssl source code.
• Trace the application while running procmon. You will probably find a sequence of CreateFile, multiple WriteFile and a CloseFile call, when writing the encrypted file, and the same with ReadFile when reading the encrypted file. In some cases, the parameters of these calls give you a hint already; for example, if the first WriteFile has a byte count of 32, and every following call 4096, this could be a hint that the first 32 bytes are a 256-bit key.
• Check the stack when a call to ReadFile or WriteFile happens, You can do this by placing a breakpoint there while running the application under a debugger, or (much easier imho) look at the stack for several of the procmon calls. This gives you a hint of which chain of functions encrypts the file and writes it.

• Pay attention to the stack differences in the CreateFile, WriteFile on the encrypted file, and possibly WriteFile on something else. These might as well give you hint which is your encryption function. For example:

  • The CreateFile for the encrypted file has the stack ABCDEFGH
  • The WriteFile for the encrypted file has the stack ABCIJKLM
  • The WriteFile for something else has the stack ABNOPKLM

  where each letter is one stack entry. In this case, i'd assume KLM to be part of the runtime library (fwrite, write, ....), since unrelated WriteFiles share it, C to be the main writer function (as it's the last to call CreateFile and WriteFile), and I and J the functions that encrypt and write.

• Load the file into IDA, or OllyDbg, and have a closer look at C, I and J. Which other functions do they call? Are some of these functions the same ones you identified with signsrch earlier?
• Run the file under the control of IDA, or OllyDbg, and place breakpoints on C, I, J, and maybe some of the functions you found with signsrch. Check the parameters on entry and exit from these functions. Do some of them ring a bell? Does one of the functions has a parameter that is a pointer to a buffer which looks like XML before the function is called, and garbage after it's called? Or vice versa when reading? Congrats, you've just found the function that does the encryption.
• When you found the function that does the encryption, instead of figuring out how it works, it might just be easier to replace the call to it with a series of NOPs (beware of return codes though, maybe you need to set eax to something). Bingo, you have a program that writes plain XML instead of encrypted XML now, without ever figuring out the details of the encryption and having to write one single line of code (unless you consider a series of NOPs code, that is).