2

For my own personal curiosity and development (building a bit of code mimicking the protocol), I am trying to figure out what the communication protocol encryption is made out of in a particular chat application for Windows.

Based on what I know from some years ago where somebody managed to reverse engineer the encryption, it uses - at least AES encryption and some XOR´ing. It´s been updated many times since then though, and when I try to run it through signsrch, there are many references in it compared to the old version.

So, I have tried to read what I could find about how one identifies encryption methods within an application, and so far I have come to the point where a breakpoint is triggered in Ollydbg whenever I´m trying to send something (login and password).

But now, I'm lost...

Should I be looking through the stack to try finding pure ASCII showing the string I entered before it getting encrypted or what ?

Also, it puzzles me that the addresses showing in signsrch seem not to be the same as the addresses in the application when viewing it in Ollydbg: As far as I have understood, the addresses are pointing to various encryption signatures, but those addresses - in Ollydbg - are just assembly commands.

Would anybody be so kind to give me a bit of advice on this subject?

Improve this question

2 Answers 2

Reset to default
2

Just as an example, consider the source code for AES at https://polarssl.org/aes-source-code. This has various tables, one of which is the Forward S-Box, FSb - which should be the same in all AES implementations. So, a signature checker like signsrch will say "AES" if it finds the fsb table, but it doesn't know which AES implemenation is used.

Now, the FSb table is used in various functions in the source code - aes_gen_tables, aes_setkey_enc, aes_setkey_dec, aes_crypt_ecb - they all use the same FSb array. If you look at a different AES implementation, these functions will probably have different names, use a different set of local variables, handle errors differently, and won't be too similar in object code to the polarssl ones.

Also, depending on the framework your compiler uses, your application will probably call a framework function which calls something else .. and arrives at the basic aes_* functions 5-10 levels deeper. So, there's no easy way to match your breakpoint to the actual aes_* functions.

Third, the fact that signsrch detects an AES signature does not neccesarily mean AES is used at all. Your application might link the openssl libraries, which pulls in all kind of encryption functions, possibly even those that aren't used in your application.

The best thing you can do is place a hardware breakpoint on the FSb or a similar table, which will trigger when the program actually uses AES. Then, walk up the stack to find out which function calls which, and how this relates to your name/password function. Try to match this function backtrace to known implementations of AES (i'd start with openssl), so you can possibly find out where the high-level routines are. Then, place breakpoints on these high-level routines to get an idea of what gets encrypted, what the encryption results are, and how this relates to the bits on the network, which you can capture using wireshark or a similar tool.

Improve this answer
4
  • Thanks Guntram, I´ll see if I can figure out how to dig out the table using ollydbg. Btw, why do you suggest hardware over software bp´s?
    – Corey Hart
    Commented Jul 30, 2014 at 15:16
  • With a software bp, you can break on an instruction, which means you have to know which processor instruction to break on. With a hardware bp, you can place a breakpoint on a memory location, which will trigger whenever the memory address gets read or written. So if you have the address of the data table (which is easy to find if you know what to search for), but don't know where your program accesses it, a hardware bp on the data is what you want.
    – Guntram Blohm
    Commented Jul 30, 2014 at 15:54
  • Thanks, that was helpful. I assume that you indeed talk about a hardware breakpoint, and not the 3rd kind of breakpoint which is not software nor hardware, but memory breakpoint, that can be done in ollydbg? Just to be sure. I found a wiki where the table you mentioned is showing, and managed to find it in the code. It´s not being triggered though upon logon if I set a hardware breakpoint on where it starts, so I guess I´ll set hw breakpoints, 4 at a time, on each interesting table that I´ll be able to find in the code, untill I see one or more of them triggered. Thanks so much again!
    – Corey Hart
    Commented Jul 31, 2014 at 10:45
  • I found out now that the only way to get a result for me is adding a memory breakpoint on the Rijndael table - not a hardware breakpoint. And not only that: I also now see the login name I´m using occuring in the stack, a big wow here :) I see it many times though, and no evidence of the password. So at the point where I am now, I´m not sure about how to proceed. How do I determine "known implementations/SSL" in the stack?
    – Corey Hart
    Commented Aug 4, 2014 at 14:28
1

This is not a reverse engineering answer per se, but more of a general common sense approach. Since it is a chat application, it presumably uses a network. I would launch the application while using a network packet dissector (sniffer) like wireshark. From there I would try to identify if a common encryption technique is being used (e.g. TLS encryption). You may find that is an adequate solution. Moreover, using a network sniffer while running the program in a debugger may help you identify the relevant portions of code that correspond to network packet transmission and thereby construction by stepping in reverse from that point.

Improve this answer
1
  • Thanks Fred. To me this is not common sense, as long as it´s not having a big sign attached to it telling which encryption methods were used in which order. I remember that in the previous version, there was both XOR and AES used - I think 2xAES. How would you possibly determine that from looking at packages only? There´s nothing raw in the protocol except for a few bytes with a 2-3 byte timestamp and such. I´m using commview, and I´m aware that it´s got less detection features than wireshark. Would wireshark really be able to see when something is AES encrypted?
    – Corey Hart
    Commented Jul 30, 2014 at 14:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.