I am trying to reverse engineer the protocol of a chat written in java to implement my own client. The official client is a java applet embedded in the website so I thought it might use just https but I found using wireshark that it uses a generic protocol and an arbitrary port. My first approach was to use the java proxy settings and proxifier to force the traffic through fiddler in hope it might give me some more information about the protocols structure but with no success - fiddler did not even recognize the traffic but I guess thats because it was developed for http/s traffic only.

So my question is: How can I reverse engineer the protocol used? I thought that there has to be something similar to fiddler using a socket proxy or a network bridge (similar to what a lot of VPN clients use) but I could not find anything working under windows. Another approach would be to decompile the java binaries but I am not really interested in the code itself only in the protocol. Any suggestions how to proceed from here?

You don't really need another tool if you know how to use wireshark. Wireshark will let you extract the TCP part of a stream, which is exactly what you'll get using any kind of proxy or network bridge as well.

From your description, i assume the protocol isn't just some kind of plain text, like your chat being encapsulated into SOAP in XML - it'd be easy to analyze this. The problem with binary data is that you can make some guesses, but if they don't fit, you can't really do much more than analyze the applet.

What i'd do if i were you is:

  • Force wireshark to treat the connection as SSL (right-click a packet, "Decode as", then select SSL). If the communication is SSL-encrypted, wireshark should at least find a decodable client hello / server hello packet exchange at the start. If that packet exchange makes sense, you could proceed with fiddling with DNS to redirect the client's request, and set up fiddler, stunnel, or mitmproxy to receive, decode, and forward those requests.

  • Capture a few dozen KB of traffic, and save the resulting TCP stream to a file. Try to compress this file with gzip. If it compresses well, then the TCP stream probably isn't compressed, just encrypted with a weak, XOR like, algorithm. Type 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' into the chat. Then type 'bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb' into the chat. Check what happens with the data on the wire. Maybe this will give you a hunch how the data is encoded and an idea of how to proceed.

  • If the TCP stream doesn't compress, it's either precompressed, or encrypted with an algorithm that adds a lot of entropy. Comparing the size of the chat text with the size of the TCP stream might give you an idea which it is. If it's compressed, check the first few bytes of each packet, if they're similar, it might be that each packet has its individual compression. Check these first few bytes, or the first bytes of the stream, against signatures of well-known compression algorithms.

  • If all this doesn't help, you best guess is running the whole thing through a java decompiler (there's a nice collection at http://www.javadecompilers.com/), as there's not really a good way to guess an encryption algorithm and/or key from the output byte stream alone.

By the way, it can't hurt to mention on which web site you found the chat (even if it's something embarassing like littlenudegirls.com, as you're not using your real name here); maybe someone else is reading this and has more detailed information to share about your particular web site, or maybe someone will google for that in the future and contact you to work on it together.

  • Hey, thanks for the answer and sorry for the late reply - I tried all of your suggestions (took me a while), maybe I am not experienced enough but I think there are different clients depending on the date you created your account, some seem to use just a gzip compression others use an SSL connection. I will try to decompile at least one client and hopefully get some answers. Will post here again as soon as I know more. – Ganymed_ Oct 16 '15 at 13:56

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.