You don't really need another tool if you know how to use wireshark. Wireshark will let you extract the TCP part of a stream, which is exactly what you'll get using any kind of proxy or network bridge as well.
From your description, i assume the protocol isn't just some kind of plain text, like your chat being encapsulated into SOAP in XML - it'd be easy to analyze this. The problem with binary data is that you can make some guesses, but if they don't fit, you can't really do much more than analyze the applet.
What i'd do if i were you is:
Force wireshark to treat the connection as SSL (right-click a packet, "Decode as", then select SSL). If the communication is SSL-encrypted, wireshark should at least find a decodable client hello / server hello packet exchange at the start. If that packet exchange makes sense, you could proceed with fiddling with DNS to redirect the client's request, and set up fiddler, stunnel, or mitmproxy to receive, decode, and forward those requests.
Capture a few dozen KB of traffic, and save the resulting TCP stream to a file. Try to compress this file with gzip. If it compresses well, then the TCP stream probably isn't compressed, just encrypted with a weak, XOR like, algorithm. Type 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' into the chat. Then type 'bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb' into the chat. Check what happens with the data on the wire. Maybe this will give you a hunch how the data is encoded and an idea of how to proceed.
If the TCP stream doesn't compress, it's either precompressed, or encrypted with an algorithm that adds a lot of entropy. Comparing the size of the chat text with the size of the TCP stream might give you an idea which it is. If it's compressed, check the first few bytes of each packet, if they're similar, it might be that each packet has its individual compression. Check these first few bytes, or the first bytes of the stream, against signatures of well-known compression algorithms.
If all this doesn't help, you best guess is running the whole thing through a java decompiler (there's a nice collection at http://www.javadecompilers.com/), as there's not really a good way to guess an encryption algorithm and/or key from the output byte stream alone.
By the way, it can't hurt to mention on which web site you found the chat (even if it's something embarassing like littlenudegirls.com, as you're not using your real name here); maybe someone else is reading this and has more detailed information to share about your particular web site, or maybe someone will google for that in the future and contact you to work on it together.
