I did this with a lot of Android apps so I'll share what I have learned.
When thinking of sniffing traffic, the first thing that comes to mind is "I'll just root the Android device, run tcpdump on it and I'm done". That works for simple things, but it gets really messy really fast. And if the app uses SSL (like 99% of what's out there) it's even worse.
Use a proxy
Most apps use HTTP/HTTPS to communicate with the server. The easiest way to sniff that kind of traffic is using a sniffing proxy (also called debugging proxy). Common ones are Charles and Fiddler. I use Charles, so I'll show you how to work with it, but Fiddler should be similiar.
First off, you need your computer and your Android device on the same LAN. Upon starting Charles, it will automatically start proxying your computer's traffic. You probably don't want this, so disable it from Proxy -> Proxy Settings... -> Windows. Now point your Android network settings to use your computer as HTTP proxy (Charles default port is 8888). Charles will show you all HTTP traffic from your device.
If you need SSL (HTTPS) proxying, you have to install the Charles certificate on your device. Android requires you to have a protected (at least pattern) lockscreen to install external certificates. You can find the certificate in the doc/ directory in Charles' install path, or you can visit http://www.charlesproxy.com/getssl/ from your device browser. Then, in Charles, go to Proxy -> Proxy Settings... -> SSL, enable SSL proxying and add the hosts you want to proxy (or just add * to proxy everything).
Non-HTTP(S) communication
An HTTP debugging proxy will cover most apps out there. In case the app is not using HTTP(S) things get a little more complex. You may want to look into the SOCKS proxy capability of Charles, and how to configure a SOCKS proxy on your device. If that doesn't suffice, then you'll really want to break out tools like tcpdump/Wireshark. You could set up a fake DNS, ARP spoof the device, use a sniffing VPN, etc. There are a lot of ways to go about it. A fake DNS is also good if you want to emulate the server, instead of the client.
SSL certificate checking
When dealing with SSL, most apps trust what the OS trusts. Since you can intall external certificates on Android, you're good. Rarely, the app will check the server certificate against an embedded CA root. This can be bypassed in a few ways:
- Using tools like apktool and smali, inject your own certificate into the app, replacing the original one;
- Write an xposed module that hooks the data send/receive methods, to sniff the decrypted messages;
- Write an xposed module that hooks the certificate checking routine, bypassing it.
I would go with the first or the last one.
Decompilation
At some point you'll have to look at the application code. Maybe it's using a proprietary binary protocol, maybe the traffic is encrypted or compressed, maybe you can't make sense of some values. I suggest trying multiple decompilers/engines. They are not perfect, and a lot of times one works well where the other fails. Virtuous Ten Studio is a good one to start with. Then I alway extract the classes.dex file from the APK (it's just a ZIP file) and run it through dex2jar. I then run the resulting JAR through JD-GUI (JD-Core engine) and SecureTeam Java Decompiler (Procyon engine). Also, jadx can target DEX directly, without dex2jar.
InstaMessage
I checked the app you're trying to reverse. It's using standard HTTPS communication, easy to sniff using either Charles or Fiddler. The server has various GET endpoints and the client sends a base64-encoded data field. This is binary data, I haven't really looked into it but the first 70% is identical in all requests, so I suspect it's some kind of session token. The final part varies with requests, but the whole field is pretty high entropy so I'm thinking it's either compressed or encrypted. I would try to decompile the app and see what's going on. The server response, on the other hand, is plain JSON.
