There's a post here (http://narenonit.blogspot.in/2011/07/my-youmint-api-to-send-free-sms.html) in which the author has tried to inform pretty well, as far as I can guess. The exact problem I'm facing is how to study the HTTP connection. Would it be possible for someone to explain the procedure a bit more clearly.
You could use some tools to view what gets sent from your browser to the web app and deduce how to "emulate" this behavior. A network sniffer would be the most useful here. Wireshark (especially it's Follow Stream feature)for example. Also , some browser plugins can be useful, for example Firebug, TamperData and Poster.
Appart from that, it's just the matter of replicating what the browser does. Basically, http communication consists of either GET or POST requests and replies (there are other requests tho and you should look into RFC2616 for more details). So, by looking at HTTP streams in Wireshark, you could deduce what kind of request gets sent to the web application, and what kind of reply it gets. Some (or most, even) of this info can be deduced by studying the html source of the webpage since the html form element will have the parameters and action url specified.
-
4
-
1mitmproxy is pretty good, and it's really easy to modify/replay packets. – toasted_flakes Apr 4 '13 at 17:53
In addition to the mentioned tools, I'd like to point out httpry, which is based on libpcap (like tcpdump and WireShark) but aimed at only listening to and listing HTTP traffic.
Fiddler is another tool, albeit not FLOSS, that is up for the job just like the Firefox "Tamper Data" add-on. Remko also mentioned it in his comment, but I think it needs to be mentioned more prominently than in a comment. Was missing it from the other answer when reading through it.
answered Apr 4 '13 at 17:27
The way to do it is using web scraping techniques. Using sniffers and proxies is overkill if you can access the web page/resource with a browser. In this case you only need to automate what the user does within the browser and using tools like Chrome Developer Tools for inspecting the HTML and AJAX connections.
I've written several articles on this subject if you want to take a look:
- Google Search NoAPI (automating Google searches)
- Automated Browserless OAuth Authentication for Twitter
- Web Scraping Ajax and Javascript Sites includes a LOT of resources in the end about offline/headless browsers and browsers emulation like HTMLUnit
For applications that has some anti-scraping techniques you can take a look at Running Your Own Anonymous Rotating Proxies
If there is some specific requirement that is not covered there please don't hesitate to leave a comment and I will help you on that.
Fiddler learn to love it..
It provides an easy to use interface for capturing requests/responses to the intended sites.
Essentially you need to:
Make the request you want to automate using your browser.
Use Fiddler to capture these requests and investigate them. (Is it a POST or GET? Is there a CSRF token? What variables map to the input I provided etc..)
Get the response back from the server.
Identify the parts that contain your output.
Then you need to write a script that takes whatever input you want to problematically send and then makes the request in the same fashion that the browser did. Store the output and pull out the portions that you want.
Once you successfully figure out how to formulate the requests you can then build a class to hide the details of how it all works and make clean looking code.
