0

I am trying to understand why if I do fetch() 4 times on the active web page, after the fifth fetch server returns response that I am a "robot" I am using google chrome I go the following link: tickets selling site (might not open in non US countries). From the 'network' tab I 'copy as fetch' that very first request and paste it to the console and execute it

copy the first page request from the network

enter image description here

after doing so 5 times, I start receiving "robots" response

This is a good response: enter image description here

this is a "robot" response: enter image description here

And after that all requests made by the site itself return the same "robots" response everytime. But refreshing the page/opening new page, resets everything and I can send those requests again.

My question is: how can I understand what happens when I open browser page? What happens that the server starts returning good responses again after I refresh the page?

I was trying to drop cookies, but this didnt help.

The code I use to send request:

fetch("https://ticketswestinw.evenue.net/cgi-bin/ncommerce3/SEGetEventInfo?ticketCode=GS%3ATWC%3AEARN20%3AEARN0807%3A&linkID=twcorp&shopperContext=&pc=&caller=&appCode=&groupCode=ARNBSB&cgc=&dataAccId=883&locale=en_US&siteId=ev_twcorp", {
  "headers": {
    "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
    "accept-language": "en-US,en;q=0.9",
    "cache-control": "max-age=0",
    "sec-fetch-dest": "document",
    "sec-fetch-mode": "navigate",
    "sec-fetch-site": "same-origin",
    "sec-fetch-user": "?1",
    "upgrade-insecure-requests": "1"
  },
  "referrer": "https://ticketswestinw.evenue.net/cgi-bin/ncommerce3/SEGetEventInfo?ticketCode=GS%3ATWC%3AEARN20%3AEARN0807%3A&linkID=twcorp&shopperContext=&pc=&caller=&appCode=&groupCode=ARNBSB&cgc=&dataAccId=883&locale=en_US&siteId=ev_twcorp",
  "referrerPolicy": "no-referrer-when-downgrade",
  "body": null,
  "method": "GET",
  "mode": "cors",
  "credentials": "include"
}).then(x => x.text().then(y => console.log(y)));
6
  • I would assume that they use JavaScript to generate something that changes in every request. If you replay existing requests 5 times it is recognized that the changing element was all the same which never happens if you use the web browser with JavaScript running.
    – Robert
    Aug 18 '20 at 15:07
  • but I send that very initial request what is sent when I open the page, that one that gets HTML markup. So I assume that nothing can modify it, because the scripts are run after browser receives request response Aug 18 '20 at 15:30
  • That long URL does not look like the start page. Also consider that you load the page but not the images, scripts, ... that are linked on it and that a regular web browser would load.
    – Robert
    Aug 18 '20 at 15:37
  • This is exactly url that used for accessing the page, it contains event id and some additional info used in further requests >Also consider that you load the page but not the images, scripts Sorry I didnt get it, the response contains all the things like HTML + scripts + images Aug 18 '20 at 15:50
  • Web browsers will download (while HTML requests is still running) all linked resources while your requests does not. This difference can be detected on server side -> robot detected. Together with TLS fingerprinting and a lot of other facts that indicator if you use a web browser or not.
    – Robert
    Aug 18 '20 at 16:49
1

There are two major differences between a web page loaded via web browser and via script:

  1. While loading the web page a web browser also loads linked resources like scripts, images, css files, ... This can be recognized on server side.

  2. A web browser not just downloads the web page but also executed the contained JavaScript. From within JavaScript additional resources can be loaded or special calculations can be performed and the result can be sent back to the web server.

  3. For web pages that are loaded via HTTPS there are additional values (TLS extensions, the list of available TLS version(s) and cipher suites, ...) in the TLS protocol that allow to perform some sort of web browser fingerprinting on TLS level.

  4. Last but not least there are scripts such as Google Recaptcha v3 available that use JavaScript for detecting web-browsers and human interaction.

All those measures together can be combined to perform a robot/script detection on server side.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.