1

I noticed something very odd about Linkedin anti-bot behavior and I am not being able to fully understand it. Basically, there are multiple people who set their profiles to public, allowing search engines such as Google to index them. If you initiate an anonymous browsing session and search for a profile, you will get a page that looks like this:

(example for https://www.linkedin.com/in/parag-agrawal-5a14742a) public profile screenshot If you try repeatedly, however, or if you test an automation software such as selenium for accessing these profiles, you will after a while be redirected to an authwall page, and no more public profiles will be visible to you.

I have already tried to hide all kinds of fingerprintable data, such as IP, user agent, screen resolution (for canvas fingerprinting), but after a few requests, my access limit drops to one single public profile. I guess they make an analysis of the request pattern within their servers, making my section identifiable.

I would like to know which features are observed to determine whether a visitor is human or not, and how could I simulate human behavior.

Improve this question
4
  • 1
    I'm not sure this can really be called RE...
    – Igor Skochinsky ♦
    Jul 26, 2019 at 17:10
  • I was also in doubt about this, @Igor, I also did not explain my main goal here: I have a similar kind of public data that I would like to make available for search engines, but not for data scraping robots. I know they have the best-in-class anti-scraping mechanism, but I was not able to find anything in literature or patents
    – Rafael Magalhães
    Jul 26, 2019 at 17:39
  • Hi and welcome to RE.SE. Perhaps the blocking is IP based? Have you tested changing the IP or masking it (thinking proxy here)? Do you or don't you use something like uMatrix and/or uBlock Origin?
    – 0xC0000022L ♦
    Jul 31, 2019 at 20:00
  • Hey @0xC0000022L♦, I tried rotating IPs and masking other fingerprinting evidences, but with little success. uMatrix and uBlock do something similar. What I noticed is that it blocks my access through analysis of HTTP request pattern.
    – Rafael Magalhães
    Aug 12, 2019 at 17:09

0

Reset to default

Browse other questions tagged or ask your own question.