Bypassing Cloudflare Scrapeshield

Question

I am working on a webscraping project and am having problems with scrapeshield cloudflare. Does anyone know how to get around this? I am using selenium webdriver which redirects to some Speedpeed scrapeshield page. Built using python on top of Firefox. Usually the view does not redirect. Is there something webdriver does differently than a regular browser?

Answer 1

See what scrapeshield does by checking to see if you are using a real browser, essentially checking your browser for specific errors. Suppose Chrome cannot handle an IFrame if there is a 303 error in the line, a certain web browser responds differently to different tests, so the webdriver should not react to this, forcing the system to say "We got an attacker, change the page!". I could be right, but not 100% sure, though ...

Additional source information:

I found most of this information in Defcon's talk about web sniffers and did not allow them to get the correct vulnerability information on the server, he also made a web browser identifier in PHP.

Answer 2

"I'm using selenium webdriver, which redirects to some Speedpeed page using scrapeshield." - This is not how Scrapeshield works.

In addition, I would suggest that this stream be closed when you ask for help, bypassing the security feature that the site owner intentionally decided to include on his site so that people do not clear their contents.