AJAX response capture with selenium and python

Question

I click the link in Firefox, the webpage sends the request using javascript, then the server sends some kind of response that includes the website address. Therefore, the new website opens in a new window. HTML code behind the link (I missed the start and end <span> tags):

 > class="taLnk hvrIE6" > onclick="ta.trackEventOnPage('AttractionContactInfo', 'Website', > 2316062, 1); ta.util.cookie.setPIDCookie(15190); > ta.call('ta.util.link.targetBlank', event, this, > {'aHref':'LqMWJQiMnYQQoqnQQxGEcQQoqnQQWJQzZYUWJQpEcYGII26XombQQoqnQQQQoqnqgoqnQQQQoqnQQQQoqnQQQQoqnqgoqnQQQQoqnQQuuuQQoqnQQQQoqnxioqnQQQQoqnQQJMsVCIpEVMSsVEtHJcSQQoqnQQQQoqnxioqnQQQQoqnQQniaQQoqnQQQQoqnqgoqnQQQQoqnQQWJQzhYmkXHJUokUHnmKTnJXB', > 'isAsdf':true})">Website ', > class="taLnk hvrIE6" > onclick="ta.trackEventOnPage('AttractionContactInfo', 'Website', > 2316062, 1); ta.util.cookie.setPIDCookie(15190); > ta.call('ta.util.link.targetBlank', event, this, > {'aHref':'LqMWJQiMnYQQoqnQQxGEcQQoqnQQWJQzZYUWJQpEcYGII26XombQQoqnQQQQoqnqgoqnQQQQoqnQQQQoqnQQQQoqnqgoqnQQQQoqnQQuuuQQoqnQQQQoqnxioqnQQQQoqnQQJMsVCIpEVMSsVEtHJcSQQoqnQQQQoqnxioqnQQQQoqnQQniaQQoqnQQQQoqnqgoqnQQQQoqnQQWJQzhYmkXHJUokUHnmKTnJXB', > 'isAsdf':true})">Website 

I want to capture the server response and extract the “new site” using Python and Selenium. I use BeautifulSoup for scraping and quite new for Selenium.

So far, I can find this element and click on it using selenium, which opens a new site in a new window. I do not know how to capture the response from the server.

Answer 1

I was not able to capture the AJAX answer with selenium, but here is what works, although without selenium:

1- Find out the XML request by checking the network analysis tools in your browser

2 = Once you have defined the query, update it using Python queries or urllib2 modules. I personally recommend queries because of my additional features; for me, the most important queries were. Session.

You can find a lot of help and related posts about these two steps.

Hope this ever helps someone.

Answer 2

I once intercepted some ajax calls by injecting javascript into the page using selenium. The bad side of the story is that selenium can sometimes be, say, "fragile." Therefore, for no reason, I received selenium exceptions during this injection.

In any case, my idea was to intercept the XHR calls and establish its response to the new dom element that I created, which I could manipulate from selenium. In the interception condition, you can even use the URL that made the request to just intercept the one you really want (self._url)

btw, did i get an idea from intercepting all ajax calls?

Perhaps this helps.

 browser.execute_script(""" (function(XHR) { "use strict"; var element = document.createElement('div'); element.id = "interceptedResponse"; element.appendChild(document.createTextNode("")); document.body.appendChild(element); var open = XHR.prototype.open; var send = XHR.prototype.send; XHR.prototype.open = function(method, url, async, user, pass) { this._url = url; // want to track the url requested open.call(this, method, url, async, user, pass); }; XHR.prototype.send = function(data) { var self = this; var oldOnReadyStateChange; var url = this._url; function onReadyStateChange() { if(self.status === 200 && self.readyState == 4 /* complete */) { document.getElementById("interceptedResponse").innerHTML += '{"data":' + self.responseText + '}*****'; } if(oldOnReadyStateChange) { oldOnReadyStateChange(); } } if(this.addEventListener) { this.addEventListener("readystatechange", onReadyStateChange, false); } else { oldOnReadyStateChange = this.onreadystatechange; this.onreadystatechange = onReadyStateChange; } send.call(this, data); } })(XMLHttpRequest); """)