How to grab headers in python selenium-webdriver

Question

I am trying to grab the headers in selenium webdriver. Something similar to the following:

>>> import requests >>> res=requests.get('http://google.com') >>> print res.headers 

I need to use Chrome webdriver because it supports flash and some other things that I need to check the webpage. Here is what I still have in Selena:

 from selenium import webdriver driver = webdriver.Chrome() driver.get('https://login.comcast.net/login?r=comcast.net&s=oauth&continue=https%3A%2F%2Flogin.comcast.net%2Foauth%2Fauthorize%3Fclient_id%3Dxtv-account-selector%26redirect_uri%3Dhttps%3A%2F%2Fxtv-pil.xfinity.com%2Fxtv-authn%2Fxfinity-cb%26response_type%3Dcode%26scope%3Dopenid%2520https%3A%2F%2Flogin.comcast.net%2Fapi%2Flogin%26state%3Dhttps%3A%2F%2Ftv.xfinity.com%2Fpartner-success.html%26prompt%3Dlogin%26response%3D1&reqId=18737431-624b-44cb-adf0-2a85d91bd662&forceAuthn=1&client_id=xtv-account-selector') driver.find_element_by_css_selector('#user').send_keys(' [e-mail] ') driver.find_element_by_css_selector('#passwd').send_keys('XXY') driver.find_element_by_css_selector('#passwd').submit() print driver.headers ### How to do this? 

I saw some other answers that recommend running the entire selenium server to get this information ( https://github.com/derekargueta/selenium-profiler ). How do I get it using something similar to the above using Webdriver?

Answer 1

Unfortunately, you cannot receive this information from the Selenium web server, and you cannot at any time in the near future. Excerpt from a very lengthy discussion on this subject :

This function will not be executed.

The essence of the main reason is that from what I am collecting from the discussion that the web server is designed to “manage the browser” and extend the API beyond this main goal, according to the developers, the overall quality and reliability of the API .

One potential workaround I've seen in a number of places, including the one mentioned above, is to use the BrowserMob Proxy , which can be used to capture HTTP content, or can be used with selenium - although the linked example does not use the Selenium Python API. It seems that there is a Python shell for the BrowserMob proxy , but I cannot vouch for its effectiveness, since I have never used it.

Answer 2

You can try Mobilenium , a Python package (still in development) that links BrowserMob Proxy and Selenium.

Usage example:

 >>> from mobilenium import mobidriver >>> >>> browsermob_path = 'path/to/browsermob-proxy' >>> mob = mobidriver.Firefox(browsermob_binary=browsermob_path) >>> mob.get('http://python-requests.org') 301 >>> mob.response['redirectURL'] 'http://docs.python-requests.org' >>> mob.headers['Content-Type'] 'application/json; charset=utf8' >>> mob.title 'Requests: HTTP for Humans \u2014 Requests 2.13.0 documentation' >>> mob.find_elements_by_tag_name('strong')[1].text 'Behold, the power of Requests' 

Answer 3

You can get the header through the log (source from Mma answer )

 from selenium import webdriver import json driver = webdriver.PhantomJS(executable_path=r"your_path") har = json.loads(driver.get_log('har')[0]['message']) # get the log print('headers: ', har['log']['entries'][0]['request']['headers']) 

Answer 4

Now, I suppose, this is very easy. https://pypi.org/project/selenium-wire/ is an extension of selenium. use from seleniumwire import webdriver and act as usual.

Answer 5

You mean the HTTP header data, right? This is actually not the scale of Selenium: Selenium automates browsers. This is true! So, if you cannot do this using your browser (and I don't know about any other), Selenium is the wrong tool to use. However, if you can do this using JavaScript, you can use driver.execute_script(script, *args) as described here .