Unable to clear some items from zillow site

Question

I am trying to clear the contents of the zillow site.

Ex- https://www.zillow.com/homedetails/689-Luis-Munoz-Marin-Blvd-APT-508-Jersey-City-NJ-07310/108625724_zpid/

The problem is that I cannot clear the contents of the price and tax history. I thought they were javascript elements loaded when the page loaded, and therefore tried to use selenium, but I still can't get them. Here is what I have tried.

code

phistory = soup.find("div",{"id": "hdp-price-history"}) print phistory 

HTML

 <div class="loading yui3-widget yui3-async-block yui3-complaintstable yui3-hdppricehistory yui3-hdppricehistory-content" id="hdp-price-history"> div class="zsg-content-section zsg-loading-spinner_lg"></div> </div> 

This is the most external element, but it has no elements inside. Also tried soup.find_all("table",class_ = "zsg-table yui3-toggle-content-minimized") , which does not give any.

Answer 1

You can try to wait until the required <table> and becomes visible:

 from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait as wait from selenium.webdriver.support import expected_conditions as EC driver.get("https://www.zillow.com/homedetails/689-Luis-Munoz-Marin-Blvd-APT-508-Jersey-City-NJ-07310/108625724_zpid/") table = wait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, '//div[@id="hdp-price-history"]//table'))) print(table.text) 

Output:

 DATE EVENT PRICE $/SQFT SOURCE 05/03/17 Listed for sale $750,000+159% $534 KELLER WILLIAM... 06/15/11 Sold $290,000-38.3% $206 Public Record 10/14/05 Sold $470,000 $334 Public Record 

You can also analyze it without using BeautifulSoup , for example

 print(table.find_element_by_xpath('.//td[text()="Listed for sale"]/following::span').text) 

Output:

 $750,000 

or

 print(table.find_element_by_xpath('.//td[text()="Sold"]/following::span').text) 

Output:

 $290,000