Geek Answers Handbook

Link scrolling through Selenium Webdriver (Python)

Afternoon. You are currently trying to use the Selenium web editor to scroll through the list of links on a page. In particular, he clicks the link, capturing a line of text from the specified page to write to the file, return and click the next link in the list. The following is what I have:

def test_text_saver(self): driver = self.driver textsave = open("textsave.txt","w") list_of_links = driver.find_elements_by_xpath("//*[@id=\"learn-sub\"]/div[4]/div/div/div/div[1]/div[2]/div/div/ul/li") """Initializing Link Count:""" link_count = len(list_of_links) while x <= link_count: print x driver.find_element_by_xpath("//*[@id=\"learn-sub\"]/div[4]/div/div/div/div[1]/div[2]/div/div/ul/li["+str(x)+"]/a").click() text = driver.find_element_by_xpath("//*[@id=\"learn-sub\"]/div[4]/div/div/div/div[1]/div[1]/div[1]/h1").text textsave.write(text+"\n\n") driver.implicitly_wait(5000) driver.back() x += 1 textsave.close()

When launched, it goes to the start page and ... returns to the main page, and not to the subpage that it should have. Seal x, I see that it increases three times, not one. After that, he also falls. I checked all my xpaths, etc., and also confirmed that it gets the correct count of the number of links in the list.

Any input is much appreciated - it really is just bending my python / automation, as I just fall into both. Thanks in advance!

+3
source share
1 answer

I'm not sure if this will fix the problem, but in general it is better to use WebDriverWait rather than implicitly_wait , since WebDriveWait.until will continue to call the provided function (e.g. driver.find_element_by_xpath ) until the return value is False -ish or not a timeout is reached (for example, 5000 seconds) - at this point it raises the value of selenium.common.execptions.TimeoutException .

 import selenium.webdriver.support.ui as UI def test_text_saver(self): driver = self.driver wait = UI.WebDriverWait(driver, 5000) with open("textsave.txt","w") as textsave: list_of_links = driver.find_elements_by_xpath("//*[@id=\"learn-sub\"]/div[4]/div/div/div/div[1]/div[2]/div/div/ul/li/a") for link in list_of_links: # 2 link.click() # 1 text = wait.until( lambda driver: driver.find_element_by_xpath("//*[@id=\"learn-sub\"]/div[4]/div/div/div/div[1]/div[1]/div[1]/h1").text) textsave.write(text+"\n\n") driver.back()
  • After you click the link, you must wait until the linked URL is loaded. Therefore, the wait.until call is placed immediately after link.click()

  • Instead of using

     while x <= link_count: ... x += 1

    better to use

     for link in list_of_links:

    We believe that this improves readability. Moreover, you really do not need to worry about the number x , all you are really interested in is a loop through the links, which makes the for-loop .

+3
source

More articles:


All Articles