Scrolling a page using Selenium Webdriver

Question

I have a dynamic page that loads products when the user scrolls down the page. I want to get the total number of products displayed on the display page. I am currently using the following code to get to the bottom until all products are displayed.

elems = WebDriverWait(self.driver, 30).until(EC.presence_of_all_elements_located((By.CLASS_NAME, "x"))) print len(elems) a = len(elems) self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") time.sleep(4) elem1 = WebDriverWait(self.driver, 30).until(EC.presence_of_all_elements_located((By.CLASS_NAME, "x"))) b = len(elem1) while b > a: self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") time.sleep(4) elem1 = WebDriverWait(self.driver, 30).until(EC.presence_of_all_elements_located((By.CLASS_NAME, "x"))) a = b b = len(elem1) print b 

This works well, but I want to know if there is a better way to do this?

Answer 1

You can easily perform this action using this line of code

 driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") 

And if you want to scroll down forever, you should try this.

 from selenium import webdriver from selenium.webdriver.common.keys import Keys import time driver = webdriver.Firefox() driver.get("https://twitter.com/BarackObama") while True: driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") time.sleep(3) 

I'm not sure if time.sleep (x value) is loading data, I take more time or less .. for more information, please check the official Doc page

enjoy:)

Answer 2

I think you could condense your code before this:

 prior = 0 while True: self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") current = len(WebDriverWait(self.driver, 30).until(EC.presence_of_all_elements_located((By.CLASS_NAME, "x")))) if current == prior: return current prior = current 

I skipped all the same lines, moving them all to a loop, which required a while True: to while True: and transfer the condition to the loop (because, unfortunately, Python has no do-while ).

I also sent sleep and print instructions - I'm not sure what their purpose is, but on my own page I found that the same number of elements load whether I sleep between scrolls or not. Also, in my own case, I don’t need to know the score at any time, I just need to know when it has exhausted the list (but I added it to the return variable so that you can get the final score if you happen If you really want to print an interim count , you can print the current text immediately after it is assigned in a loop.

Answer 3

If you have no idea how many elements can be added to the page, but you just want to get all of them, it might be nice to loop:

• scroll down as described above
• Wait a few seconds.
• save page source size (xxx.page_source)
• if the page source size is larger than the saved size of the last page source, go back and scroll down a little more

I believe that the screenshot size may work fine, depending on the page you are loading, but this works in my current program.