Selenium: Iterating through Groups of Elements

Question

I did this with BeautifulSoup, but it's a little cumbersome, and I'm trying to figure out if I can do this directly with Selenium.

Let's say I have the following HTML that repeats several times in the page source with the same elements but with different content:

<div class="person"> <div class="title"> <a href="http://www.url.com/johnsmith/">John Smith</a> </div> <div class="company"> <a href="http://www.url.com/company/">SalesForce</a> </div> </div> 

I need to create a dictionary in which the entry for each person will look:

 dict = {'name' : 'John Smith', 'company' : 'SalesForce'} 

I can easily get Selenium to create a list of the contents of each top-level element by doing:

 driver.find_elements_by_class_name('person') 

But then I can not iterate over the list, because the above method does not limit the scope / source only to the contents of this element.

If I try to do something like this:

 people = driver.find_elements_by_class_name('person') for person in people: print person.find_element_by_xpath['//div[@class="title"]//a').text 

I get the same name over and over again.

I need to make this group into groups, because in my case iterating all over the page and adding each tag individually will not work (infinite scroll, so it will be really inefficient).

Does anyone know if this is possible to do directly in Selenium, and if so, how?

Answer 1

Use find_elements_by_class_name() to get all the blocks and find_element_by_xpath() to get the title and company for each person:

 persons = [] for person in driver.find_elements_by_class_name('person'): title = person.find_element_by_xpath('.//div[@class="title"]/a').text company = person.find_element_by_xpath('.//div[@class="company"]/a').text persons.append({'title': title, 'company': company})