Do not click on all tabs and do not loop

Question

I am trying to click tabs on a web page as shown below. Unfortunately, it seems that only clicking on some tabs, despite the correct xpath, check Chrome. I can only assume that it does not click all the tabs because the full xpath is not used.

However .. I tried changing xpath:

//div[@class="KambiBC-collapsible-container KambiBC-mod-event-group-container"] To:

//div[@class='KambiBC-event-groups-list']//div[@class="KambiBC-collapsible-container KambiBC-mod-event-group-container"] FOR:

 clickMe = wait(driver, 10).until(EC.element_to_be_clickable((By.XPATH,'(//div[@class="KambiBC-collapsible-container KambiBC-mod-event-group-container"])[%s]' % str(index + 1)))) 

However, the problem persists. I also tried using CSS:

 #KambiBC-contentWrapper__bottom > div > div > div > div > div.KambiBC-quick-browse-container.KambiBC-quick-browse-container--list-only-mode > div.KambiBC-quick-browse__list.KambiBC-delay-scroll--disabled > div > div.KambiBC-time-ordered-list-container > div.KambiBC-time-ordered-list-content > div > div > div.KambiBC-collapsible-container.KambiBC-mod-event-group-container > header 

However, this continues to give me errors ... For:

 clickMe = wait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR,'("#KambiBC-contentWrapper__bottom > div > div > div > div > div.KambiBC-quick-browse-container.KambiBC-quick-browse-container--list-only-mode > div.KambiBC-quick-browse__list.KambiBC-delay-scroll > div > div.KambiBC-time-ordered-list-container > div.KambiBC-time-ordered-list-content > div > div > div > header")[%s]' % str(index + 1)))) 

It should be noted that I want to click all unopened tabs, and I cannot use CSS Selectors to search for a sufficiently specific element, since I believe that this does not allow to narrow the class element in this case.

Is there a way around this problem if you don't click everything?

It should be noted that I use ...

for index in indexes:

 indexes = [index for index in range(len(options))] shuffle(indexes) for index in indexes: 

Is there a more elegant way to use for 1 cycle?

 [import sys sys.exit()][1] 

Full code

Answer 1

This cycle goes through all matches from each league 1 to 1, collecting all the necessary data as necessary. You can collect additional data in each match, the prefix of each request with . and match matching via match.find_element_by_xpath('.//your-query-here') . Let me know if this is a trick!

 import sys, io, os, csv, requests, time from selenium.webdriver.support.ui import WebDriverWait as wait from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.common.by import By from selenium.common.exceptions import TimeoutException from selenium import webdriver driver = webdriver.Chrome() driver.set_window_size(1024, 600) driver.maximize_window() try: os.remove('vtg121.csv') except OSError: pass driver.get('https://www.unibet.com.au/betting#filter/football') time.sleep(1) clickMe = wait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, ('//div[@class="KambiBC-collapsible-container '\ 'KambiBC-mod-event-group-container"]')))) time.sleep(0) xp_opened = '//div[contains(@class, "KambiBC-expanded")]' xp_unopened = '//div[@class="KambiBC-collapsible-container ' \ 'KambiBC-mod-event-group-container" ' \ 'and not(contains(@class, "KambiBC-expanded"))]' opened = driver.find_elements_by_xpath(xp_opened) unopened = driver.find_elements_by_xpath(xp_unopened) data = [] for league in opened: xp_matches = './/li[contains(@class,"KambiBC-event-item")]' matches = league.find_elements_by_xpath(xp_matches) try: # League Name xp_ln = './/span[@class="KambiBC-mod-event-group-header__main-title"]' ln = league.find_element_by_xpath(xp_ln).text.strip() except: ln = None print(ln) for match in matches: # get all the data per 'match group' xp_team1_name = './/button[@class="KambiBC-mod-outcome"][1]//' \ 'span[@class="KambiBC-mod-outcome__label"]' xp_team1_odds = './/button[@class="KambiBC-mod-outcome"][1]//' \ 'span[@class="KambiBC-mod-outcome__odds"]' xp_team2_name = './/button[@class="KambiBC-mod-outcome"][3]//' \ 'span[@class="KambiBC-mod-outcome__label"]' xp_team2_odds = './/button[@class="KambiBC-mod-outcome"][3]//' \ 'span[@class="KambiBC-mod-outcome__odds"]' try: team1_name = match.find_element_by_xpath(xp_team1_name).text except: team1_name = None try: team1_odds = match.find_element_by_xpath(xp_team1_odds).text except: team1_odds = None try: team2_name = match.find_element_by_xpath(xp_team2_name).text except: team2_name = None try: team2_odds = match.find_element_by_xpath(xp_team2_odds).text except: team2_odds = None data.append([ln, team1_name, team1_odds, team2_name, team2_odds]) for league in unopened: league.click() time.sleep(0.5) matches = league.find_elements_by_xpath(xp_matches) try: ln = league.find_element_by_xpath(xp_ln).text.strip() except: ln = None print(ln) for match in matches: try: team1_name = match.find_element_by_xpath(xp_team1_name).text except: team1_name = None try: team1_odds = match.find_element_by_xpath(xp_team1_odds).text except: team1_odds = None try: team2_name = match.find_element_by_xpath(xp_team2_name).text except: team2_name = None try: team2_odds = match.find_element_by_xpath(xp_team2_odds).text except: team2_odds = None data.append([ln, team1_name, team1_odds, team2_name, team2_odds]) with open('vtg121.csv', 'a', newline='', encoding="utf-8") as outfile: writer = csv.writer(outfile) for row in data: writer.writerow(row) print(row) 

Answer 2

OP code without additional import

The error is due to the fact that the XPaths tab site OP does not want to touch. This one has a space. For example, now I can’t find

// * [@id = "KambiBC-contentWrapper__bottom"] / div / div / div / div / div [3] / div 1 / div / div [3] / div [2] / DIV / DIV / DIV [ 2 ] / title

Some time ago, before the game comes out live, I can’t find

// * [@id = "KambiBC-contentWrapper__bottom"] / div / div / div / div / div [3] / div 1 / div / div [3] / div [2] / DIV / DIV / DIV [ 1 ] / title

When I talk about index , I mean the bold part above.

When the game goes live, the tab suddenly turns the index from 2 to 1. (The bold part changes.) In both cases there are spaces: either 1 cannot be found, or 2 cannot be found.

The reason for the presence of a space , I think, is that between them there is no other element that cannot be clicked. See the figure below.

league is the cause of the gap. Thus, whenever the code hits the league index, it does not execute . Since the league button and other tabs switch the league position and live game, the indices are swapped when changing positions. (I think that why I can’t find the Xpath when the bold part is the first, and later cannot find that it is 2.)

Below is a piece of code from OP. You can see that at the end is str (index + 1).

 indexes = [index for index in range(len(options))] # shuffle(indexes) # the OP use shuffle from random. Still 0 and 1 is contained. path = '(//div[@class="KambiBC-collapsible-container KambiBC-mod-event-group-container"])' for index in indexes: # Because there are some indexes are missing because of League button, # nothing can be found at the index and it times out. clickMe = wait(driver, 10).until( EC.element_to_be_clickable((By.XPATH, path + '[%s]' % str(index + 1)))) 

Decision

Try catching a timeout exception to skip this league occupied index. You can also save a counter to exclude only one timeout exception per page. If there is a second timeout, you know that there is something wrong, except for the league button and should stop.

 from selenium import webdriver from selenium.webdriver.support.ui import WebDriverWait from selenium.common.exceptions import TimeoutException from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.common.by import By import time driver = webdriver.Firefox() driver.set_window_size(1024, 600) driver.maximize_window() wait = WebDriverWait driver.get('https://www.unibet.com.au/betting#filter/football') time.sleep(5) options = driver.find_elements_by_xpath("""//*[@id="KambiBC-contentWrapper__bottom"]/div/div/div/div/div[3]/div[1]/div/div[3]/div[2]/div/div/div""") print("Total tabs that we want to open is {}".format(len(options))) indexes = [index for index in range(len(options))] for index in indexes: print(index) try: clickMe = wait(driver, 5).until(EC.presence_of_element_located((By.XPATH, """//*[@id="KambiBC-contentWrapper__bottom"]/div/div/div/div/div[3]/div[1]/div/div[3]/div[2]/div/div/div[{}]/header""".format(str(index+1))))) clickMe.click() except TimeoutException as ex: print("catch you! {}".format(index)) pass