Navigating to next page of a javascript table using python selenium chrome webdriver

Question

My goal is to loop through each of the pages and scrape the contents into a DataFrame. So far, I can get the first 20 no problem. I cannot seem to figure out how to navigate to the next page on a javascript table using python selenium chrome webdriver. Have tried a few of the solutions (below seems like the closest) but cannot replicate the result. I would post the website but it is one where you need log in credentials to access. Provided a page source screenshot for the relevant ul and li elements

stackoverflow solution tried: How to click on the list of the <li> elements in an <ul> elements with selenium in python?

Am able to get the first 20 rows using: driver.find_element_by_xpath('//*[@id="allPicksId"]/div/div[2]').text

Then, when trying to get the next rows on next table page using the code below I get an exception:

next_table_page = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".ant-pagination-next"))).click()

Yields this exception:
ElementClickInterceptedException: Message: element click intercepted: Element is not clickable at point (546, 996)

Any guidance would be appreciated!

Page Source Photo

Table Page Navigation Bar Photo

did u tried the button instead of li ant-pagination-item-link ? as they shared on same space ..then next attempt would be to hit span area. — simpleApp
– simpleApp, Commented Apr 24, 2021 at 19:52

Y4RD13 · Accepted Answer · 2022-11-18 05:30:33Z

1

I think I figured it out by using the code below:

element = driver.find_element_by_class_name('ant-pagination-next')
driver.execute_script("arguments[0].click();", element)

edited Nov 18, 2022 at 5:30

Y4RD13

9842 gold badges20 silver badges49 bronze badges

answered Apr 24, 2021 at 21:12

geodude

454 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Sullivan · Accepted Answer · 2021-04-24 19:42:43Z

0

you can use for loop to handle this

example:

for I in range(1, 23):
      driver.find_element_by_xpath().click()

Or

you can use the site URL to go next page

see this url

and second url

now you must change the number of pages to go next page

answered Apr 24, 2021 at 19:42

Sullivan

1613 silver badges9 bronze badges

3 Comments

geodude Over a year ago

Unfortunately, there is no page number in the URL. It is all navigated based on the buttons. I plan on just using a loop whenever I figure out how to get to the next table page

Sullivan Over a year ago

send XPath of pages 18, 19, 20, 21, 22

geodude Over a year ago

I got the final page number text extracted and used a while loop to control the amount of clicks for the next page button to terminate once the end page number is reached. Took a little digging but finally got the answer to my question and posted it above.

Collectives™ on Stack Overflow

Navigating to next page of a javascript table using python selenium chrome webdriver

2 Answers 2

Comments

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related