Scrolling inside html table with selenium and python

Question

I'm trying to scroll all the way down inside an HTML table, it isn't the "main" scroll of the webpage, but just the scroll of the HTML table. I plan to just scroll all the way down and then find the certain html tags I want. This is my script at the moment, but the scrolling would just work if I wanted to scroll on the main document.

Essentially I'm trying to retrieve all the data from the main table after scrolling, but there is an additional issue as the table only loads X amount of rows given the dimensions of the window browser of the table, I want to able to append all of them together.

url = "https://governmentofbc.maps.arcgis.com/home/item.html?id=b8a2b437ccc24f04b975f76df6814cb1#data"
driver = webdriver.Chrome()
driver.get(url)
last_height = driver.execute_script("return document.body.scrollHeight")
while True:
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(3)
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

soup = BeautifulSoup(driver.page_source, "html.parser")

Do you exclusively have to use selenium? From what I see you can easily get what you need using the requests library, no more worrying with scrolling. — West
– West, Commented Dec 1, 2020 at 8:36

Timeler · Accepted Answer · 2020-12-01 07:46:42Z

1

Had the same problem, therefore i can tell you what worked for me:

element = driver.find_element_by_xpath("the body you want to scroll in")
for i in range(60):
driver.execute_script("arguments[0].scrollBy(0, 500)", element)
time.sleep(2)

you can adjust those "60" depending on how far you need to scroll and how big the site is. you can also replace the time.sleep() by driver.implicitly_wait(2)

answered Dec 1, 2020 at 7:46

Timeler

3871 silver badge11 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

kjay Over a year ago

does this work when the table isn't entirely loaded? For my example, <divs> are appended as you scroll in the table.

Timeler Over a year ago

yeah, only the body itself where you wanna scroll needs to be defined and visible, the table will load while scrolling i assume

Timeler Over a year ago

pretty strange, it worked for me quite well

kjay Over a year ago

Can you adjust your answer so I see what you exactly did?

Timeler Over a year ago

i meant in my other script, sorry, i didnt try it out in this case, might do that later

Collectives™ on Stack Overflow

Scrolling inside html table with selenium and python

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related