How to use Selenium to repeatedly obtain content <Python>?

Question

I am crawling text from here. I need to repeatedly click on "Load More Arguments" to obtain all arguments listed on the page. Here is my code:

try:
    while True:
        link = WebDriverWait(driver, 5).until(
        EC.element_to_be_clickable((By.LINK_TEXT, "Load More Arguments")))

        ActionChains(driver).move_to_element(link).perform()
        link.click()
        time.sleep(3) #wait for the update to occurr so the page loads"new arguments for you"
        print(driver.execute_script("return document.documentElement.outerHTML;"))

        if not (link):
            break
finally:
    None

Here is the error:

File "debate.py", line 42, in <module>
  EC.element_to_be_clickable((By.LINK_TEXT, "Load More Arguments")))
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/selenium-2.44.0-py2.7.egg/selenium/webdriver/support/wait.py", line 71, in until
  raise TimeoutException(message)
selenium.common.exceptions.TimeoutException: Message:

I do obtain the arguments listed on the page but I think my code is a bit strange, especially in the loop. I think I need something to replace "if not" like "not clickable". Could you please give me some suggestion?

Thank you.

Louis · Accepted Answer · 2015-02-07 13:30:50Z

Yep, if not link: is not the condition you want. If this condition were ever to be true, then neither your move_to_element nor the click would work, because there is nothing between them and your if not link: test that can alter link from referring to an element be None (or some other false value). The element reference that link contains could become stale but this kind of issue generates a specific exception. There's no magic that turns a stale reference to None.

In all likelihood, the timeout event that you currently run into is the indication that you are done. I would expect that the button to load more arguments would be unclickable (because it is no longer present or because it is disabled) once you've loaded everything. (I cannot verify this, as the link you provided gets me to a page that does not illustrate the problem.)

If you break on the exception, you can exit your loop. This also takes care of the case where there are no arguments yet or there are not enough arguments to warrant the presence of the button:

from selenium.common.exceptions import TimeoutException

[....]

while True:
    try:
        link = WebDriverWait(driver, 5).until(
            EC.element_to_be_clickable((By.LINK_TEXT, "Load More Arguments")))
    except TimeoutException:
        break
    [...]

A few notes:

The outer try... finally block does not seem to do anything.
Usually, you can move and click a link in one operation link.click() should work without the ActionChains. There are some rather rare cases where you need to perform a move before the click.
You should replace your time.sleep call with an explicit wait that inspects the state of the page to determine when the arguments have finished loading.

Collectives™ on Stack Overflow

How to use Selenium to repeatedly obtain content <Python>?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related