0

The aim of my program is to enter a set of postcodes into a flatshare website (https://www.spareroom.co.uk/flatshare-house-share/uk/) and return the number of rooms wanted and rooms available for each postcode.

My problem is that after clicking a radio button (rooms for rent) the program is meant to re-enter the postcode and search for results after having done the same with the rooms wanted button. It works for the rooms wanted button, but when I reuse the same send_keys() command, it doesn't seem to work, generating the following error...

Traceback (most recent call last):
  File "c:\Users\SS\Dropbox\PC\Programming\Python Programming\Web Automation\WebScraper.py", line 34, in <module>
    search_box.send_keys(postcode, Keys.ENTER)
  File "C:\Users\SS\AppData\Local\Programs\Python\Python312\Lib\site-packages\selenium\webdriver\remote\webelement.py", line 303, in send_keys
    self._execute(
  File "C:\Users\SS\AppData\Local\Programs\Python\Python312\Lib\site-packages\selenium\webdriver\remote\webelement.py", line 572, in _execute
    return self._parent.execute(command, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\SS\AppData\Local\Programs\Python\Python312\Lib\site-packages\selenium\webdriver\remote\webdriver.py", line 448, in execute
    self.error_handler.check_response(response)
  File "C:\Users\SS\AppData\Local\Programs\Python\Python312\Lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 232, in check_response    
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: stale element not found

Below is a snippet of my code for this specific task:

wanted_button = driver.find_element(By.ID, "flatshare_type-wanted")
driver.execute_script("arguments[0].checked = true;", wanted_button)
search_box = driver.find_element(By.ID, "search_by_location_field")
print(wanted_button)
search_box.send_keys(postcode, Keys.ENTER)
time.sleep(5) 

rent_button = driver.find_element(By.ID, "flatshare_type-offered")
driver.execute_script("arguments[0].checked = true;", rent_button)
search_box.send_keys(postcode, Keys.ENTER)
print(rent_button)
time.sleep(5)

Potentially useful background information:

I've had a number of errors such as...

  • The click(); method was not working for the radio buttons so I tried to fix it using a Python version of the execute_script() method that someone suggested in Java online
  • Sometimes the whole automation seems to close within a few seconds, even though it's not programmed to do so, but when I rerun it, without altering the code, it suddenly starts working again
  • There's another error that continually pops up, before it even reaches this part of the program, because it comes before the error listed above. It didn't seem to be causing issues, but I'm not sure if it is affecting this part of the code for some reason. It looks like this...

DevTools listening on ws://127.0.0.1:57005/devtools/browser/1f515eee-5ae0-4407-a2bc-4d42cd7bb3e6

[34088:37684:0512/205949.001:ERROR:services\device\usb\usb_descriptors.cc:146] Failed to read length for configuration 1.
[34088:37684:0512/205949.037:ERROR:services\device\usb\usb_descriptors.cc:146] Failed to read length for configuration 2.
[34088:37684:0512/205949.037:ERROR:services\device\usb\usb_descriptors.cc:105] Failed to read all configuration descriptors. Expected 3, got 1.
[34088:37684:0512/205949.037:ERROR:components\device_event_log\device_event_log_impl.cc:202] [20:59:49.038] USB: usb_device_win.cc:95 Failed to read descriptors from \\?\usb#vid_0b95&pid_1790#00a226a5#{a5dcbf10-6530-11d2-901f-00c04fb951ed}

I'm a beginner to Python, so any help would be greatly appreciated, thanks.

3 Answers 3

1

The problem is that you are storing an element in search_box and then using it after executing the search, triggering the stale element exception. The way I generally get around this problem is to always fetch any element before using it unless I'm executing steps in sequence and I know that the page/partial page doesn't change during those actions. I wrote a simple script that demonstrates this using some of your steps.

from selenium import webdriver

from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait

driver = webdriver.Chrome()

url = "https://www.spareroom.co.uk/flatshare-house-share/uk/"
driver.get(url)

post_codes = ["B16", "B20"]

wait = WebDriverWait(driver, 10)
# dismiss cookie prompt
wait.until(EC.element_to_be_clickable((By.ID, "onetrust-accept-btn-handler"))).click()

for post_code in post_codes:
    print("\r\n", post_code, "=================")

    wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "label[title='Rooms in existing shares']"))).click()
    search_location = wait.until(EC.element_to_be_clickable((By.ID, "search_by_location_field")))
    search_location.clear()
    search_location.send_keys(post_code)
    wait.until(EC.element_to_be_clickable((By.ID, "submitButton"))).click()
    rooms = wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "h2.listing-card__title")))
    print("\r\nROOMS")
    for room in rooms:
        print(room.text)

    wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "label[title='Potential flatmates and tenants seeking accommodation']"))).click()
    search_location = wait.until(EC.element_to_be_clickable((By.ID, "search_by_location_field")))
    search_location.clear()
    search_location.send_keys(post_code)
    wait.until(EC.element_to_be_clickable((By.ID, "submitButton"))).click()
    tenants = wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "h2.listing-card__title")))
    print("\r\nTENANTS")
    for tenant in tenants:
        print(tenant.text)

driver.quit()

This prints

 B16 =================

ROOMS
Studios Flats Edgbaston B16 perfect for couples!!
Luxury Supported Accommodation rooms Edgbaston   
Double rooms for DSS tenants Edgbaston
All bills included - Large Self contained studio 
Spacious furnished room in a Shared House B16     
Affordable rooms in student house share 2025-2026 
Quiet n Large en-suite 10 minutes from City Centre
Large Double in Rotton Park
Affordable Available Big Rooms Edgbaston    
A City centre room available. All bills inc.
Professional houseshare 1 Single Room May 2025

TENANTS
Professional looking for en-suite room
Professional male seeking en-suite room in Birming
I am a professional male aged 25. I am lookin...
Professional Male seeks a double room
Female looking for a Female House Share
Seeking room in shared house with fun professional
Student Female seeks a single or double room
Professional female
Shared apartment hunt
33 year old Male seeks a single or double room

 B20 =================

ROOMS
Room in Shared Apartment Available in Birmingham
Modern 5 Bedroom House Close To One Stop!
Luxurious New Single Room with En-suite furnished
Bedsite available inclusive of most bills
One Double Room in Two Bed Room Flat.
Room in Shared Apartment Available in Birmingham
Large double room, all bills included, furnished
Newly refurbished property in Handsworth
Cosy, bright single or double room for a female
Shared accommodation To rent
Good Size Single Room

TENANTS
Professional looking for en-suite room
Professional male seeking en-suite room in Birming
I am a professional male aged 25. I am lookin...
Professional Male seeks a double room
Female looking for a Female House Share
Seeking room in shared house with fun professional
Student Female seeks a single or double room
Professional female
Shared apartment hunt
33 year old Male seeks a single or double room
Sign up to request clarification or add additional context in comments.

2 Comments

You did greate work
i saw your bio you was SDE in microsoft wow
0

I think I might see what’s going on here. You're getting a StaleElementReferenceException, right? That usually happens when the element you’re trying to interact with is no longer attached to the page — maybe because the page has refreshed or the DOM has changed after switching the radio button.

After selecting the "Rooms Wanted" option and submitting the search, are you sure the search_box element is still the same? Could it be that the page reloads or rerenders that part of the DOM when the radio button is changed?

You should try to re-find the search_box element after switching to the second radio button like this:

rent_button = driver.find_element(By.ID, "flatshare_type-offered")
driver.execute_script("arguments[0].checked = true;", rent_button)
search_box = driver.find_element(By.ID, "search_by_location_field")
search_box.send_keys(postcode, Keys.ENTER)

Comments

0

You can click radio buttons to activate them. Be sure to use WebDriverWait to [try to] ensure that the element(s) you're looking have been rendered.

With a constant postcode (for testing purposes) consider this:

from selenium.webdriver import Chrome, ChromeOptions
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.remote.webelement import WebElement
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.remote.webdriver import WebDriver


URL = "https://www.spareroom.co.uk/flatshare-house-share/uk/"

QTYPES = [
    ("Rooms for rent", "flatshare_type-offered"),
    ("Rooms wanted", "flatshare_type-wanted"),
]

POSTCODE = "SW19"


def click(driver: WebDriver, e: WebElement) -> None:
    ActionChains(driver).move_to_element(e).click().perform()


def accept_cookies(driver: WebDriver) -> None:
    wait = WebDriverWait(driver, 5)
    ec = EC.presence_of_element_located
    loc = (By.ID, "onetrust-accept-btn-handler")
    try:
        click(driver, wait.until(ec(loc)))
    except Exception:
        pass


def do_search_type(driver: WebDriver, search_type: str) -> None:
    wait = WebDriverWait(driver, 5)
    ec = EC.presence_of_element_located
    loc = (By.ID, search_type)
    click(driver, wait.until(ec(loc)))


def do_query_input(driver: WebDriver, data: str) -> None:
    wait = WebDriverWait(driver, 5)
    ec = EC.presence_of_element_located
    loc = (By.ID, "search_by_location_field")
    input_ = wait.until(ec(loc))
    input_.clear()
    input_.send_keys(data)


def do_submit(driver: WebDriver) -> None:
    wait = WebDriverWait(driver, 5)
    ec = EC.presence_of_element_located
    loc = (By.ID, "submitButton")
    button = wait.until(ec(loc))
    click(driver, button)


def do_search(driver: WebDriver, search_type: str, postcode: str) -> None:
    do_search_type(driver, search_type)
    do_query_input(driver, postcode)
    do_submit(driver)


def get_nresults(driver: WebDriver) -> str:
    try:
        wait = WebDriverWait(driver, 5)
        ec = EC.presence_of_element_located
        loc = (By.CSS_SELECTOR, "p.navcurrent")
        return wait.until(ec(loc)).find_elements(By.TAG_NAME, "strong")[-1].text
    except Exception:
        pass
    return "Unable to get results"


if __name__ == "__main__":
    options = ChromeOptions()
    options.add_argument("--headless=new")
    with Chrome(options) as driver:
        driver.get(URL)
        accept_cookies(driver)
        for label, qtype in QTYPES:
            do_search(driver, qtype, POSTCODE)
            print(label, get_nresults(driver))

Output:

Rooms for rent 260
Rooms wanted 1000+

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.