Selenium python code to click button on interactive map for web scraping not working

Question

I am trying to web scrape data from an interactive map using Selenium in python. I have been having difficulties with the code to click certain buttons to get to the data. The first "click" works fine, but the second is not working. I have tried adjusting the time and nothing is working. I want to be able to do that second click. Any help would be appreciated, thank you. Here is my code:

!apt-get update # to update ubuntu to correctly run apt install
!apt install chromium-chromedriver
!cp /usr/lib/chromium-browser/chromedriver /usr/bin
import sys
sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')
from selenium import webdriver
from selenium.webdriver.common.keys import Keys  
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import Select
import os
from bs4 import BeautifulSoup
import re
import pandas as pd
import sys
import re
import csv
import time
import shutil
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
wd = webdriver.Chrome('chromedriver',chrome_options=chrome_options) 

from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium import webdriver
import time
wd.maximize_window()
wd.get("https://hazards.geoplatform.gov/portal/apps/MapSeries/index.html?appid=ddf915a24fb24dc8863eed96bc3345f8")
wd.execute_script("arguments[0].click();", WebDriverWait(wd, 20).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="nav-bar"]/div/div[1]/ul/li[4]/a'))))
wd.execute_script("arguments[0].click();", WebDriverWait(wd, 20).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="widgets_About_Widget_34"]/div/div/div/div[10]/font/a'))))

The error occurs on the last line, and here is the full error:

TimeoutException Traceback (most recent call last) in () 7 wd.get("https://hazards.geoplatform.gov/portal/apps/MapSeries/index.html?appid=ddf915a24fb24dc8863eed96bc3345f8") 8 wd.execute_script("arguments[0].click();", WebDriverWait(wd, 20).until(EC.element_to_be_clickable((By.XPATH, '//[@id="nav-bar"]/div/div1/ul/li[4]/a')))) ----> 9 wd.execute_script("arguments[0].click();", WebDriverWait(wd, 60).until(EC.element_to_be_clickable((By.XPATH, '//[@id="widgets_About_Widget_34"]/div/div/div/div[10]/font/a')))) 10 11

/usr/local/lib/python3.7/dist-packages/selenium/webdriver/support/wait.py in until(self, method, message) 78 if time.time() > end_time: 79 break ---> 80 raise TimeoutException(message, screen, stacktrace) 81 82 def until_not(self, method, message=''):

TimeoutException: Message:

Here is the image of what I want to click: "Census Tracts Table" Click to see image

Please let me know if this resolved your problem.

Prophet
– Prophet

2021-08-03 15:49:24 +00:00
Commented Aug 3, 2021 at 15:49 — Prophet
– Prophet, Commented Aug 3, 2021 at 15:49
Hello, this unfortunately did not resolve my problem.

Jake Primack
– Jake Primack

2021-08-03 16:04:19 +00:00
Commented Aug 3, 2021 at 16:04 — Jake Primack
– Jake Primack, Commented Aug 3, 2021 at 16:04

Prophet · Accepted Answer · 2021-08-03 15:42:38Z

1

There are several issues with your code.

The main blocker avoiding you from access the second element is because it is inside an iframe, so to acceess it you first have to switch to that iframe.
Your locators are bad.
Don't click elements with JavaScript until you have no alternative.
In headless mode you have to define the screen size.
wd.maximize_window() is used for normal mode, not headless.
So you code should be something like this:

import sys
sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')
from selenium import webdriver
from selenium.webdriver.common.keys import Keys  
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import Select
import os
from bs4 import BeautifulSoup
import re
import pandas as pd
import sys
import re
import csv
import time
import shutil
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--window-size=1920,1080')
chrome_options.add_argument('--disable-dev-shm-usage')
wd = webdriver.Chrome('chromedriver',chrome_options=chrome_options) 

from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium import webdriver
import time
#wd.maximize_window()
wd.get("https://hazards.geoplatform.gov/portal/apps/MapSeries/index.html?appid=ddf915a24fb24dc8863eed96bc3345f8")
wait.until(EC.visibility_of_element_located((By.XPATH, "//li[contains(@class,'entry  visible')]//a[contains(text(),'Data Download Tool')]"))).click()
wait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it(driver.find_element_by_css_selector("div[class='mainMediaContainer active']>iframe")))
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div[title='Download']"))).click()

edited Aug 3, 2021 at 15:42

answered Aug 3, 2021 at 15:37

Prophet

33.5k28 gold badges58 silver badges90 bronze badges

Sign up to request clarification or add additional context in comments.

13 Comments

Prophet Over a year ago

I have edited this several times. Now it should work. Also I guess this solution is better...

Jake Primack Over a year ago

Hello thank you for your help. However, this code is still not working for me and I am confused with using CSS. I actually found an easier second click than the one I had before. Now the 2 clicks are this: 1. wd.execute_script("arguments[0].click();", WebDriverWait(wd, 20).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="nav-bar"]/div/div[1]/ul/li[4]/a')))) 2. wd.execute_script("arguments[0].click();", WebDriverWait(wd, 60).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="widgets_About_Widget_34"]/div/div/div/div[10]/font/a')))) The first click works, but the second doesn't.

Prophet Over a year ago

Please post the code inside the question to make it readable

Jake Primack Over a year ago

Also, when running the code that you provided, I am getting the following error: MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=35089): Max retries exceeded with url: /session/f98930adcbf12e63b916d1d1def06c8b/element (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f37f30d2d10>: Failed to establish a new connection: [Errno 111] Connection refused'))

Prophet Over a year ago

I'm not sure this caused by my code... Also, again, don't use JavaScript clicks until you have no alternative. You should use driver.click() since only this method mimics user GUI actions

|

pmadhu · Accepted Answer · 2021-08-04 11:22:44Z

0

The Download button is in the 4th iframe. Below code did click on the Download icon.

driver.implicitly_wait(50)
driver.get("https://hazards.geoplatform.gov/portal/apps/MapSeries/index.html? appid=ddf915a24fb24dc8863eed96bc3345f8")
driver.find_element_by_xpath("//div[@id='nav-bar']/div/div/ul/li[4]/a").click()
driver.switch_to.frame(3)
driver.find_element_by_xpath("//div[@settingid='widgets_Query_Widget_32']").click()
driver.find_element_by_xpath("//div[text()='Find census tracts']").click()

answered Aug 4, 2021 at 11:22

pmadhu

3,4332 gold badges14 silver badges24 bronze badges

Collectives™ on Stack Overflow

Selenium python code to click button on interactive map for web scraping not working

The error occurs on the last line, and here is the full error:

2 Answers 2

13 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

The error occurs on the last line, and here is the full error:

2 Answers 2

13 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related