I want to scrape the data on the website https://www.climatechangecommunication.org/climate-change-opinion-map/. I am somewhat familiar with selenium. But the data I need which is below the map and the tooltip on the map is not visible in the source file. I have read some posts about using PhantomJS and others. However, I am not sure where and how to start. Can someone please help get me started.
Thanks, Rexon
You can use this sample code:
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.get("https://www.climatechangecommunication.org/climate-change-opinion-map/")
# switch to iframe
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, "//iframe[@src = 'https://environment.yale.edu/ycom/factsheets/MapPage/2017Rev/?est=happening&type=value&geo=county']")))
# do your stuff
united_states = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//*[@id='document']/div[4]//*[name()='svg']")))
print(united_states.text)
# switch back to default content
driver.switch_to.default_content()
Output:
50%
No
12%
Yes
70%
United States
Screenshot of the element:
Explanantion: first of all, to be able to interact with elements below the map you have to switch to iframe content, otherwise it is not possible to interact with this elements. Then the data below the map is in svg tags, which are also not trivial. To be able to do this, you the sample I have provided.
PS: I have used WebDriverWait in my code. With WebDriverWait your code becomes quickier and stable, since Selenium waits for particular conditions like visibility or clickable of particular element. In the sample code the driver wait at least 10 seconds until expected condition will be satisfied.
