Return html code of dynamic page using selenium

Question

I'm trying to crawl this website, problem is it's dynamically loaded.

Basically I want what I can see from the browser console, not what I see when I right click > show sources.

I've tried some selenium examples but I can't get what I need. The code below uses selenium and get only what you get in right click -> show code. How can I get the content of the loaded page?

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium import webdriver
import time

# Start the WebDriver and load the page
wd = webdriver.Firefox()
wd.get("https://www.leforem.be/particuliers/offres-emploi-recherche-par-criteres.html?exParfullText=&exPar_search_=true&    exParGeographyEdi=true")

# Wait for the dynamically loaded elements to show up
time.sleep(5)

# And grab the page HTML source
html_page = wd.page_source
wd.quit()

# Now you can use html_page as you like

print(html_page)

alecxe · Accepted Answer · 2015-06-17 13:23:30Z

2

You need to explicitly wait for the search results to appear before getting the page source:

from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC


wd = webdriver.Firefox()
wd.get("https://www.leforem.be/particuliers/offres-emploi-recherche-par-criteres.html?exParfullText=&exPar_search_=true&    exParGeographyEdi=true")

wd.switch_to.frame("cible")

wait = WebDriverWait(wd, 10)
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'td.resultatIntitule')))

print(wd.page_source)

edited Jun 17, 2015 at 13:23

answered Jun 17, 2015 at 12:31

alecxe

476k127 gold badges1.1k silver badges1.2k bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

dkx22 Over a year ago

Thx I didn't know that. However it's hanging now and I get an error like

selenium.common.exceptions.TimeoutException: Message:  Stacktrace:     at FirefoxDriver.prototype.findElementInternal_ (file:///tmp/tmpnfzF0g/extensions/[email protected]/components/driver-component.js:10299)

So I guess it didn't find it.. But I see it in the elements

alecxe Over a year ago

@dkx22 thanks for trying it out. There is an iframe that you need to switch to - updated.

Collectives™ on Stack Overflow

Return html code of dynamic page using selenium

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related