Dynamic web page scraping using python

Question

I am running below code, but i am getting an empty list. Can you please help me in find out the issue.

execution: xvfb-run python dynamic_scrapy.py

import sys
from PyQt4.QtGui import QApplication
from PyQt4.QtCore import QUrl
from PyQt4.QtWebKit import QWebPage
import bs4 as bs


class Client(QWebPage):
    def __init__(self, url):
        self.app = QApplication(sys.argv)
        QWebPage.__init__(self)
        self.loadFinished.connect(self.on_page_load)
        self.mainFrame().load(QUrl(url))
        self.app.exec_()
    def on_page_load(self):
        self.app.quit()

url = "https://pythonprogramming.net/parsememcparseface/"
client_response = Client(url)
source = client_response.mainFrame().toHtml()
soup = bs.BeautifulSoup(source, 'lxml')
print(soup)
js_test = soup.find_all('p', class_='jstest')
print(js_test)

As i see, this is not a dynamic content as class jstest exists in the source. — Satish Prakash Garg
– Satish Prakash Garg, Commented Apr 3, 2017 at 10:26
if you do view page source. Please find out these code: <p class='jstest' id='yesnojs'>y u bad tho?</p> <script> document.getElementById('yesnojs').innerHTML = 'Look at you shinin!'; </script> — Venkatesh Panabaka
– Venkatesh Panabaka, Commented Apr 3, 2017 at 10:29
if i do print(source) i am getting error: UnicodeEncodeError: 'ascii' codec can't encode character u'\u1d90' in position 6781: ordinal not in range(128) — Venkatesh Panabaka
– Venkatesh Panabaka, Commented Apr 3, 2017 at 10:31

Satish Prakash Garg · Accepted Answer · 2017-04-03 10:46:20Z

1

You need to convert the QString into string to pass it into BeautifulSoup. You can do something like this :

import sys
from PyQt4.QtGui import QApplication
from PyQt4.QtCore import QUrl
from PyQt4.QtWebKit import QWebPage
import bs4 as bs

class Client(QWebPage):
    def __init__(self, url):
        self.app = QApplication(sys.argv)
        QWebPage.__init__(self)
        self.loadFinished.connect(self.on_page_load)
        self.mainFrame().load(QUrl(url))
        self.app.exec_()
    def on_page_load(self):
        self.app.quit()

url = "https://pythonprogramming.net/parsememcparseface/"
client_response = Client(url)

source = client_response.mainFrame().toHtml()
source_utf = unicode(source.toUtf8(), encoding="UTF-8") # Added
soup = bs.BeautifulSoup(source_utf, 'lxml')
js_test = soup.find_all('p', class_='jstest')
print(js_test)

This will result in :

[<p class="jstest" id="yesnojs">Look at you shinin!</p>]

answered Apr 3, 2017 at 10:46

Satish Prakash Garg

2,2232 gold badges19 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

Venkatesh Panabaka Over a year ago

getting error: NameError: name 'unicode' is not defined

Satish Prakash Garg Over a year ago

You using python3 ?

Venkatesh Panabaka Over a year ago

can you please help me.

Venkatesh Panabaka Over a year ago

my new question is stackoverflow.com/questions/43188569/…

Venkatesh Panabaka Over a year ago

can you please help me with these. i am struggling last five days

|

Collectives™ on Stack Overflow

Dynamic web page scraping using python

1 Answer 1

10 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

10 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related