trigger js event during the python web scraping

Question

# ./scrape.py
from lxml import html
import requests

url = "http://www.my-target-url.com"

page = requests.get(url)
# can I insert some js event codes to execute here?
tree = html.fromstring(page.content)
print tree.xpath("/html/to/target/data/text()")[0]

I made this for scraping target page which has several buttons to change data. I want all the data which can be derived from those buttons. I searched for the way to send POST or GET with parameters. There seems no other way but to trigger js event from that web page though(like I can do in chrome developer's console).

Is there any way I can execute js events to change response object for needed data in this python code? should I use another library other than requests? or some other way which I can search for?(like make some web browser object behind and do with it? then what can help?)

Ibrahim · Accepted Answer · 2015-12-27 16:06:23Z

1

Short answer, no.

All the javascript events are handled by the browser's js engine. This means you will need a javascript engine too to handle and execute the scripts and trigger the events.

answered Dec 27, 2015 at 16:06

Ibrahim

2,06216 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Jeongbin Kim Over a year ago

Could you recommend something that can work like you said? any useful js or python library?

theideasmith Over a year ago

Use phantomjs or selenium-webdriver

Ibrahim Over a year ago

I don't think there is an easy way of doing this. Since you want to create a scraper and i'm assuming that it will work in background, maybe you can find what requests are made when you click a button and do the same requests with the same POST/GET params in your script.

Jeongbin Kim Over a year ago

Thank you all! I could have a glimpse of the background system. I first tried phantomjs but it was very slow and sometimes not working even example codes(I don't know why untill now) but selenium was great. It must have decreased my wasteful time if I knew it eariler. Selenium is very easy to use and understand. Thank you @theideasmith.

theideasmith Over a year ago

This is an interesting problem that should be solved with a well tested DSL - and in fact it is. Check out Wombat, felipecsl/wombat · GitHub github.com/felipecsl/wombat

Collectives™ on Stack Overflow

trigger js event during the python web scraping

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related