Is there a way to scrap css values while scraping using python scrapy framework or by using php scraping. any help will be appreaciated
2
-
Google is your friendRafael Almeida– Rafael Almeida2016-09-02 21:20:28 +00:00Commented Sep 2, 2016 at 21:20
-
1please use google for finding such things and use this site if you get any errors while implementing them.Sachith Muhandiram– Sachith Muhandiram2016-09-03 01:43:31 +00:00Commented Sep 3, 2016 at 1:43
Add a comment
|
2 Answers
scrapy.Selector allows you to use xpath to extract properties of HTML elements including CSS.
e.g. https://github.com/okfde/odm-datenerfassung/blob/master/crawl/dirbot/spiders/data.py#L83
(look around that code for how it fits into an entire scrapy spider)
If you don't need web crawling and just html parsing, you can use xpath directly from lxml in python. Another example:
https://github.com/codeformunich/feinstaubbot/blob/master/feinstaubbot.py
Finally, to get at the css from xpath I only know how to do it via css=element.attrib['style'] - this gives you everything inside of the style attribute which you further split by e.g. css.split(';') and then each of those by ':'.
It wouldn't surprise me if someone has a better suggestion. A little knowledge is enough to do a lot of scraping and that's how I would approach it based on previous projects.
Comments
Yes, please check the documentation for selectors basically you've two methods response.xpath() for xpath and response.css() for css selectors. For example, to get a title's text you could do any of the following:
response.xpath('//title/text()').extract_first()
response.css('title::text').extract_first()
Comments
