4

I've got to find the images in a html source code. I'm using regex instead of html.parser because I know it better, but if you can explain to me how to use html parsing like you would a child, I'll be happy to go down that road too.

Can't use beautifulsoup, wish I could, but I got to learn to do this the hard way.

I've read through a lot of questions and answers on here on regex and html (example) so I'm aware of the feelings on this topic.

But hear me out!

Here's my coding attempt (Python 3):

import urllib.request
import re

website = urllib.request.urlopen('http://google.com')
html = website.read()
pat = re.compile (r'<img [^>]*src="([^"]+)')
img = pat.findall(html)

I double checked my regex on regex101.com and it works at finding the img link, but when I run it on IDLE, I get a syntax error and keeps highlighting the caret. Why?

I'm headed in the right direction... yes?

update: Hi, I was thinking may be I get short quick answer, but it seems I may touched a nerve in the community.

I am definitely new and terrible at programming, no way around that. I've been reading all the comments and I really appreciate all the help and patience users have shown me.

Improve this question
18
  • 1
    You're getting a syntax error because... this is invalid syntax (hint: re.compile expects a string). But you should just take a look at the BeautifulSoup html parser, there's enough examples on here and elsewhere that should get you started. Commented Oct 20, 2013 at 12:31
  • 3
    @user2799617 The person has asked a valid question, showed us what he's tried, and checked it on regex101 (which we need a link of). I highly doubt that he has done anything wrong. Commented Oct 20, 2013 at 12:35
  • 1
    @pythonintraining For the gz issue, I guess you're using Windows. Install a utility like 7Zip. Commented Oct 20, 2013 at 12:37
  • 2
    @user2799617 Not always. Commented Oct 20, 2013 at 12:37
  • 2
    hey user2799617, you don't need to ride me, i already ride myself hard enough. i thought the point of stackoverflow was to help people like me, go to reddit or craigslist if you want to keep on ranting. Commented Oct 20, 2013 at 12:39

3 Answers 3

Reset to default
3

There is nothing wrong with the regex, you are missing two things:

  1. Python does not have a regex type, so you have to wrap it in a string. Use a raw string so that the string is passed as-is to the regex compiler, without any escape interpretation
  2. The result of the .read() call is a byte sequence, not a string. So you need a byte sequence regex.

The second one is Python3-specific (and I see that you are using Py3)

Putting all together, just fix the aforementioned line like this:

pat = re.compile (rb'<img [^>]*src="([^"]+)')

r stands for raw and b for byte sequence.

Additionally, test on a website that actually embeds images in <img> tags, like http://stackoverflow.com. You will not find anything when processing http://google.com

Here we go:

Python 3.3.2+
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib.request
>>> import re
>>> website = urllib.request.urlopen('http://stackoverflow.com/')
>>> html = website.read()
>>> pat = re.compile (rb'<img [^>]*src="([^"]+)')
>>> img = pat.findall(html)
>>> img
[b'https://i.sstatic.net/tKsDb.png', b'https://i.sstatic.net/dmHl0.png', b'https://i.sstatic.net/dmHl0.png', b'https://i.sstatic.net/tKsDb.png', b'https://i.sstatic.net/6QN0y.png', b'https://i.sstatic.net/tKsDb.png', b'https://i.sstatic.net/L8rHf.png', b'https://i.sstatic.net/tKsDb.png', b'http://pixel.quantserve.com/pixel/p-c1rF4kxgLUzNc.gif']
Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Thanks! I knew I was close!
1

Instead of using urllib, I used requests, you can download it from here. They do the same thing, I just like requests better since it has a better API. The regex string is only slightly changed. \s is just added in case there are a few whites spaces before the img tag. You were headed in the right direction. You can find out more about the re module here.

Here is the code

import requests
import re

website = requests.get('http://stackoverflow.com//')
html = website.text
pat = re.compile(r'<\s*img [^>]*src="([^"]+)')
img = pat.findall(html)

print img

And the output:

[u'https://i.sstatic.net/tKsDb.png', u'https://i.sstatic.net/L8rHf.png', u'https://i.sstatic.net/tKsDb.png', u'https://i.sstatic.net/Ryr18.png', u'https://i.sstatic.net/ASf0H.png', u'https://i.sstatic.net/tKsDb.png', u'https://i.sstatic.net/tKsDb.png', u'https://i.sstatic.net/tKsDb.png', u'https://i.sstatic.net/Ryr18.png', u'https://i.sstatic.net/VgvXl.png', u'https://i.sstatic.net/tKsDb.png', u'https://i.sstatic.net/tKsDb.png', u'https://i.sstatic.net/tKsDb.png', u'https://i.sstatic.net/tKsDb.png', u'https://i.sstatic.net/6QN0y.png', u'http://pixel.quantserve.com/pixel/p-c1rF4kxgLUzNc.gif']
Improve this answer

1 Comment

I will add one suggestion. This answer is good. The question would have been valid without any code to retrieve a web page. In the future, it might be worthwhile to make a function that finds what you want from a string or array of bytes. Then the function has only a single concern, finding a list of images.
0

re.compile (r'<img [^>]*src="([^"]+)')

you are missing the quotation marks (single or double) around the pattern

Improve this answer

8 Comments

"and just to be sure it's good to escape quotation marks within the expresion" - what? That's more than wrong in this case...
agreed, but thanks for catching the missing quotation marks. now my error reads as: TypeError: can't use a string pattern on a bytes-like object
it's a general remark regarding regex. in his case of parsing html he should catch both single and double quotation marks, but that is his job to do
@mislav do you know what the r in front of the string means? "escaping" the quotation marks should only be done if they actually need to be escaped. Your regex matches \" instead of just the ".
i'm changing the answer just not do derail someone in the future to just include the remark about the missing quotes. worrying about quotes inside the regex for html is a whole new issue
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.