python encoding error when searching a string

Question

I get the following error while trying to search the string below

ERROR:

SyntaxError: Non-ASCII character '\xd8' in file Hadith_scraper.py on line 44, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details

STRING:

دَّثَنَا عَبْدَانُ، قَالَ أَخْبَرَنَا عَبْ

CODE:

arabic_hadith = "دَّثَنَا عَبْدَانُ، قَالَ أَخْبَرَنَا عَبْ"
arabic_hadith.encode('utf8')
print arabic_hadith
if "الجمعة" in arabic_hadith:‎
    day = "5"
else:
    day = ""

Martijn Pieters · Accepted Answer · 2013-12-15 09:23:46Z

4

You have a byte string, not a unicode value. Trying to encode a byte string in Python 2 means that Python will first try to decode it to unicode so that it can then encode.

Use unicode values here instead, and make sure you set the codec at the top of the file first. See PEP 263 - Defining Python Source Code Encodings (which your error message pointed you to).

Note that there is no need to encode to UTF8 here, that'll only complicate text comparisons:

# encoding: utf8
arabic_hadith = u"دَّثَنَا عَبْدَانُ، قَالَ أَخْبَرَنَا عَبْ"
print arabic_hadith
if u"الجمعة" in arabic_hadith:‎
    day = "5"
else:
    day = ""

Rule of thumb: decode bytes from incoming sources (files, network data) to Unicode, process only Unicode in your program, and only encode again for any outgoing data.

I urge you to read:

The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) by Joel Spolsky
The Python Unicode HOWTO
Pragmatic Unicode by Ned Batchelder

before you continue.

edited Dec 15, 2013 at 9:23

answered Dec 15, 2013 at 9:18

Martijn Pieters

1.1m326 gold badges4.2k silver badges3.4k bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

thefourtheye Over a year ago

I always have problems with Unicode... :( I tried this and it failed with the same error :(

Ossama Over a year ago

what if arabic_hadith is a variable? doing this is doesn't solve it: arabic_hadith = u(arabic_hadith)

Martijn Pieters Over a year ago

@Ossama: Nope, it would not. arabic_hadith.decode('utf8') would though.

Martijn Pieters Over a year ago

@Ossama: You need to add a line, at the top (first or second line of the file) starting with #, containing the word coding followed by : or = plus the codec used to save the file.

Martijn Pieters Over a year ago

@Ossama: you could start by reading the link the error message provided. It is hotlinked in my answer.

|

Collectives™ on Stack Overflow

python encoding error when searching a string

1 Answer 1

10 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

10 Comments

Your Answer

Sign up or log in

Post as a guest

Related