Python decode from a specific string

Question

Someone gave me this string: "Al BaÅ£á¸©ah" (it's probably an Arabic name), and asked me to translate it (and a list of other similar strings) in Python.

This website converts it to "Al Baţḩah" with the operation "UTF8 Decode". I'm trying to do the same in Python, but is what I tried and my results. Most of the examples took a series of unicode bytes. I tried the "detect" to see exactly what it is he was giving me. If it's already UTF8, then I'm not sure what that website is convert it to.

import chardet

byte_string = b"\x61\x62\x63"
decoded_string = byte_string.decode("utf8")
print(decoded_string)

sourceText = "Al BaÅ£á¸©ah"
sourceTextBytes = bytes(sourceText, 'utf-8')
print(chardet.detect(sourceTextBytes))
decoded_string2 = sourceTextBytes.decode("utf")
print("Result2=", decoded_string2)

Output of above:

abc
{'encoding': 'utf-8', 'confidence': 0.9690625, 'language': ''}
Result2= Al BaÅ£á¸©ah

The output is same as the input. I've tried ascic, utf8, etc... as the parms for the decode statement.

Part 2 - Here's another weird one that the solution below didn't work for (these are subdivison names from an ISO document a colleague purchased.)

GÉ™dÉ™bÉ™y

anthony sottile · Accepted Answer · 2020-11-10 04:08:27Z

1

looks like a classic case of mojibake -- in this case it's interpreted using latin1 when it should be UTF-8:

>>> "Al BaÅ£á¸©ah".encode('latin1')
b'Al Ba\xc5\xa3\xe1\xb8\xa9ah'
>>> "Al BaÅ£á¸©ah".encode('latin1').decode('UTF-8')
'Al Baţḩah'

Code for those that want to copy/paste into a program instead of command line:

source_text = "Al BaÅ£á¸©ah"
print("source_text=", source_text)
encoded_source_text = source_text.encode('latin1')
decoded_text = encoded_source_text.decode('UTF-8')
print("decoded_text=", decoded_text)

edited Nov 10, 2020 at 4:08

answered Nov 10, 2020 at 2:22

anthony sottile

72.1k19 gold badges199 silver badges248 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

NealWalters Over a year ago

You're a genius! How did you arrive at that conclusion?

anthony sottile Over a year ago

years and years of sadness

lenz Over a year ago

@NealWalters a bunch of people collected their experience from more years of sadness and cast them into the Python library ftfy (fixes text for you), which can go directly from garbled text: ftfy.fix_text("Al BaÅ£á¸©ah") → 'Al Baţḩah'. It's not error-free (because this task is basically a guessing problem), but it's pretty good.

NealWalters Over a year ago

@lenz - cool, I used that library. I had some that caused an error on the decode solution, but ftfy didn't decode them either. I put a "Part 2" in my question above for a weird one that wouldn't fix.

anthony sottile Over a year ago

your second one is probably cp1252 mojibake: >>> s.encode('cp1252').decode('utf-8') 'Gədəbəy'

|

Collectives™ on Stack Overflow

Python decode from a specific string

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related