11 
UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 8: character maps to <undefined>

Am seeing the above error in this simple SQL query:

df = pd.read_sql(query,connection)

I tried the query normally on SQL developer and it works perfectly fine. I am really stumped here as to how to specify encoding in a read_sql call.

Am using Python 3.4 and pandas version 0.14.1

Improve this question
14
  • 5
    When you connect() to your database, pass the charset='utf8' parameter. Commented Oct 25, 2016 at 4:45
  • 1
    What is the encoding of the data in the database? Commented Oct 25, 2016 at 4:55
  • 1
    to get encoding SELECT * FROM NLS_DATABASE_PARAMETERS Commented Oct 25, 2016 at 5:00
  • 1
    then show me the value of NLS_CHARACTERSET Commented Oct 25, 2016 at 5:02
  • 1
    now try charset='iso-8859-1' Commented Oct 25, 2016 at 5:11

2 Answers 2

Reset to default
12

the proper encoding for your database is iso-8859-1 according to oracle docs so When you connect() to your database, pass the charset='iso-8859-1' or encoding='iso-8859-1' try both.

Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

@HishamKaram I have faced the same error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 1: invalid continuation byte. But, unfortunatly, none of your solutions helped me
please let me know your database encoding by executing this line SELECT * FROM NLS_DATABASE_PARAMETERS @LadenkovVladislav
@Thanks, but it's okay now. I just figured out, that i just had some corrupted non-utf8 values
@hisham Hey Hisham how do you find the encoding map from Oracle NLS_NCHAR_CHARACTERSET ?
When i do conn = sqlite3.connect(sql_path,charset='iso-8859-1'), I get TypeError: 'charset' is an invalid keyword argument for this function.... same for encoding!
|
2

Python3.7:

con = sqlite3.connect(path_to_db)
encoding = "latin1"
con.text_factory = lambda x: str(x, encoding)
# do not preserve non-printable
# con.text_factory = lambda x: str(x, "ascii", errors="ignore")
data = pd.read_sql_query(QUERY, con)

Pydocs on text_factory

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.