Pandas: Read CSV: ValueError: could not convert string to float

Question

I'm trying to read a large and complex CSV file with pandas.read_csv. The exact command is

pd.read_csv(filename, quotechar='"', low_memory=True, dtype=data_types, usecols= columns, true_values=['T'], false_values=['F'])

I am pretty sure that the data types are correct. I can read the first 16 million lines (setting nrows=16000000) without problems but somewhere after this I get the following error

ValueError: could not convert string to float: '1,123'

As it seems, for some reason pandas thinks two columns would be one.

What could be the problem? How can I fix it?

Have you done a visual inspection of the line at which the error is raised? Alternatively, could you provide us with that line +/- 1 line (so three lines in total)? — Nelewout
– Nelewout, Commented Dec 16, 2015 at 18:04
if loosing some data is not an issue you could probably add 'error_bad_lines=False' in order to skip problematic rows — Ezer K
– Ezer K, Commented Dec 16, 2015 at 18:12
I think it is very hard without checking problematic rows. But you can check divide by zero - string like something/0 - it can cause this error. — jezrael
– jezrael, Commented Dec 16, 2015 at 20:39
How can I find the row? The error message does not say the row. — SpiderOtto
– SpiderOtto, Commented Dec 16, 2015 at 20:55

SpiderOtto · Accepted Answer · 2015-12-16 22:57:28Z

1

I found the mistake. The problem was a thousand separator.

When writing the CSV file, most numbers were below thousand and were correctly written to the CSV file. However, this one value was greater than thousand and it was written as "1,123" which pandas did not recognize as a number but as a string.

answered Dec 16, 2015 at 22:57

SpiderOtto

992 silver badges5 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Pandas: Read CSV: ValueError: could not convert string to float

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related