Auto convert strings and float columns using genfromtxt from numpy/python

Question

I have several different data files that I need to import using genfromtxt. Each data file has different content. For example, file 1 may have all floats, file 2 may have all strings, and file 3 may have a combination of floats and strings etc. Also the number of columns vary from file to file, and since there are hundreds of files, I don't know which columns are floats and strings in each file. However, all the entries in each column are the same data type.

Is there a way to set up a converter for genfromtxt that will detect the type of data in each column and convert it to the right data type?

Thanks!

Can you use pandas? pandas.readcsv is much more powerful than numpy.genfromtxt, and will do all of this for you automatically. — jakevdp
– jakevdp, Commented Sep 28, 2016 at 13:52
The problem is that I need to be able to convert the output to a numpy array for further processing. I've heard that datafame.as_matrix() returns a object array. So I prefer to use genfromtxt. — Emmanuel Sunil
– Emmanuel Sunil, Commented Sep 28, 2016 at 14:03
dataframe.to_records() will give you a record array, which is probably what you want then. Storing mixed types in a numpy array can only be done via a record array or an object array. — jakevdp
– jakevdp, Commented Sep 28, 2016 at 14:10

jakevdp · Accepted Answer · 2016-09-28 14:26:54Z

1

If you're able to use the Pandas library, pandas.read_csv is much more generally useful than np.genfromtxt, and will automatically handle the kind of type inference mentioned in your question. The result will be a dataframe, but you can get out a numpy array in one of several ways. e.g.

import pandas as pd
data = pd.read_csv(filename)

# get a numpy array; this will be an object array if data has mixed/incompatible types
arr = data.values

# get a record array; this is how numpy handles mixed types in a single array
arr = data.to_records()

pd.read_csv has dozens of options for various forms of text inputs; see more in the pandas.read_csv documentation.

answered Sep 28, 2016 at 14:26

jakevdp

88.6k15 gold badges140 silver badges177 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Auto convert strings and float columns using genfromtxt from numpy/python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related