Couldn't read dat file from online source in python

Question

I could not read this dat file in python.

I tried the following :

url3 = 'https://www2.census.gov/programs-surveys/saipe/datasets/2002/2002-state-and-county/est02all.dat'
import pandas as pd
saipe02 = pd.read_csv(url3, sep='\s+', header=None, skiprows=1)

Or :

import numpy as np
saipe02 = np.fromfile(url3)

but they don't work. Only this code works but I couldn't put any delimiter to separate the data into column. I tried many delimiters, but it doesn't work:

saipe02=pd.read_table(url3,header=None)

jezrael · Accepted Answer · 2018-05-23 13:04:17Z

1

For me working read_fwf, docs:

saipe02 = pd.read_fwf(url3, header=None)
print (saipe02.head())
   0   1         2         3     ...       30  31            32         33
0   0   0  34569951  33912173    ...      NaN  US  est02ALL.dat  29OCT2004
1   1   0    679856    646828    ...      NaN  AL  est02ALL.dat  29OCT2004
2   1   1      4795      3728    ...      NaN  AL  est02ALL.dat  29OCT2004
3   1   3     16175     12558    ...      NaN  AL  est02ALL.dat  29OCT2004
4   1   5      6152      4767    ...      NaN  AL  est02ALL.dat  29OCT2004

[5 rows x 34 columns]

answered May 23, 2018 at 13:04

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

zilong Over a year ago

thanks very much. I have one more question: saipe02 = pd.read_fwf(url3, header=None,dtype=object) medin02=saipe02.iloc[:,[0,1,20]] medin02.columns = ['state_fips', 'county_fips','med_income_02'] medin02['state_fips']=medin02.loc[:,'state_fips'].apply(lambda x: '{0:0>2}'.format(x)) It warns Try using .loc[row_indexer,col_indexer] = value instead After I changed to this code: medin02['state_fips']=medin02['state_fips'].apply(lambda x: '{0:0>2}'.format(x)) It still warns the same thing.

jezrael Over a year ago

You need copy like medin02=saipe02.iloc[:,[0,1,20]].copy(), check this for explanation.

Collectives™ on Stack Overflow

Couldn't read dat file from online source in python

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related