Python Read Formatted String

Question

I have a file with a number of lines formatted with the following syntax:

FIELD      POSITION  DATA TYPE
------------------------------
COOP ID       1-6    Character
LATITUDE     8-15    Real
LONGITUDE   17-25    Real
ELEVATION   27-32    Real
STATE       34-35    Character
NAME        37-66    Character
COMPONENT1  68-73    Character
COMPONENT2  75-80    Character
COMPONENT3  82-87    Character
UTC OFFSET  89-90    Integer

The data is all ASCII-formatted.

An example of a line is:

011084  31.0581  -87.0547   26.0 AL BREWTON 3 SSE                  ------ ------ ------ +6

My current thought is that I'd like to read the file in a line at a time and somehow have each line broken up into a dictionary so I can refer to the components. Is there some module that does this in Python, or some other clean way?

Thanks!

From what I understand, there may be no delineation between columns, so simple application of split() is not an option. — Richard
– Richard, Commented Aug 18, 2011 at 17:42
If there is no delineation, how can we know that a part of a line is in the first column and not the second? (Edit: I do not understand what you mean in your second comment. Is the file you want to read different from the one in your post?) — murgatroid99
– murgatroid99, Commented Aug 18, 2011 at 17:45
Because, @murgatroid99, the first column is characters 1-6, the second column is characters 8-15, and so on. — Richard
– Richard, Commented Aug 18, 2011 at 17:46

Kimvais · Accepted Answer · 2011-08-20 10:14:24Z

16

EDIT: You can still use the struct module:

See the struct module documentation. Looks to me like you want to use struct.unpack()

What you want is probably something like:

import struct
with open("filename.txt", "r") as f:
    for line in f:
        (coop_id, lat, lon, elev, state, name, c1, c2, c3, utc_offset
         ) = struct.unpack("6sx8sx9sx6sx2sx30sx6sx6sx6sx2s", line.strip())
        (lat, lon, elev) = map(float, (lat, lon, elev))
        utc_offset = int(utc_offset)

edited Aug 20, 2011 at 10:14

answered Aug 18, 2011 at 17:43

Kimvais

39.9k17 gold badges111 silver badges143 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

Richard Over a year ago

Sorry, the data is in ASCII format. I will clarify the question.

Richard Over a year ago

By the way, I didn't give you the down-vote. Struct looks like it would work fine if the data had been packed in that way; alas.

flying sheep Over a year ago

then you just have to read the text file in binary mode and unpack each line with stuct or encode each line and unpack the resulting bytes

Richard Over a year ago

I tried the code a couple of times, I think it's assuming the data is non-ASCII in nature.

Kimvais Over a year ago

@Richard, just changed my example to assume all data is strings, not packed values.

|

murgatroid99 · Accepted Answer · 2011-08-18 17:58:41Z

I think I understand from your question/comments what you are looking for. If we assume that Real, Character, and Integer are the only data types, then the following code should work. (I will also assume that the format file you showed is tab delimited):

format = {}
types = {"Real":float, "Character":str, "Integer":int}

for line in open("format.txt", "r"):
    values = line.split("\t")
    range = values[1].split("-")
    format[values[0]]={"start":int(range[0])-1, "end":int(range[1])-1, "type":types[values[2]]}

results=[]
for line in open("filename.txt"):
    result={}
    for key in format:
        result[key]=format["type"](line[format["start"]:format["end"]])
    results.append(result)

You should end up with results containing a list of dictionaries where each dictionary is a mapping from key names in the format file to data values in the correct data type.

Jonathan Adelson · Accepted Answer · 2011-08-18 17:52:12Z

0

It seems like you could write a function using strings and slices fairly simply. string[0:5] would be the first element. Does it need to be extensible, or is it likely a one off?

answered Aug 18, 2011 at 17:52

Jonathan Adelson

3,3415 gold badges32 silver badges39 bronze badges

2 Comments

Richard Over a year ago

There will be a number of data files from different sources with formats not unlike this. Extensibility would be desirable. It's easy enough to write as a one-off, but it also seemed like a module might exist for this purpose.

Jonathan Adelson Over a year ago

That I don't know, but I'm interested to see the answer, as well. :)

Collectives™ on Stack Overflow

Python Read Formatted String

3 Answers 3

8 Comments

Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

8 Comments

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related