Reading structured column data with numpy

Question

So I'm trying to use the following to read the following file with numpy:

import numpy as np

recordtype = np.dtype([('name', 'S24'),
                       ('ra', [(np.int32,np.int32,np.float32)]),
                       ('dec', [(np.int32,np.int32,np.float32)]),
                        ('z',np.float32), ('rmag',np.float32),('rmag_error',np.float32),
                        ('gmag',np.float32),('gmag_error',np.float32)])
qData = numpy.loadtxt("SDSS_quasar_cat5(1)", skiprows=6)

Here's a sample of the datafile:

Results from heasarc_sdssquasar: Sloan Digital Sky Survey Quasar Catalog (10th Data Release: DR10Q)
Coordinate system:  Equatorial
|name                    |ra          |dec         |vi_redshift|rmag        |rmag_error  |gmag     |gmag_error  |
|SDSS J163313.26+352050.7|16 33 13.265|+35 20 50.77|     2.0000|2.480190e+01|3.865410e+00|2.511420e+01|9.026890e-01|
|SDSS J164135.36+372726.9|16 41 35.362|+37 27 26.96|     2.0000|2.279110e+01|1.722010e-01|2.203690e+01|7.126500e-02|
|SDSS J160420.86+275634.5|16 04 20.869|+27 56 34.52|     2.0010|2.008330e+01|2.592250e-02|2.048210e+01|2.699570e-02|
|SDSS J163948.06+331030.2|16 39 48.069|+33 10 30.27|     2.0010|1.945350e+01|1.478850e-02|1.971600e+01|1.617890e-02|
|SDSS J162006.83+395403.8|16 20 06.834|+39 54 03.83|     2.0020|1.937250e+01|2.033600e-02|1.942160e+01|2.408500e-02|

I can't seem to get past defining the dtype. I'm new to python, and looked for examples, but I must have something wrong. Any suggestions?

Thx!

With your example code I get a TypeError: data type not understood. That's because of the Right Ascension and Declination definitions: aside from (fixed length) strings, you can only use discrete types for each field — Ricardo Cárdenes
– Ricardo Cárdenes, Commented Oct 13, 2014 at 22:27
Rats -- does that mean I have to read the 2 integers and one float (for RA and Dec) as separate arrays? — earnric
– earnric, Commented Oct 13, 2014 at 22:30
Or you can convert them to decimal degrees. Would that be acceptable? — Ricardo Cárdenes
– Ricardo Cárdenes, Commented Oct 13, 2014 at 22:31
Well that's what I want to get to, but I don't have control over the data file if that's what you mean. I have to read Hr, Min, Sec and then convert them... I'm still stuck on just reading the dang file!! I don't know how to use loadtxt and dtype yet! — earnric
– earnric, Commented Oct 13, 2014 at 22:32
Yeah, yeah... Well you can provide "converters" for each column if needed. Let me write down a short example so that you can see how is it done... — Ricardo Cárdenes
– Ricardo Cárdenes, Commented Oct 13, 2014 at 22:34

Ricardo Cárdenes · Accepted Answer · 2014-10-13 22:54:14Z

2

Here's an example.

import numpy as np

recordtype = np.dtype([('name', 'S24'),
                       ('ra', np.float32),
                       ('dec', np.float32),
                       ('z',np.float32),
                       ('rmag',np.float32),
                       ('rmag_error',np.float32), 
                       ('gmag',np.float32),
                       ('gmag_error',np.float32)])

def ratodeg(string):
    hh, mm, ss = string.split()

    return (int(hh) * 15) + (int(mm) * 0.25) + (float(ss)  * 0.0042)

def dectodeg(string):
    return 

qData = np.loadtxt("sample",
                   skiprows=1,
                   dtype=recordtype,
                   delimiter = '|',
                   usecols = (1, 2, 3, 4, 5, 6, 7, 8),
                   converters = {1:str,
                                 2:ratodeg,
                                 3:dectodeg})

Note a few things in the call to loadtxt:

The use of dtype to specify the right one. Otherwise NumPy will assume you want an array of floats
The use of delimiter to tell loadtxt how to break the columns
The usecolumns to specify which ones to use. Column 0 and 9 are the blanks before the first | and after the last one
The use of converters. It assigns converter functions for each specified column. Only addressed columns 1-3. The default converter is float(...)

I left dectodeg blank because I can't ever remember on top of my head how to do the conversion :P (I'm not even sure ratodeg is ok...), so right now dec is nan for each row.

answered Oct 13, 2014 at 22:54

Ricardo Cárdenes

9,1941 gold badge23 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

earnric Over a year ago

Thx Ricardo! I'm gonna give it a try! Didn't know about converters.

earnric Over a year ago

Worked beautifully Ricardo... Thank you.

earnric Over a year ago

That's the truth... Hey, I'm a c++ guy too. Pro software developer who's back in school to become an astronomer! Thx again for the help.

Collectives™ on Stack Overflow

Reading structured column data with numpy

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related