arrange N different vectors into an object array - not matrix

Question

In python and/or numpy: I have several 1D vectors (arrays) that I read in from files one at a time. They do not all have the same number of coefficients, i.e. not the same length. i would like to add each vector to a list of vectors and address them as objects, sucking them back out one at a time for processing. I have read any questions/answers that sound like this process: How to build a matrix one column at a time

Vectorized way to arrange vector into matrix (numpy)

and I've tried various python/numpy functions including append, concatenate, stack, appending to an object array using an increasing index, etc. For example:

filenames = pd.read_csv('filenames', header=None)
numFiles = filenames.shape[0]

# read in all files
firstTime = True
x0 = [numFiles]
y0 = [numFiles]
for i in range(numFiles):

    fn = filenames.iloc[i, 0]
    df = pd.read_csv(fn, delimiter='\t', header=None)

    # select first two columns in data frame
    nparr = np.array(df)[:,0:2]

    # this one line does a loop, discarding entries with zero flux
    # THIS IS THE PLACE where all vectors wind up with a different length
    nparr = nparr[nparr[:,1] != 0]

    x0[i] = nparr[:,0]
    y0[i] = nparr[:,1]

# by this point, object array is complete

Any syntax I try either genrates an error, in this case:

Traceback (most recent call last):
  File "InterpolateToSameDMValues-26-Jun-2019.py", line 48, in <module>
    x0[i] = nparr[:,0]
IndexError: list assignment index out of range

or else numpy simply appends all my vectors into one very long vector.

How can I tell python to not do anything fancy, and simply create an array of objects I can iterate through?

for obj in x0:
    # do something with vector

x0 = [10] doesn't make a list 10 items long. It makes a list with one item, the number 10. — hpaulj
– hpaulj, Commented Jul 1, 2019 at 18:36

Mike Mint · Accepted Answer · 2019-07-01 18:05:54Z

1

You can try to create list of vectors. List has great function append():

import numpy as np
my_list = []
#you can append your list like this
sample_vector = np.array([1, 2, 3])
sample_vector2 = np.array([3, 2, 1])
my_list.append(sample_vector)
my_list.append(sample_vector2)
#working with your stored vectors
for vector in my_list:
    print(vector)
    #or do what you want

Hope you will get an idea

answered Jul 1, 2019 at 18:05

Mike Mint

1455 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Gerry Harp Over a year ago

The changing meaning of append in python, numpy and pandas is confusing.

Gerry Harp Over a year ago

That worked. Thanks. But I'm still interested to know, how could you write the same loop using a numpy array instead of a list?

Mike Mint Over a year ago

Gerry, I do not have such experience, but I suppose you could get some info from this post stackoverflow.com/a/31250215/11715830

Gerry Harp Over a year ago

It says: "Never append to numpy arrays in a loop: it is the one operation that NumPy is very bad at compared with basic Python. This is because you are making a full copy of the data each append, which will cost you quadratic time." Thanks, that's my answer.

Mike Mint Over a year ago

Great, that you finally get your answer

Collectives™ on Stack Overflow

arrange N different vectors into an object array - not matrix

1 Answer 1

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related