numpy "extend/append" ndarray

Question

The problem:

I have a numpy array

tf = numpy.full((500, 4, 1, 2), [500, 1])

tf : 
array([[[[ 500.,    1.]],

    [[ 500.,    1.]],

    [[ 500.,    1.]],

    [[ 500.,    1.]]],

   ..., 
   [[[ 500.,    1.]],

    [[ 500.,    1.]],

    [[ 500.,    1.]],

    [[ 500.,    1.]]]])


tf.shape :
(500, 4, 1, 2)

Consider the first group: tf[0][0] this is : array([[ 500., 1.]])

I need to be able to append (in place) additional values say [[100, 0.33], [1, 0.34], [15, 0.33]] so that the end result looks like (this operation is carried out for each of the elements):

tf : 
array([[[[ 500.,  1.], [100.,  0.33], [1.,  0.34], [15.,   0.33]],

    [[ 500.,    1.]],

    [[ 500.,    1.]],

    [[ 500.,    1.]]],

   ..., 
   [[[ 500.,    1.]],

    [[ 500.,    1.]],

    [[ 500.,    1.]],

    [[ 500.,    1.]]]])

I tried numpy.concatenate((tf[0][0], [[100, 0.33]]), axis = 0) this returns a new appended ndarray, but I cannot assign it back to tf[0][0] as it fails with the following error. ValueError: could not broadcast input array from shape (2,2) into shape (1,2)

Is there any other way to achieve what I want using numpy?

==========================================================

Inefficient list way of doing it:

# initialization
tf = [[[]] for i in xrange(500)]
for i in xrange(500):
    tf[i] = [[] for a in xrange(4)]
    for j in xrange(4):
        tf[i][j].append([500, 1.0])

# usage: (for any 0 < i < 500; 0 < j < 4 )

tf[i][j].append([100, 0.33])

but this is inefficient (considering I need to do this over a million times)

You cannot do this using numpy.ndarrays which are true, fixed-size multidimensional arrays. The efficient way to do this is with a list. — juanpa.arrivillaga
– juanpa.arrivillaga, Commented Dec 20, 2017 at 9:21
@juanpa.arrivillaga this operation is carried out for each of the elements - So it seems it won't be a ragged array, but a regular one. — Divakar
– Divakar, Commented Dec 20, 2017 at 9:22
A follow up question, then would be, how can I convert this nested list effectively into ndarray for further processing? — okkhoy
– okkhoy, Commented Dec 20, 2017 at 9:22
@okkhoy to convert to an ndarray your only option is to use some sort of dtype=object array, which will essentially be a bad list. In what sense of "inefficient" do you mean "inefficient list way"? — juanpa.arrivillaga
– juanpa.arrivillaga, Commented Dec 20, 2017 at 9:39

Anand C U · Accepted Answer · 2017-12-20 09:56:33Z

2

The problem in your approach is that the shape of each element varies and hence you cannot have a fixed shape. However you can define each element as type object and achieve what you are trying to do.

import numpy as np

tf = np.empty((500, 4, 1), dtype= object)

will produce

array([[[None],
        [None],
        [None],
        [None]],

       [[None],
        [None],
        [None],
        [None]],

       [[None],
        [None],
        [None],
        [None]],

       ...,
       [[None],
        [None],
        [None],
        [None]],

       [[None],
        [None],
        [None],
        [None]],

       [[None],
        [None],
        [None],
        [None]]], dtype=object)

Now add your constant initial element as a list to each of these array elements. You might be tempted to use fill() here, but that assigns a single object to each array element and modifying individual array elements will change the entire array. To initialize, you cannot avoid iterating through the entire array.

for i,v in enumerate(tf):
    for j,w in enumerate(v):
        tf[i][j][0] = [[500.0,1.0]]

will produce

array([[[list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])]],

       [[list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])]],

       [[list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])]],

       ...,
       [[list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])]],

       [[list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])]],

       [[list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])]]], dtype=object)

Now you can access each element separately. Use append or extend as you prefer.

 tf[0][0][0].append([100,0.33])

will give

array([[[list([[500.0, 1.0], [100, 0.33]])],
        [list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])]],

       [[list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])]],

       [[list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])]],

       ...,
       [[list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])]],

       [[list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])]],

       [[list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])],
        [list([[500.0, 1.0]])]]], dtype=object)

Only the initialization requires iterating through the array.

edited Dec 20, 2017 at 9:56

answered Dec 20, 2017 at 9:49

Anand C U

9159 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

okkhoy Over a year ago

I have a follow up question, can I use this as a normal numpy array later; e.g., to do array multiplication, dot product etc?

Anand C U Over a year ago

I didn't get that. Can you explain?

okkhoy Over a year ago

sorry! what I meant was: The deepest data type is a list , so was wondering if I can use numpy operations like dot, ndarray multiplication etc.,

Anand C U Over a year ago

It depends on the operation. I suggest that after appending all required elements, you can iterate over the array and convert each individual list to a numpy array. This way scalar operations and few other array-wide operations should work.

hpaulj Over a year ago

@okkhoy, math operations with an object dtype array are hit and miss. Some propagate to the objects, some don't. All will slower than with regular arrays. Indexing may be slower than with a list. Test small cases first.

Collectives™ on Stack Overflow

numpy "extend/append" ndarray

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related