Does numpy handle arrays with dtype wrong?

Question

The following code snippet

f_folds = 3
fold_quantities = np.array([(0, 0, 0)])
for i in np.arange(n_folds) + 1:
    fold_quantities = np.concatenate(
        (fold_quantities, [(i, 0, 0)])
    )
print(fold_quantities)

gives me

array([[ 0,  0,  0],
       [ 1,  0,  0],
       [ 2,  0,  0],
       [ 3,  0,  0]])

When changing nothing but specifying the dtype of the ndarray

f_folds = 3
fold_quantities = np.array([(0, 0, 0)],
    dtype=[('index', int), ('#datapoints', 'int'), ('#pos_labels', 'int')])
for i in np.arange(n_folds) + 1:
    fold_quantities = np.concatenate(
        (fold_quantities, [(i, 0, 0)])
    )
print(fold_quantities)

it throws an error

ValueError   Traceback (most recent call last)
<ipython-input-174-649369eed10a> in <module>
      5     fold_quantities = np.concatenate(
      6         (fold_quantities,
----> 7          [(i, 0, 0)])
      8     )
      9 print(fold_quantities)

<__array_function__ internals> in concatenate(*args, **kwargs)

ValueError: all the input arrays must have same number of dimensions, but the array at index 0 has 1 dimension(s) and the array at index 1 has 2 dimension(s)

This message seems to make no sense. The array dimensions did not change.

How should that be handled? I would like to have the dtype specified since I want to sort the array according to single columns with sorted(key=).

hpaulj · Accepted Answer · 2020-02-06 22:20:24Z

Your first array should be make with list append, or a list comprehension. Repeated concatenate is slower

In [97]: np.array([[i,0,0] for i in range(4)])                                                 
Out[97]: 
array([[0, 0, 0],
   [1, 0, 0],
   [2, 0, 0],
   [3, 0, 0]])

With the compound dtype:

In [100]: np.array([(i,0,0) for i in range(4)], dtype=dt)                                      
Out[100]: 
array([(0, 0, 0), (1, 0, 0), (2, 0, 0), (3, 0, 0)],
      dtype=[('index', '<i8'), ('#datapoints', '<i8'), ('#pos_labels', '<i8')])

Note the use of dt and tuple instead of list. Data for a structured array has to be in the form of a list of tuples (just like the display).

With the change in dtype, the shape changes:

In [101]: _100.shape                                                                           
Out[101]: (4,)
In [102]: _97.shape                                                                            
Out[102]: (4, 3)

To add an array to structured array, it has to have a compatible dtype and shape:

In [104]: np.array([(4,0,0)],dt)                                                               
Out[104]: 
array([(4, 0, 0)],
      dtype=[('index', '<i8'), ('#datapoints', '<i8'), ('#pos_labels', '<i8')])

This is a (1,) array with dt dype.

In [105]: np.concatenate([_100, _104])                                                         
Out[105]: 
array([(0, 0, 0), (1, 0, 0), (2, 0, 0), (3, 0, 0), (4, 0, 0)],
      dtype=[('index', '<i8'), ('#datapoints', '<i8'), ('#pos_labels', '<i8')])
In [106]: _.shape                                                                              
Out[106]: (5,)

Another way of making the structured array - start with a list of arrays with the correct dtype:

In [107]: alist = [np.array((i,0,0),dt) for i in range(4)]                                     
In [108]: alist                                                                                
Out[108]: 
[array((0, 0, 0),
       dtype=[('index', '<i8'), ('#datapoints', '<i8'), ('#pos_labels', '<i8')]),
 array((1, 0, 0),
       dtype=[('index', '<i8'), ('#datapoints', '<i8'), ('#pos_labels', '<i8')]),
 array((2, 0, 0),
       dtype=[('index', '<i8'), ('#datapoints', '<i8'), ('#pos_labels', '<i8')]),
 array((3, 0, 0),
       dtype=[('index', '<i8'), ('#datapoints', '<i8'), ('#pos_labels', '<i8')])]

I use stack to join them since all 3 have 0d, scalar arrays.

In [109]: np.stack(alist)                                                                      
Out[109]: 
array([(0, 0, 0), (1, 0, 0), (2, 0, 0), (3, 0, 0)],
      dtype=[('index', '<i8'), ('#datapoints', '<i8'), ('#pos_labels', '<i8')])

Collectives™ on Stack Overflow

Does numpy handle arrays with dtype wrong?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related