storing data in numpy array without overwriting previous data

Question

i am running a for loop to store the data in a numpy array. the problem is that after every iteration previous data is overwritten by latest one . i want to be able to store all data bu using some "extend" function as used for simple arrays. i tried append but it's not storing all the values of all arrays.

code

data1=np.empty((3,3),dtype=np.int8)
top_model_weights_path = '/home/ethnicity.071217.23-0.28.hdf5'
df = pd.read_csv('/home/instaurls.csv')
for row in df.itertuples():
    data = io.imread(row[1])
    data1=np.append(data1,data)
print(data1)

expected output

[[[ 34  34  34]
  [ 35  35  35]
  [ 40  40  40]
  ..., 
  [  8   8   8]
  [ 12  12  12]
  [ 12  12  12]]

 [[ 39  39  39]
  [ 30  30  30]
  [ 25  25  25]
  ..., 
  [ 11  11  11]
  [  1   1   1]
  [  5   5   5]]

 [[ 54  54  54]
  [ 44  44  44]
  [ 34  34  34]
  ..., 
  [ 32  32  32]
  [  9   9   9]
  [  0   0   0]]

 ..., 
 [[212 212 210]
  [167 167 165]
  [118 118 116]
  ..., 
  [185 186 181]
  [176 177 172]
  [170 171 166]]

 [[220 220 218]
  [165 165 163]
  [116 116 114]
  ..., 
  [158 159 154]
  [156 157 152]
  [170 171 166]]

 [[220 220 218]
  [154 154 152]
  [106 106 104]
  ..., 
  [144 145 140]
  [136 137 132]
  [158 159 154]]]

why you are using numpy.append() with dataframe? Use data1.append(data) instead. For more details pandas append — Sociopath
– Sociopath, Commented Feb 12, 2018 at 13:54
@AkshayNevrekar without it, i get the error 'numpy.ndarray' object has no attribute 'append'. i am using the numpy array because image being read is converted into numpy array — Rehan
– Rehan, Commented Feb 12, 2018 at 13:57
provide expected output from above code. From np.append you can get 1D array only and I'm assuming you want 2D array as output. — Sociopath
– Sociopath, Commented Feb 12, 2018 at 14:06

Floyd4K · Accepted Answer · 2018-02-12 14:16:34Z

1

top_model_weights_path = '/home/ethnicity.071217.23-0.28.hdf5'
df    = pd.read_csv('/home/instaurls.csv')
data1 = np.array([io.imread(row[1]) for row in df.itertuples()])

If your dataset is not too big, there is no problem with using a standard list first and then convert to numpy array, I guess.

If you're not familiar with implicit lists:

data1 = []
for row in df.itertuples():
    data1.append(io.imread(row[1]))
data1 = np.array(data1)

answered Feb 12, 2018 at 14:16

Floyd4K

964 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Rehan Over a year ago

exactly what i needed. although in the long run my dataset is going to get large as i am building up this model for big data but this should work for now. thanks a ton!

Collectives™ on Stack Overflow

storing data in numpy array without overwriting previous data

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related