Vectorizing operations in nested loops: Python

Question

I have 2D numpy array, I need two nested loops to iterate over each of its elements. I want to make some vectorization on the nested loops, but I keep getting an error saying,

j= np.arange (0,x.shape [1])
IndexError: tuple index out of range

These are the original nested loops:

for k in range(A.shape[0]):
        for j in range(A.shape[1]):
            A[k,j] = method1(x[k],x[j],a,c0,c1)

This is how I tried to make vectorization based on this answer, https://codereview.stackexchange.com/questions/17702/python-numpy-running-15x-slower-than-matlab-am-i-using-numpy-effeciently,

j= np.arange (0, A.shape [1])
    for k in range(A.shape[0]):

            A[k,j] = method1(x[k],x[j],a,c0,c1)

I tried to change the indices in np.arange, but it didn't work.

Can anyone please tell me how to fix this?

Thanks.

EDIT: By @ajcr comment, my mistake was in this line, j= np.arange (0,x.shape [1]), I was supposed to use the column count of the 2D array "A" as j= np.arange (0,A.shape [1]), but I have mistakenly used the 1D array x, hence the error. It's working perfectly now.

The IndexError for x.shape[1] implies x only has one dimension - are you sure it's 2D? — Alex Riley
– Alex Riley, Commented Jun 29, 2015 at 11:04
Oh! I really apologize for this mistake, I was supposed to have it as, j= np.arange (0,A.shape [1]). Thanks a lot for your comment, it's now working perfectly and much faster than before :) — Dania
– Dania, Commented Jun 29, 2015 at 11:10

Moritz · Accepted Answer · 2015-06-30 07:19:45Z

2

It is hard to give you an answer since you do not provide the function itself. If it is possible to vectorize your calculations, this would be the way to go. Otherwise, you may use np.vectorize:

import numpy as np

def my_fun(x,y,a,b):
    if x > y:
        return a*x**2 + y
    else:
        return b*x**2 + a*y

vec_fun = np.vectorize(my_fun)
x = np.random.rand(100000)
y = np.random.rand(100000)

%%timeit
for xx,yy in zip(x,y):
    my_fun(xx,yy,1,2)

10 loops, best of 3: 138 ms per loop

%%timeit
    vec_fun(x,y,1,2)

10 loops, best of 3: 65.4 ms per loop

If you just loop over a list of input arguments, you could use multiprocessing.Pool in order to utilize all of your cores.

EDIT: As BlackCat noted, the np.vectorize function may not speed up things. If this is the case, the only solution I can think of is either use all cores or switch to Cython.

edited Jun 30, 2015 at 7:19

answered Jun 29, 2015 at 11:23

Moritz

5,44813 gold badges47 silver badges89 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Dania Over a year ago

thank you for your clear explanation, I will also try what you suggested.

TheBlackCat Over a year ago

From the documentation "The vectorize function is provided primarily for convenience, not for performance. The implementation is essentially a for loop." So it isn't what the OP really wants. My results have the loop time as 94.8 ms and the vectorized time as 271 ms, so the version using vectorize is actually slower.

Moritz Over a year ago

Good point. I forgot that. But the performance difference puzzles me.

Dania Over a year ago

Thank you, actually it worked well in my case, after applying vectorization, the performance time dropped from 61 ms to 5 ms. It seems that it difference depending on the case.

Moritz Over a year ago

Did you check your calculations ?

|

Collectives™ on Stack Overflow

Vectorizing operations in nested loops: Python

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related