Pandas Vectorization speed up dataframe function

Question

I have this python code of the supertrend implementation. i am using pandas dataframe. the code works fine but, the supertrend function runs slower and slower as the dataframe increases in length. i was wondering how i could convert the for loop in the supertrend function to a Pandas Vectorization or using the apply() method

def trueRange(df):
    df['prevClose'] = df['close'].shift(1)
    df['high-low'] = df['high'] - df['low']
    df['high-pClose'] = abs(df['high'] - df['prevClose'])
    df['low-pClose'] = abs(df['low'] - df['prevClose'])
    tr = df[['high-low','high-pClose','low-pClose']].max(axis=1)
    
    return tr

def averageTrueRange(df, peroid=12):
    df['trueRange'] = trueRange(df)
    the_atr = df['trueRange'].rolling(peroid).mean()
    
    return the_atr
    

def superTrend(df, peroid=5, multipler=1.5):
    df['averageTrueRange'] = averageTrueRange(df, peroid=peroid)
    h2 = ((df['high'] + df['low']) / 2)
    df['Upperband'] = h2 + (multipler * df['averageTrueRange'])
    df['Lowerband'] = h2 - (multipler * df['averageTrueRange'])
    df['inUptrend'] = None

    for current in range(1,len(df.index)):
        prev = current- 1
        
        if df['close'][current] > df['Upperband'][prev]:
            df['inUptrend'].iloc[current] = True
            
        elif df['close'][current] < df['Lowerband'][prev]:
            df['inUptrend'].iloc[current] = False
        else:
            df['inUptrend'].iloc[current] = df['inUptrend'][prev]
            
            if df['inUptrend'][current] and df['Lowerband'][current] < df['Lowerband'][prev]:
                df['Lowerband'].iloc[current] = df['Lowerband'][prev]
                
            if not df['inUptrend'][current] and df['Upperband'][current] > df['Upperband'][prev]:
                df['Upperband'].iloc[current] = df['Upperband'][prev]

vector version

def superTrend(df, peroid=5, multipler=1.5):
    df['averageTrueRange'] = averageTrueRange(df, peroid=peroid)
    h2 = ((df['high'] + df['low']) / 2)
    df['Upperband'] = h2 + (multipler * df['averageTrueRange'])
    df['Lowerband'] = h2 - (multipler * df['averageTrueRange'])
    df['inUptrend'] = None


    cond1 = df['close'].values[1:] > df['Upperband'].values[:-1]
    cond2 = df['close'].values[1:] < df['Lowerband'].values[:-1]

    df.loc[cond1, 'inUptrend'] = True
    df.loc[cond2, 'inUptrend'] = False

    df.loc[(~cond1) & (cond2), 'inUptrend'] = df['inUptrend'][:-1]
    df.loc[(~cond1) & (cond2) & (df['inUptrend'].values[1:] == True) & (df['Lowerband'].values[1:] < df['Lowerband'].values[:-1]), 'Lowerband'] = df['Lowerband'][:-1]
    df.loc[(~cond1) & (cond2) & (df['inUptrend'].values[1:] == False) & (df['Upperband'].values[1:] > df['Upperband'].values[:-1]), 'Upperband'] = df['Upperband'][:-1]

Traceback (most recent call last):

  File "<ipython-input-496-ad346c720199>", line 3, in <module>
    superTrend(df, peroid=2, multipler=1.5)

  File "<ipython-input-495-57c750e273c2>", line 16, in superTrend
    df.loc[(~cond1) & (cond2) & (df['inUptrend'].values[1:] == True) & (df['Lowerband'].values[1:] < df['Lowerband'].values[:-1]), 'Lowerband'] = df['Lowerband'][:-1]

  File "C:\Users\fam\Anaconda3\lib\site-packages\pandas\core\indexing.py", line 189, in __setitem__
    self._setitem_with_indexer(indexer, value)

  File "C:\Users\fam\Anaconda3\lib\site-packages\pandas\core\indexing.py", line 606, in _setitem_with_indexer
    raise ValueError('Must have equal len keys and value '

ValueError: Must have equal len keys and value when setting with an iterable

Gilseung Ahn · Accepted Answer · 2021-12-14 04:58:43Z

2

Use .values[1:] and .values[:-1] for the vectorized comparison.
That is, .values[1:] is current, .values[:-1] is prev in your code.

Here is example to convert IF statements into vectorized comparison.

cond1 = df['close'].values[1:] > df['Upperband'].values[:-1]
cond1 = np.insert(cond1, 0, False)
df.loc[cond1, 'inUptrend'] = True

The reason using insert is the 0'th element has no element to be compared with.

answered Dec 14, 2021 at 4:58

Gilseung Ahn

2,6421 gold badge6 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

smith Over a year ago

Hey i have updated the function and the question with a vectorized version can you check the code for me plss. it is not working like the loop version

Collectives™ on Stack Overflow

Pandas Vectorization speed up dataframe function

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related