Learning rate in Gradient Descent algorithm

Question

In the gradient descent algorithm, I update the B and M values according to their derivatives and then multiply them with the Learning rate value, but when I use the same value for L, such as 0.0001, it does not work correctly. Decreasing or increasing the L value does not work. As a workaround, I had to set different L values for both b and m values. Is this normal or is there an error?

import pandas as pd
import matplotlib.pyplot as plt
import time
import random

# Veri seti
veri_seti = "study_score_decreasing.csv" # study_score_decreasing.csv # study_score_increasing.csv 
data = pd.read_csv(veri_seti)

# Gradient Descent Fonksiyonu
def gradient_descent(m_next, b_next, points, L):
    m_gradient = 0
    b_gradient = 0
    n = len(points)
    
    for i in range(n):
        x = points.iloc[i].study_time
        y = points.iloc[i].score
        
        m_gradient += -(2/n) * x * (y - (m_next * x + b_next))
        b_gradient += -(2/n) * (y - (m_next * x + b_next))
    
    m = m_next - m_gradient * 0.0001 #(L = 0.0001)
    b = b_next - b_gradient * 0.1    #(L = 0.1)
    
    return m, b

# Grafik Gösterim Fonksiyonu
def show_graph(m, b):
    plt.scatter(data.study_time, data.score, color="red")
    x_range = range(int(data.study_time.min()), int(data.study_time.max()) + 1)
    plt.plot(x_range, [m * x + b for x in x_range], color="blue")
    plt.xlabel('Study Time')
    plt.ylabel('Score')
    plt.title('Study Time vs Score')
    plt.show()
    time.sleep(0.001)
    print("=>  F(X):", round(m, 1), "X +", round(b, 3))

# Ana Fonksiyon
def main(m, b, L, epochs):
    print("=>  F(X):", m, "X", b)
    
    for i in range(epochs):
        m, b = gradient_descent(m, b, data, L)
        show_graph(m, b)
        
# Başlangıç değerleri
main(random.uniform(-1, 110), random.uniform(-10, 10), 0.1, 250)

I updated the L values one by one and got a logical result, but with a common L value, why does the solution seem illogical?

L Tyrone · Accepted Answer · 2024-08-14 00:23:03Z

I have seen that the mathematical formula used is to implement a linear regression model? For clarification, if the data is not suited for linear regression, then the model will definitely struggle since the parameters you've set are relatively rigid.

Gradient Calculation: The calculation for the gradients appears to be correct, but it’s always good to double-check the formulas. My point is that maybe you can clarify a bit more? Here are the code changes I made:

import pandas as pd
import matplotlib.pyplot as plt
import time
import random

# Veri seti
data = {'study_time': [1, 2, 3, 4, 5], 'score': [2, 4, 9, 16, 25]}
points = pd.DataFrame(data)

print(points)

# Gradient Descent Fonksiyonu
def gradient_descent(m_next, b_next, points, L):
    m_gradient = 0
    b_gradient = 0
    n = len(points)

    for i in range(n):
        x = points.iloc[i].study_time
        y = points.iloc[i].score
    
    m_gradient += -(2/n) * x * (y - (m_next * x + b_next))
    b_gradient += -(2/n) * (y - (m_next * x + b_next))

    m = m_next - (L * m_gradient * 0.0001) #(L = 0.0001)
    b = b_next - (L * b_gradient * 0.1 )   #(L = 0.1)

    return m, b

# Grafik Gösterim Fonksiyonu
def show_graph(m, b):
    plt.scatter(points.study_time, points.score, color="red")
    x_range = range(int(points.study_time.min()), int(points.study_time.max()) + 1)
    plt.plot(x_range, [m * x + b for x in x_range], color="blue")
    plt.xlabel('Study Time')
    plt.ylabel('Score')
    plt.title('Study Time vs Score')
    plt.show()
    time.sleep(0.001)
    print("=>  F(X):", round(m, 1), "X +", round(b, 3))

# Initial values for m and b
m_next = 0
b_next = 0

# Single learning rate
L = 0.01


# Ana Fonksiyon
def main(m, b, L, epochs):
    print("=>  F(X):", m, "X", b)

    for i in range(epochs):
        m, b = gradient_descent(m, b, points, L)
        show_graph(m, b)
    
 # Başlangıç değerleri
 main(random.uniform(-1, 110), random.uniform(-10, 10), 0.1, 250)

For context, f(x)**2 is the synthetic data I used.

Collectives™ on Stack Overflow

Learning rate in Gradient Descent algorithm

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related