How can we approximate the minimum of a function like y = 2x1+x2x2+4 with gradient descent in the error backpropagation? if we consider initial values for x1 and x2 are zero and step width of 0.5.
1 Answer
In general, you can solve this kind of problem from scratch using the autograd package to compute gradient of y(x1, x2). Here is an example:
import autograd
# define your function to mimimize
def y(x1, x2):
return 2*x1 + x2*x2 + 4
# get analytical gradients of y w.r.t the variables
dy_dx1 = autograd.grad(y, 0)
dy_dx2 = autograd.grad(y, 1)
# define starting values, step size
x1, x2 = 0.0, 0.0
step_size = 0.5
num_iterations = 100
ys = []
for iteration in range(num_iterations):
# record value
y_value = y(x1, x2)
ys.append(y_value)
print(f'at iteration {iteration}, y({x1:.1f}, {x2:.1f}) = {y_value}')
# compute gradients
der_x1 = dy_dx1(x1, x2)
der_x2 = dy_dx2(x1, x2)
# update variables to minimize
x1 -= step_size * der_x1
x2 -= step_size * der_x2
Note that in your case, computing the gradient analytically is straightforward a well. Also the minimum of this function will be -∞ as x1 → -∞ so the result of this kind of gradient descent might give unhelpful results.
