1260 words - 6 pages

Steepest Descent Direction in Optimization - Application and Algorithm

Hemanand. T

Department of Chemical Engineering,

St. Joseph’s College of Engineering,

Chennai – 600 119

Abstract: An analytical solution to identify the minimum value of a function used for optimization based on steepest descent technique was extensively discussed with applications in a process. The properties of gradient vector, the oscillation of function values and overshoot were analyzed in a function for the search of minimum. The best step size in each iteration was found by conducting a one-D optimization in the steepest descent direction. The five steps in the algorithm for steepest descent direction ...view middle of the document...

The first optimization technique, which is known as steepest descent, goes back to Gauss. Historically, the first term to be introduced was linear programming, which was invented by George Dantzig in the 1940s.

Gradient descent is a first-order optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or the approximate gradient) of the function at the current point. If instead one takes steps proportional to the gradient, one approaches a local maximum of that function; the procedure is then known as gradient ascent. Gradient descent is also known as steepest descent, or the method of steepest descent. When known as the latter, gradient descent should not be confused with the method of steepest descent for approximating integrals.

2. Properties of steepest descent technique:

Properties of Gradient Vector

The gradient vector of a scalar function [pic]is defined as a column vector

[pic]

For example

[pic]

at the point[pic]

[pic]

The normalized gradient vector

[pic]

For example, at the point [pic]

[pic]

2.1 Property 1. The gradient vector represents a direction of maximum rate of increase for the function f(x) at x*. For example,

[pic]

If we increase x in the direction [pic] by a step size of α = 0.5

[pic]

The function value becomes

[pic]

If we move in a direction [pic]

[pic] + .5[pic] [pic]

The function value becomes

[pic]

If we move in a direction [pic]

[pic] + .5[pic] [pic]

The function value becomes

[pic]

We can see that moving along the gradient direction results in direction results in the maximum increase in the function.

2.2 Property 2.

The gradient vector c of [pic]) at the point [pic]is orthogonal (normal) to the tangent plane for the surface [pic] constant. For example, [pic])= 25[pic] + [pic] = 25 the slope at [pic]=.6, [pic]=4 can be found to be

2(25)[pic]

Slope = [pic] = -[pic]

The direction of the tangent line is given by

t = [pic][pic]

c and t are normal each other as

[pic]t = [pic] = 30-8(3.75)=0

2.3 Property 3:

The maximum rate of change of [pic] at any point [pic] is the magnitude of the gradient vector by

[pic]

Steepest descent direction. Let [pic] be a differentiable function with respect to X. The direction of steepest descent for [pic] at any point is

d=-c or [pic]

3. Application of steepest descent direction:

3.1 Application of the steepest descent direction to search for the minimum for [pic]=25[pic] + [pic] starting at [pic]=[pic] with a step size of α=.5. the function value at the starting point is

[pic]

Ana analytical solution reveals that the minimum point is at [pic]=[pic] and[pic]. Let us start the process of iterations.

[pic]

[pic] - .5[pic] [pic]

[pic]

[pic]

[pic] - .5[pic] [pic]

[pic]

[pic]

[pic] - .5[pic] [pic]

...

Find the perfect research document on any subject