Skip to main content

Gradient Descent Algorithm Fully Explained| Machine Learning

GRADIENT DESCENT ALGORITHM

When one start Learning For Machine Learning, As a beginner one founds a very complex explanation of Gradient Descent Algorithm and since it is a very important algorithm in machine Learning,understanding it, is also much important. Today , in this article we will try to understand Gradient Descent  in very easy, brief and well explained way.

 Gradient Descent is the most common optimization algorithm in machine learning and deep learning. It is a first-order optimization algorithm. This means it only takes into account the first derivative when performing the updates on the parameters.
If we start searching for Gradient Descent Algorithm ,we found this picture.
Don't panic! Let me explain you
z is a cost function of x and y.We have to find those value of x and y for which value of cost function (z) is minimum.

 If we visualize it, splitted in two part (z VS x for any fix value of y) and (z VS y for any fix value of x)  it looks easier.
 

Gradient Descent Algorithm  provides an efficient way to find the optimum value of y and x (or all the features on which cost function depends), which will minimize the value of cost function(z).

Points to Know Before Proceeding:

  • This is an iterative method ,you would have familiarity with iterative method(If not, no need to panic).Iterative method is which,where we start with pre-assumed value of any variable and iterate over it,find new values and get closer to desired value
  •  From any point, if we go opposite direction of slope,we will tend toward minima of that region.This is the key behind Gradient descent algorithm.If I plot above two graphs again with arrow direction, you'll understand.


 From Above graphs, It is quite clear that if we go opposite to the slope from any point, we will tend toward minima point.
But now what matters is, how much distance we'll take in single step, If we take a quite big step in opposite direction, we may cross the minima point.
Fig: Jump Over minima

  •  The Step size is quantified by a variable called 'learning rate(η)'. We choose a value of  'η' such that, while iterating, we don't jump over the minima .It is generally taken 0.001(we can change it depends upon data-set given)

Finally the iterating formula for Gradient Descent Algorithm:

(y)new=(y)old - η*(∂j/∂y)

(x)new=(x)old - η*(∂j/∂x)

In General:

(θ)new=(θ)old - η*∇θ

Where θ = [θ1,θ2, ... ,θn]

∇θ = Gradient of θ= [∂z/∂θ1,∂z/∂θ2, ... ,∂z/∂θn]




Above is Called Gradient Descent Algorithm. In next post, we will discuss about implementation of it.We will try to understand Linear Regression ( fitting a line to given dataset ) using Gradient Descent.


If You have any doubt , Ask us in comment. Also u can mail us at 4wallspace@gmail.com





Labels:

Comments

Post a Comment

Popular posts from this blog

What is Kernel Function | Fully Explained

Kernel Function In this post, We'll know what exactly is kernel function. Kernel Function is used to transform n - dimensional input to m- dimensional input , where m is much higher than n then find the dot product in higher dimensional efficiently.It also helps sometimes to do calculation easily in infinite dimensional space without going to infinite dimensions. Mathematical definition :  K(x, y) = <f(x), f(y)> . Here K is the kernel function, x, y are n dimensional inputs. f is a map from n-dimension to m-dimension space. < x,y> denotes the dot product. usually m is much larger than n. Intuition :  Normally calculating <f(x), f(y)> requires us to calculate f(x), f(y) first, and then do the dot product. These two computation steps can be quite expensive as they involve manipulations in m dimensional space, where m can be a large number. But after all the trouble of going to the high dimensional space, the result of the dot product is really a scala
Post a Comment
Read more

Linear Regression(With Gradient Descent ) Fully Explained| Machine Learning

Linear Regression (Gradient Descent Method) If you don't have any idea of Gradient Descent Algorithm , please check our previous post , there I have explained Gradient Descent Algorithm very well explained in brief. Now moving toward our current topic Linear Regression . In the previous post , we have just discussed the theory behind the Gradient Descent . Today we will learn Linear Regression where we will use  Gradient Descent to minimize the cost function. WHAT IS LINEAR REGRESSION: Suppose you are given a equation: y=2x1+3x2+4x3+1 and you are said to find the value at any point (1,1,1) corresponds to x1, x2, x3 respectively. You'll simply put the value of x1, x2, x3 into equation and tell me the answer :10,Right? But What if you are given different set of (x1, x2, x3,y) and you are said to find the equation. Here's what,Linear Regression Comes into picture.It helps us to find out or fit a Linear equation to datasets  given. Above equation can be easily tra
Post a Comment
Read more

Perceptron Algorithm | Pegasos Algorithm | Support Vector Machine | Linear Classifier Among Dataset | Deep Learning | Machine Learning

PERCEPTRON ALGORITHM Perceptron Algorithm is one of the most used algorithm in classifying data.It frequently seems little tough while learning first time. In today's post , We'll try to understand it well in brief with full explanation.  Perceptron Algorithm is used whenever given dataset can be classified into two parts or it has only two labels.(One can consider it as +1, -1) There are three version of Perceptron Algorithm: Simple Perceptron  Average Percepron Pegasos Algorithm or Support Vector  Machine Relax! Terms are sounding dangerous but they are really very easy.Let me explain you. 1.Simple Perceptron: The single target of perceptron algorithm is to find a Linear classifier( say Linear equation) on one side of which are all positively labelled point and on another side all negatively labelled points are there. As we know any linear equation can be written as, Y= θ .X + θ 0 Using Perceptron Our Aim is to find those value of θ vector an
Post a Comment
Read more