‫In the last lecture, we discussed a single cell called Perceptron.
‫Now, in this lecture, we are going to extend the concepts that we learned in the last one.
‫I told you that a Perceptron takes in binary input that is one and zero and gives out a single binary
‫output.
‫But there is no logical reason to put this limitation.
‫We can easily extend this to any real input values.
‫So instead of having black and white only or zero and one only, we can have different shades of grey
‫as well.
‫That is, we accept any real value as input. The weights and threshold function in the same way.
‫Next, we will take a look at this equation of Perceptron will slightly modify it to reach at generally
‫used equation. In this equation
‫We are multiplying weights, adding these terms and comparing them with the threshold.
‫We will make a small change here, bring this threshold to the left and write
‫this new term as B basically it means that we have B is equal to minus threshold
‫People usually call this constant as the bias doesn't really make any difference.
‫But this is the mathematical representation of Perceptron, as you would find in most of the books.
‫Now, let's move on.
‫and look at the graphical representation of this function.
‫If you look at this graph, if the calculated value of this left part, that is summation of weight
‫multiplied by features, plus the bias, if the summation if this left part is less than zero, the
‫output comes out to be zero.
‫So you can see in the graph till zero, the output of the function is also zero.
‫When this left part is greater than zero.
‫This function suddenly activates and gives an output of one.
‫This type of function is called a simple step function.
‫This is one type of activation function activation functions are basically those functions which take
‫into account some type of threshold value. Here
‫the threshold value is zero.
‫And this function takes a sudden step at this threshold value, which is why it is called a step activation
‫function.
‫There are many other types of activation functions.
‫Most popular one is the sigmoid function.
‫It is a pictorial representation of how sigmoid function looks.
‫It is a smooth s shaped curve.
‫It also has a minimum of zero at minus infinity and maximum of one at plus infinity.
‫But instead of having a step and rising suddenly, this function rises gradually and continuously.
‫This function is also called logistic function and is also used in logistic regression, which is a
‫very basic classification algorithm.
‫Now, this sigmoid function solves a major problem that we have with this step function when we are training
‫our Perceptron using historical data to find the value of weights and threshold.
‫This step function is very sensitive to individual observations.
‫For example, when we are classifying fashion objects in our fashion MNISD dataset and our algorithm
‫is misclassifying a particular image of boots as trousers to rectify this, our model will need to find
‫new weights and bias values.
‫This is where the problem comes.
‫Small change in the weight and the bias values will completely flip the output for a lot of the other observations.
‫This makes the step function very hard to control with sigmoid function.
‫The change is gradual, so it is easier to control the behavior.
‫Now, when we replace this step function with a sigmoid activation function, we call this new cell
‫as a sigmoid neuron or a logistic neuron instead of Perceptron.
‫Mathematically, a sigmoid function formula looks like this.
‫It is sigmoid
‫of z is equal to one upon one.
‫1+ e raise to the power of minus z
‫And if you plot this function on the graph, that is, if you have the Z on X axis and you calculate
‫the value of this function using this formula and plot it on the Y axis.
‫This is how this formula looks like.
‫Now we will replace the value of Z.
‫With the summation plus bias value.
‫So WjXj plus B was the input to our activation function.
‫So we input this in place of z.
‫So this is what the output of our neuron looks like.
‫It is one upon one plus exponential minus summation of words with features.
‫Minus B, if you calculate this value, it will always lie between zero to one and it will have a shape
‫like this.
‫So you can compare it with this step function also in step function.
‫We calculated output using this formula where we got zero.
‫If this summation was less than zero and we got one, if the summation was greater than equal to zero.
‫We have replaced this step with a sigmoid function.
‫This is a continuous function.
‫We do not need two parts to it.
‫So we just input the value of Wj Xj and the bias to calculate the output, which is a continuous
‫function.
‫Now with this our artificial neural cell is ready, which takes in any number of real value inputs and
‫gives an output between zero and one.
‫It is time to create an artificial neural network, which is basically a network of these individual
‫cells.
‫So just a brief recap of this class.
‫Initially, I said that we taken binary input and gave out one single binary output.
‫We replaced the input from binary to any real value, and we have replaced the binary output to a value
‫between zero and one.
‫So in this generalized form, we taken any input which have any real value and we get one output with
‫lies between zero and one.