From Regression to Neural Network

In order to really understand how neural networks work, it is essential to understand their core concepts. Practically, a multi layer perceptron (fully connected feed forward network) consists of many logistic regression tasks. So in order to understand neural networks you need to understand logistic regression, which is practically linear regression extended with a sigmoid function.

Linear regression is used to predict continous values (e.g. the temperature tomorrow), based on some features (e.g. temperature today, humidity, calender week, etc…). By adding a sigmoid function at the end of linear regression, the final output will be a value between 0.0 and 1.0. Adding a threshold at 0.5 then gives you a binary classifier (e.g. warm or cold day tomorrow). Stacking several logistic regressions in the same layer (after the input layer) and adding a last logistic regression after that, results in a fully connected neural network with one hidden layer. The following gif depicts the whole picture:

linear-logistic-nn

The following notebooks chronologically teach all necessary steps to understand the evoltuion from linear regression to neural networks: