# Exercise - Neural Network - Pen & Paper

## Introduction

In this notebook you'll visualize small neural networks using pen and paper. The idea is to gain an intuition about how the components of a neural network interact with each other, and how the input flows through the network to produce an output. In succeeding notebooks, you'll implement neural networks as code.

## Exercises

### Logistic Regression as a Neuron

Consider a basic neural network which consists of just one single neuron. As it turns out this is equivalent to the familiar model of logistic regression.

Given

• the input vector$\vec x$
• the weights$\vec \theta = \left[3, -1, 0.5, 2\right]$
• the first element of$\vec \theta$ (which is$\theta_0$) is the bias.

Illustrate a neuron which produces the logistic hypothesis as its ouput:

$o = \sigma( \theta_0 + \theta_1 \cdot x_1 + \theta_2 \cdot x_2 + \theta_3 \cdot x_3 )$

or written in vector form:

$o = \sigma( \theta_0 + \vec \theta_{1:} \cdot \vec x)$

Your illustration should include the following components

• Input layer
• Output layer
• Connections, weights, bias
• Sigmoid function$\sigma$

### Draw the Network

Given is the following information for a neural network:

\begin{equation} W_{HIDDEN} = \begin{pmatrix} 10 & -20 & 20 & -40 \ 20 & -40 & 0 & 0 \end{pmatrix} \end{equation}

\begin{equation} W_{OUTPUT} = \begin{pmatrix} 20 & 40 & -40 \end{pmatrix} \end{equation}

Each row of$W_{HIDDEN}$ and$W_{OUTPUT}$ represents the weights of one neuron in layer HIDDEN, resp. the OUTPUT layer. The first column equals the bias(es).

Further, activation function$g(z)$, which applies to all neurons in the network:

\begin{equation} g(z)=\left{\begin{array}{cc} 0 & z\le-10\ 1 & z\ge10 \ 0.5 & else\end{array} \right. \end{equation}

Draw a graph of the network$N_{SIMPLE}$ including all neurons and their connections. Note all weight and bias values on the corresponding nodes and edges of the graph.

### Calculate the forward pass

Use the given vectors$x_1, x_2, x_3$ to create a mini-batch matrix as input for the network and calculate its output. Only use matrix operations for the calculation and note all intermediate results.

\begin{equation} \vec{x}{(1)} = [0,1,1] ,\; \vec{x}{(2)} =[1,1,0] ,\; \vec{x}_{(3)} = [1,0,1] \end{equation}

## Literature

The following license applies to the complete notebook, including code cells. It does however not apply to any referenced external media (e.g., images).

Exercise - Neural Network - Pen & Paper
by Benjamin Voigt