ML-Fundamentals - Bias Variance Tradeoff
Table of Contents
If you completed the exercises simple-linear-regression, multivariate-linear-regression and logistic-linear-regression you know how to fit these models according to your training data.
This alone so far has no practical use case. The benefit of learning a model is to predict unseen data. Additionally, only with unseen data your model has not learnt from, it is possible to say if your model generalizes well or not. One way to measure this, is calculating the out of sample error , which consists of the measures bias and variance.
In this notebook you will calculate two simple hypothesis for linear regression based on training data and compare them with the use of unseen validation data by calculating , bias and variance.
You should have a basic knowledge of:
- Univariate linear regression
- Out of sample error (bias variance)
Suitable sources for acquiring this knowledge are:
By deep.TEACHING convention, all python modules needed to run the notebook are loaded centrally at the beginning.
import numpy as np import matplotlib.pyplot as plt import hashlib
def round_and_hash(value, precision=4, dtype=np.float32): """ Function to round and hash a scalar or numpy array of scalars. Used to compare results with true solutions without spoiling the solution. """ rounded = np.array([value], dtype=dtype).round(decimals=precision) hashed = hashlib.md5(rounded).hexdigest() return hashed
Exercise inspired by lecture 8 from:
Implement the following simulation for calculating the bias and variance:
- is unformly distributed in the interval of
- The unknown target function is the sinus function:
- There is no noise if a -value is drawn from , so
We consider two hypthesis sets:
- Hypthesis set :
- Hypthesis set : (a constant)
Do the following 10.000 times:
Draw two random examples and from and calculate the corrsponding s to get (training data)
Using your training data calculate the parameters for and the parameter for
Numerically calculate the out of sample error for and for 100 data points uniformly distributed in the interval of (validation data)
Now calculate the average and of all 10.000 experiments.
Also calculate the average "out of sample error" for both hypothesis sets and .
Use the above to calculate the bias and the variance.
Plot the target function together with both average hypotheses and using the average and
Considering your results, which hypothesis seems to better model the target function?
Practically this explanation is all you need to solve the exercise. You are free to complete it without any further guiding or by proceeding with this notebook.
Implement the function to draw two random training examples with:
def train_data(): raise NotImplementedError()
x_train, y_train = train_data() print(x_train, y_train)
# If your implementation is correct, these tests should not throw an exception assert len(x_train) == 2 assert len(y_train) == 2 np.testing.assert_array_equal(np.sin(x_train), y_train) for i in range(1000): x_tmp, _ = train_data() assert x_tmp.min() >= 0.0 assert x_tmp.max() <= 2*np.pi
For our training data we will now model two different hypothesis sets:
Implement the functions to calculate the parameters for and for using the two drawn examples.
For later purpose (passing functions as argument) it is important that both functions accept the same amount of parameters and also return the same amount. Therefore we also pass to
get_w, although we do not need it. And for the same reason
get_thetas should return a list of two values instead of two seperate values.
def get_thetas(x, y): raise NotImplementedError() def get_w(x, y): raise NotImplementedError()
thetas = get_thetas(x_train, y_train) w = get_w(x_train, y_train) print(thetas, thetas) print(w)
# If your implementation is correct, these tests should not throw an exception x_train_temp = np.array([0,1]) y_train_temp = np.array([np.sin(x_i) for x_i in x_train_temp]) thetas_test = get_thetas(x_train_temp, y_train_temp) w_test = get_w(x_train_temp, y_train_temp) np.testing.assert_almost_equal(thetas_test, 0.0) np.testing.assert_almost_equal(thetas_test, 0.8414709848078965) np.testing.assert_almost_equal(w_test, 0.42073549240394825)
Implement the hypothesis and . Your function should return a function.
def get_hypothesis_1(thetas): raise NotImplementedError() def get_hypothesis_2(w): raise NotImplementedError()
# we want to compute numerically the expectation w.r.t. x x_grid = np.linspace(0, 2*np.pi, 100) y_grid = np.sin(x_grid)
# If your implementation is correct, these tests should not throw an exception h1_test = get_hypothesis_1(thetas_test) h2_test = get_hypothesis_2(w_test) np.testing.assert_almost_equal(h1_test(x_grid), 0.5340523361780719) np.testing.assert_almost_equal(h2_test(x_grid), 0.42073549240394825)
Following the original exercise it is not yet necessary to plot anything. But it also does not hurt to do so, since we need to implement code for the plot anyways.
Write the function to plot:
- the two examples and
- the true target function in the interval .
- the hypothesis in the interval
- the hypothesis in the interval
Your plot should look similar to this one:
def plot_true_target_function_x_y_h1_h2(x, y, hypothesis1, hypothesis2): raise NotImplementedError()
thetas = get_thetas(x_train, y_train) w = get_w(x_train, y_train) plot_true_target_function_x_y_h1_h2(x_train, y_train, get_hypothesis_1(thetas), get_hypothesis_2(w))
Out of Sample Error
The out of sample error is the expected error on new unseen data.
In our example we don't need to take the expectation w.r.t. because we have have no noise:
Here we will compute the out of sample error numerically.
We already have discretized the -axis (
x_grid) for .
So to compute the expectation we just need to average over
x_grid (remember is uniform).
- : number of elements in
- is the -element of
Implement the function to numerically calculate the out of sample error with the mean squared error as loss function.
def out_of_sample_error(y_preds, y): raise NotImplementedError()
# If your implementation is correct, these tests should not throw an exception e_out_h1_test = out_of_sample_error(h1_test(x_grid), y_grid) np.testing.assert_almost_equal(e_out_h1_test, 11.525485917588728)
Now instead of drawing two examples (one training data set), draw
now 10.000 times different training sets with two examples.
Calculate for the differnt and numerically.
For each run, keep track of the following parameters and return them at the end of the function:
def run_experiment(m, x_val, y_val): raise NotImplementedError()
xs, ys, t0s, t1s, ws, e_out_h1s, e_out_h2s = run_experiment( 10000, x_grid, y_grid)
Average and Plot
Now we can calculate the average of , and and already plot the resulting averaged and together with the target function .
Your plot should look similar to the one below:
t0_avg = t0s.mean() t1_avg = t1s.mean() thetas_avg = [t0_avg, t1_avg] w_avg = ws.mean() h1_avg = get_hypothesis_1(thetas_avg) h2_avg = get_hypothesis_2(w_avg) print(thetas_avg)
plot_true_target_function_x_y_h1_h2(, , h1_avg, h2_avg)
expectation_Eout_1 = e_out_h1s.mean() print ("expectation of E_out of model 1:", expectation_Eout_1)
expectation_Eout_2 = e_out_h2s.mean() print ("expectation of E_out of model 2:", expectation_Eout_2)
The bias for the mean-squared error is:
The expectation w.r.t. vanishes if we have no noise:
- the average hypothesis
Implement the function to calculate the expecation of the bias numerically.
def bias(y_true, y_predicted): raise NotImplementedError()
bias_1 = bias(y_grid, h1_avg(x_grid)) print ("Bias of model 1:", bias_1)
bias_2 = bias(y_grid, h2_avg(x_grid)) print ("Bias of model 2:", bias_2)
Variance for the mean-squared-error:
- the average hypothesis
- the learnt hypothesis for training data set .
Implement the function to calculate the variances for each of the 10.000 experiements and return them as list or array.
Now we benefit from our implementation of
get_hypothesis2, which accept and return the same amount of parameters, so we can write a generalized function.
thetas_avg = [0.17, 0.68]
def variances(hypothesis_func, param_func, xs, ys, x_val, y_preds): return NotImplementedError()
vars_1 = variances(get_hypothesis_1, get_thetas, xs, ys, x_grid, h1_avg(x_grid)) var_1_avg = vars_1.mean() print(var_1_avg)
vars_2 = variances(get_hypothesis_2, get_w, xs, ys, x_grid, h2_avg(x_grid)).mean() var_2_avg = vars_2.mean() print(var_2_avg)
print("model 1: E_out ≈ bias^2 + variance: %f ≈ %f + %f" % (expectation_Eout_1, bias_1, var_1_avg)) print("model 2: E_out ≈ bias^2 + variance: %f ≈ %f + %f" % (expectation_Eout_2, bias_2, var_2_avg))
Summary and Outlook
Notebook License (CC-BY-SA 4.0)
The following license applies to the complete notebook, including code cells. It does however not apply to any referenced external media (e.g., images).
Exercise: Bias Variance Tradeoff
by Christian Herta, Klaus Strohmenger
is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Based on a work at https://gitlab.com/deep.TEACHING.
Code License (MIT)
The following license only applies to code cells of the notebook.
Copyright 2018 Christian Herta, Klaus Strohmenger
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.