## Introduction

In statistic the bias of an estimator is the difference between the true value of a certain quantity of interest and the calculated expected value from the data. A bias may occur either if one missunderstoods the true nature of the quantity or uses a worng mathematical calculation. In this notebook you will see an example for the second case.

## Requirements

### Knowledge

To complete this exercise notebook you should possess knowledge about the following topics:

• Monte Carlo estimator
• Jensen's inequality, see notebook 'exercise-jensen-inequality'

## Python Modules

import numpy as np
import scipy.stats
from matplotlib import pyplot as plt

%matplotlib inline

## Exercise

Assume the data generating distribution$p(x)$, which is a mixture of two Gaussians$\mathcal N(10,0.6)$ and$\mathcal N(15,1.4)$ .

# data generating distribution is a mixture of two Gaussians with parameters
loc1=10
loc2=15
scale1=0.6
scale2=1.4

# samples from p(x)
def samples(n=10000, p=0.3, loc1=loc1, loc2=loc2, scale1=scale1, scale2=scale2):
z = np.random.binomial(n,p=p)
x1 = np.random.normal(loc1,scale1,z)
x2 = np.random.normal(loc2,scale2,n-z)
x = np.concatenate((x1,x2))
np.random.shuffle(x)
return x

x = samples()
_ = plt.hist(x, bins=int(np.sqrt(len(x))))

Assume you need an approximation for$K = \mathbb E_{p(x)}[\log(x)]$.

What's wrong with the following Monte Carlo estimator

$\hat K = \log \left [\frac{1}{M}\sum_{i=1}^M x_i \right]$ with$x_i \sim p(x)$ ?

Do you expect that$K = \hat K$ ?

#### Hint

Read about Jensen's inequality, and argue for the specific case of$\log$.

Give the formula for an unbiased estimator$\tilde K$ of$K$ !

Implement both estimators and compare their results for the mixture of Gaussian's samples.

#### Hint

For better results sample 100 samples and take the average of the calculated means.

The following license applies to the complete notebook, including code cells. It does however not apply to any referenced external media (e.g., images).

exercise-biased-monte-carlo-estimator
by Christian Herta