# Exercise - Estimating Mean and Standard Deviation of Normal Distribution with pyro

## Introduction

[TODO]

In order to detect errors in your own code, execute the notebook cells containing assert or assert_almost_equal.

## Requirements

[TODO]

### Python Modules

import numpy as np

import scipy.stats
from scipy.stats import norm

from matplotlib import pyplot as plt
from IPython.core.pylabtools import figsize

%matplotlib inline
import torch
from torch.distributions import constraints

import pyro
import pyro.infer
import pyro.optim
import pyro.distributions as dist

## Data

Data:

$X \sim \mathcal N(\mu, \frac{1}{\tau})$

Probability Density Function:

$p(X \mid \mu, \tau) = \sqrt{\frac{\tau}{2\pi}} \exp\left( -\frac{\tau (X-\mu)^2 }{2} \right)$

with

• $\mu$: mean
• $\sigma^2$: variance
• $\tau =\frac{1}{\sigma^2}$ : precision
dtype=torch.float32
torch.manual_seed(101);
np.random.seed(10)
# generate observed data
N = 10
mu_ = 10.
sigma_=2.
X = np.random.normal(mu_, sigma_, N)
X = np.array(X, dtype=np.float32)
X
x = np.arange(3,18,0.01)
p_x = scipy.stats.norm.pdf(x, loc=mu_, scale=sigma_)
plt.plot(x, p_x, label="true Gaussian")
plt.plot(X, np.zeros_like(X), "ro", label="Data")
plt.title("")
plt.xlabel("x")
plt.ylabel("p(x)")
plt.legend();

## Exercises

### Exercise - Model

Build the following model with pyro

• The observed data should come from a Gaussian distribution $\mathcal N(\mu, \sigma^2)$.
• Use a Uniform prior for the mean: $\text{Uniform}(-25,25)$
• Use a constant $\tau=1/4$ for the precision.

• Implement the "Guide".
• Use as variational distribution also a Gaussian.
$\mu \sim \mathcal N(mean_{\mu}, scale_{\mu}^2)$

### Exercise - Estimate Mean

Optimize the variational parameters, i.e. find vales for $mean_{\mu}, scale_{\mu}$

X
plt.xlabel("# iteration")
plt.ylabel("MC-Estimate of ELBO")
plt.plot(range(len(losses)), losses)


### Exercise - Estimate Precision $\tau$ (and Mean)

Extend the model and the Guide by using additionally a variational distribution for $\tau$:

• Use a Uniform distribution for $\tau \sim \text{Uniform}(0.01, 2)$
• Use a Gamma distribution as variational distribution for $\tau$: $\text{Gamma}(a, b)$
• Find the parameters $a, b$ (and $mean_{\mu}, scale_{\mu}$) via optimization.

If your extensions are correct, executing the cells below should plot figures similar to these:

# Adjust the strings according to your names for
# the parameters "mu_mean", etc...
mu_mean_param = pyro.param("guide_mu_mean")
mu_scale_param = pyro.param("guide_mu_scale")
mu_mean_param, mu_scale_param
# Adjust the strings according to your names for
# the parameters "mu_mean", etc...
tau_concentration_param = pyro.param("guide_tau_concentration")
tau_rate_param = pyro.param("guide_tau_rate")
tau_concentration_param, tau_rate_param
plt.figure(figsize=(12,4))

mu_mean = mu_mean_param.detach().numpy()
mu_scale = mu_scale_param.detach().numpy()

x = np.arange(5,15,0.01)
p_mu = scipy.stats.norm.pdf(x, loc=mu_mean, scale=np.sqrt(mu_scale))
ax = plt.subplot(121)
ax.plot(x, p_mu)
ax.set_xlabel("$\mu$")
ax.set_ylabel("q($\mu$)")
ax.set_title("Mean: q($\\mu$)")
print("true mu: ", mu_)

tau_concentration =tau_concentration_param.detach().numpy()
tau_rate = tau_rate_param.detach().numpy()

x = np.arange(0,1,0.01)
p_tau = scipy.stats.gamma.pdf(x, a=tau_concentration, scale=1/tau_rate)
ax = plt.subplot(122)
ax.plot(x, p_tau)
ax.set_xlabel("$\\tau$")
ax.set_ylabel("q($\\tau$)")
ax.set_title("Precision: q($\\tau$)")
print("true tau: ", 1/sigma_**2)

true mu:  10.0
true tau:  0.25


## Literature

[TODO]

The following license applies to the complete notebook, including code cells. It does however not apply to any referenced external media (e.g., images).

Exercise - Variational Mean Field Approximation for Univariate Gaussian
by Christian Herta