Exercise - Variational Mean Field Approximation for Univariate Gaussian

Introduction

[TODO]

In order to detect errors in your own code, execute the notebook cells containing assert or assert_almost_equal.

Requirements

[TODO]

Python Modules

%matplotlib inline
import numpy as np
from matplotlib import pyplot as plt
import scipy.stats
np.random.seed(40)

Data

Data: $X \sim \mathcal N(\mu, \frac{1}{\tau})$

Probability Density Function: $p(X \mid \mu, \tau) = \sqrt{\frac{\tau}{2\pi}} \exp\left( -\frac{\tau (X-\mu)^2 }{2} \right)$

with
-$\mu$: mean -$\sigma^2$: variance -$\tau =\frac{1}{\sigma^2}$ : precision

# generation of observed data
N = 10
mu = 10.
sigma = 2.
X = np.random.normal(mu, sigma, N)
x = np.arange(3,18,0.01)
p_x = scipy.stats.norm.pdf(x, loc=mu, scale=sigma)
plt.plot(x, p_x, label="true Gaussian")
plt.plot(X, np.zeros_like(X), "ro", label="Data")
plt.title("")
plt.xlabel("x")
plt.ylabel("p(x)")
plt.legend();

Exercises

Exercise - Mean Field Approximation of the Posterior

Find the mean field approximation of the posterior:

$p(\mu, \tau \mid X) \approx q(\mu)q(\tau)$.

• The observed data was sampled from a Gaussian distribution:$X \sim \mathcal N(\mu, \frac{1}{\tau})$.
• Use a constant prior for the mean and the precision$\tau = \frac{1}{\sigma^2}$: $p(\mu, \tau) = const. \quad \text{ for } \tau > 0$ $\theta = (\theta_0, \theta_1) = (\mu, \tau)$

Note: Typically the mean-field approximation in closed form is done with the conjugate distributions. But here we use the constant (improper) prior because it's a little bit easier.

Recap: Mean field approximation

Loop until convergence: $\log q({\theta}_k) = \mathbb E_{q_{-k}} \left[ \log{\hat p( {\theta} \mid {\mathcal D} )} \right] + const.$

Hint:

Use the "Sum of difference of the mean": $\sum_{i=1}^n (x_i-\mu)^2 = \sum_{i=1}^n(x_i-\bar{x})^2 + n(\bar{x} -\mu)^2$ with $\bar{x} = \frac{1}{n}\sum_{i=1}^n x_i$

Exercise - Proof

Show that: $q(\mu) = \mathcal N(\bar{X}, \frac{1}{\gamma_1})$ with$\gamma_1 = m \mathbb E_{q_{\tau}}[\tau]$

and

$q(\tau) = \text{Gamma}(\frac{m}{2}+1, \frac{2}{\gamma_2})$

with $\gamma_2= \sum_{i=1}^m \left(X_i^2\right)-m\bar X^2 + m\text{var}_{q_{\mu}}(\mu)$

Gamma distribution

$\text{Gamma}(k, \theta') = p(x) = x^{k-1}\frac{e^{-x/\theta'}}{\theta'^k\Gamma(k)}$ -$k$ is the shape -$\theta'$ the scale, -$\Gamma(.)$ is the Gamma function

Exercise - Implementation of the Mean Field Approximation

Implement the mean field approximation. Like always, you are free to implement as many helper functions as you want.

If everything is correct, executing the cells below should plot figures similar to these:

def optim(X=X,  mean_mu=1, sigma_quare_mu=1, loc_tau=1.,  scale_tau=1.):
raise NotImplementedError()
mean_mu, sigma_quare_mu, loc_tau, scale_tau = optim(X)
mean_mu, sigma_quare_mu, loc_tau, scale_tau
plt.figure(figsize=(12,4))

x = np.arange(5,15,0.01)
p_mu = scipy.stats.norm.pdf(x, loc=mean_mu, scale=np.sqrt(sigma_quare_mu))
ax = plt.subplot(121)
ax.plot(x, p_mu)
ax.set_xlabel("$\mu$")
ax.set_ylabel("q($\mu$)")
ax.set_title("Mean: q($\\mu$)")
print("true mu: ", mu)

x = np.arange(0,1,0.01)
p_tau = scipy.stats.gamma.pdf(x, a=loc_tau, scale=scale_tau)
ax = plt.subplot(122)
ax.plot(x, p_tau)
ax.set_xlabel("$\\tau$")
ax.set_ylabel("q($\\tau$)")
ax.set_title("Precision: q($\\tau$)")
print("true tau: ", 1/sigma**2)

Literature

[TODO]

The following license applies to the complete notebook, including code cells. It does however not apply to any referenced external media (e.g., images).

Exercise - Variational Mean Field Approximation for Univariate Gaussian
by Christian Herta