Let's approximate a complicated distribution P(x) by a simpler
distribution Q(x), possibly a separable distribution.
It's often the case that variational free energy minimization
(also known as mean field) leads to an approximating distribution
Q that is `more compact' than the true distribution.
Is there in fact a theorem that we could prove along the lines of
`optimized $Q$ is always more compact'?
We show, with a counterexample,
that the folk theorem about variational
approximations being `more compact' is not always true.
All postscript files are compressed with gzip - see this page for advice about gzip, if needed.