He might be getting a Nobel prize some day.

But – no one can understand him.

You don’t believe me?

Have a quick glance at Scott Alexander’s article, “God Help Us, Let’s Try To Understand Friston On Free Energy”.

We’re referring, of course, to Karl Friston.

I’ve spent the past three-and-a-half years studying Friston’s approach to free energy, which he treats as the guiding principle in the brain. He has extended the classic variational Bayes treatment (frontier-material in machine learning) to the next logical level.

This requires some fine and elegant mathematics. 

He presents (in the grand tradition of physicists everywhere) his final results. He references Matthew Beal’s 2003 dissertation, as well as a few others, as starting points.

In his dissertation, Beal gives us a very nice derivation for the (by now classic) approach to “Variational Algorithms for Approximate Bayesian Inference.”

However, Karl takes this a full step beyond – he’s not explicitly modeling a system; he’s separating an external system (that which we’d ultimately like to model) from a “representational system” by means of a Markov blanket. (The Markov blanket provides a means for passing signals into and out from the representational system, so that it can be influenced by and in turn influence the external system.)

Then … and THEN … he extends the variational Bayes method so that it can model the external system via modeling the representational system (plus its Markov blanket inputs).

So the whole mathematical thing (already very arcane) becomes much more complex and delicate.

Up until now, we haven’t had a Rosetta stone that would facilitate translation from the classic formulation (e.g., Beal and others) to the extended formulation (aka Friston’s).

Creating such a Rosetta stone was a primary motivation in my writing the derivation for the variational Bayes approach – first using Beal’s notation, and then using Friston’s.

I semi-quasi-sort-of published this article, on this website, about two years ago. But it has taken me two years since that time (that is, up until now) to put together what I believe to be the (ahem!) authoritative translation.

Jump to the arXiv article HERE:

Derivation of Variational Bayes

 

Here’s a crucial figure from that article; it gives a diagrammatic illustration of the key variational Bayes equation.

{More to follow … }

Live free or die, my friend –

AJ Maren

Live free or die: Death is not the worst of evils.
Attr. to Gen. John Stark, American Revolutionary War

References for Variational Bayes

• Beal, M. (2003). Variational Algorithms for Approximate Bayesian Inference, Ph.D. Thesis, Gatsby Computational Neuroscience Unit, University College London. pdf.
• Blei, D.M., Kucukelbir, A., McAuliffe, J.D. (2016), Variational inference: a review for statisticians. pdf.
• Feynman, R.P. (1972, 1998). Statistical Mechanics: A Set of Lectures. Reading, MA: Addison-Wesley; Amazon book listing.
• Friston, K.; Levin, M.; Sengupta, B.; Pezzulo, G. Knowing one’s place: a free-energy approach to pattern regulation. J. R. Soc. Interface 2015, 12, 20141383. doi:10.1098/rsif.2014.1383. pdf.
• Friston, K. Life as we know it. Journal of The Royal Society Interface 2013, 10. pdf.
• Friston, K. The free-energy principle: a unified brain theory? Nature Reviews Neuroscience 2010, 11 (2), 127-138. online access.

Previous Related Posts

• How to Read Karl Friston (In the Original Greek). (Note: This was my first draft; the post has links to Friston’s works along with links to Beal, Blei, and others.)
• Approximate Bayesian Inference. (Note: This post has links to lots of good resources (mostly tech-blogs) attempting to understand and present basic variational Bayes. A bit historical; all before late 2016.)