Most of us don’t always use graph language to describe neural networks, but if we dig into the implications of graph theory language, we get some surprising (and very useful) insights!

We probably all know that a typical feedforward neural network can be described as a “directed graph.” Many of us also know that a restricted Boltzmann machine (RBM) is an “undirected graph.” In this little difference of terms, there is a wealth of meaning.

Salakhutdinov, Mnih, and Hinton (2007; see reference at end of post) provide a good description of the restricted Boltzmann machine (RBM), emphasizing that it is a two-layer undirected graph. This is very (and necessarily) different from the directed graph structure of a Multilayer Perceptron (MLP), which is the structure used for neural network classification applications.

Chapter 7 of my book (just released in draft form, see the link at the end of this post) takes you through the implications of an undirected graph (autoencoder network) vs. a directed graph (classifier network). We can make an autoencoder function as a classifier, and (with suitable working of the output layer) make a classifier network work as an autoencoder, but it helps so much if we can understand the inherent differences between the two.

Live free or die, my friend –

AJ Maren

Live free or die: Death is not the worst of evils.
Attr. to Gen. John Stark, American Revolutionary War

Resources

Link to The Book Page

• Book: Statistical Mechanics, Neural Networks, and Artificial Intelligence

• Salakhutdinov, R., Minh, A., and Hinton, G.E. (2007). Restricted Boltzmann Machines for Collaborative Filtering, Proc. 24th Int’l Conf Machine Learning (Corvallis, OR, June 20-24, 2007). pdf