Readings – Deep Learning – Energy-Based Methods

Readings – Deep Learning – Energy-Based Methods

The “Rocky Mountains” of Deep Learning

Good Introductions

  • Bengio, Y. (2009), Learning Deep Architectures for AI, Foundations and Trends in Machine Learning, (January, 2009). doi:10.1561/2200000006. pdf.

Dr. A.J.’s Secret Deep Learning Reading List

Right now, I’m focusing on the statistical mechanics underpinnings of deep learning algorithms. This is mostly a crib-sheet for myself, so that I can find the sites and papers that are useful to me right now. ( Updated: August 18, 2017)

  • Salakhutdinov, R. and Hinton, G. (2012), An Efficient Learning Procedure for Deep Boltzmann Machines, Neural Computation, 24 (8) (August 2012), 1967-2006. article online,
    • Abstract: We present a new learning algorithm for Boltzmann machines that contain many layers of hidden variables. Data-dependent statistics are estimated using a variational approximation that tends to focus on a single mode, and data-independent statistics are estimated using persistent Markov chains. The use of two quite different techniques for estimating the two types of statistic that enter into the gradient of the log likelihood makes it practical to learn Boltzmann machines with multiple hidden layers and millions of parameters.
    • AJM’s notes: Good history and intro; good context, links to variational theory (which I still have to read in order to assess).
  • Hopfield, J. et al. (2016), Neural Networks for Machine Learning: Lecture 11a – Hopfield Networks (Feb. 4, 2016), YouTube course link. AJM’s notes: Good vid series; I’m watching and re-watching the whole thing.
  • LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M.-A. and Huang, F.J. (2006), A Tutorial on Energy-Based Learning, in Predicting Structured Data, G. Bakir, T. Hofman, B. Sch¨olkopf, A. Smola, B. Taskar (eds) (MIT Press).
  • Ngiam, J., Chen, Z.-H., Koh, P.W., Ng, A.Y. (2011), Learning Deep Energy Models, Proc. 28th Intl Conf Machine Learning (Bellevue, WA, USA).
  • Ng, S.-K. and McLachlan, G.J. (2004), Using the EM Algorithm to Train Neural Networks: Misconceptions and a New Algorithm for Multiclass Classification, IEEE Trans. Neural Networks, 15 (3) (May 2004), 738-749.
  • Song, J.-M. and Yuan, C. (2016), Learning Boltzmann Machine with EM-like Method, ArXiv 1609.01840v1 [cs.LG] 7 Sept. 2016.
  • How to Compute the Free Energy of a RBM Given Its Energy; query and answer (2015) on CrossValidated; answer by akxlr, StackExchange HERE; AJM’s notes: Read this ASAP; I’m looking for the interpretation of Boltzmann machine in terms of free energy
  • Martin, C.H. (2016), Improving RBMs with physical chemistry, Blogpost on Calculated Content (Oct. 15, 2016), Blogpost HERE. AJM’s notes: Charles always writes a good blogpost; this is to help fill out the whole thing.
  • Larochelle, H. (2013), Neural networks [5.3] : Restricted Boltzmann machine – free energy (Nov. 15, 2013), YouTube Link HERE. AJM’s notes: More on my reading/watching list; I’m in the middle of this.
  • Restricted Boltzmann machines – Free Energy StackExchange link HERE. AJM’s notes: Probability and RBMs, another thing on my reading/watching list; I’m in the middle of this.
  • Why is EM algorithm not used for learning the parameters of Neural Network? (2015), Quora question and answers Quora link HERE. AJM’s notes:Another bit-and-piece on Expectation Maximzation, another thing on my reading/watching list; I’m in the middle of this one also.
  • Koller, D., Lecture 24 – Expectation Maximization: Intro Coursera course link HERE. AJM’s notes:Just getting started with this; it’s good.

Found some really great reading lists by working my way through the Data Science Group posts (via LinkedIn groups). Here’s one that looks excellent: