Deep Learning and AI: Introductory Readings and Resources

 

YouTube Vids

Personally, I like to read more than I like to watch a YouTube vid. I feel that I’m controlling the pace more, and can move my attention around where I want to go, when I’m reading.

A lot of people really like vids, though, and I’ve heard great things from my students about Andy Ng’s YouTube vids from his machine learning course at Stanford. I’m personally going through Hinton’s vids, as I want to focus on energy-based models.

• Andy Ng machine learning playlist (multiple one-hour lectures).
• Geoffrey Hinton, Machine Learning (multiple short lectures)
• Geoffrey Hinton – a one-hour overview lecture

 

What I Would Read If I Were Starting Today

 

• Deep Learning Tutorials AJM’s Note: I like these Deep Learning 4 Java (deeplearning4j) posts; they’re well done. I’m reading them even though I don’t code in Java, and don’t intend to – the tutorials are so nice and clear! Right now, I’m going through Deep Learning Tutorial: Restricted Boltzmann Machines as a resource for studying energy-based models. The others are comparably well done. To get going: (Menu on Left-Hand Sidebar) Introduction to Deep Learning > Deep Learning for Beginners > Free Deep- and Machine-Learning Courses Online. These are exactly the ones that my students have been using (students are liking Andy Ng’s Coursera course; I’m going through Geoffrey Hinton’s one).

 

Deep Learning Courses – Online, Well-Organized Materials, Free Access

 

• Yann LeCun’s Deep Learning course AJM’s Note: TBD.

 

Historical Surveys and Introductions

 

• A.J. Maren, A logical topology of neural networks, Proceedings of the Second Workshop in Neural Networks – Academia, Industry, NASA, & Defense (WNN-AIND 91) (Auburn, GA; Feb. 14-16, 1991). doi:10.13140/RG.2.1.3503.2806 PDF. AJM’s Note: When I was first teaching myself neural networks, I wrote a book on the subject. This is a condensation of the core chapter; it describes neural networks in terms of their structures. Many of the ones identified here have gone the way of dinosaurs. Others, as with all surviving species, have evolved further. This is good for historical context and overview.

 

Beyond Basic Neural Networks and Deep Learning

 

• Alexander Graves and Greg Wayne, Differentiable neural computers (a DeepMind blogpost, Wednesday, 12 October 2016). AJM’s Note: This paper by Graves and Wayne, and the following, talk about a different kind of neural network, particularly its implementation. Sometimes, it’s good to get our noses out of the standard sequence and take a look at broader problems / bigger picture.
• Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, Adrià Puigdomènech Badia, Karl Moritz Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield, Phil Blunsom, Koray Kavukcuoglu & Demis Hassabis, Hybrid computing using a neural network with dynamic external memory, Nature, 2016 (Published online 12 October 2016; accessed Oct. 21, 2016), doi:10.1038/nature20101,

 

What I Would NOT Read (as a Beginner)

About a year ago, I did a quick survey of deep learning books. There was only one (Deep Learning, by Goodfellow et al.) that was getting five-star Amazon reviews, so of course, I ordered it, and made it the “required reading” for the class.

Big mistake!

It’s a good book. It will be a classic in its own right.

For most of us, though, it is WAY TOO ABSTRACT.

Many of us are engineering-type thinkers. We can visualize just fine, but we typically don’t visualize really abstract things.

Fortunately, there’s been a lot more material coming out over this past year. Stay with the tutorials and, if you must read something abstract, there are a lot of good papers.

AJM’s Editorial Note: I’m building this page in real time, today, Thursday, August 31st, 2017. To see more, come back in a few hours, or tomorrow. Thank you!