Browsed by
Category: A Resource – Article

Interpreting Karl Friston (Round Deux)

Interpreting Karl Friston (Round Deux)

Comments 2 comments

He might be getting a Nobel prize some day. But – no one can understand him. You don’t believe me? Have a quick glance at Scott Alexander’s article, “God Help Us, Let’s Try To Understand Friston On Free Energy”. We’re referring, of course, to Karl Friston. I’ve spent the past three-and-a-half years studying Friston’s approach to free energy, which he treats as the guiding principle in the brain. He has extended the classic variational Bayes treatment (frontier-material in machine learning)…

Read More Read More

Generative vs. Discriminative – Where It All Began

Generative vs. Discriminative – Where It All Began

Comments 0 Comment

Working Through Salakhutdinov and Hinton’s “An Efficient Learning Procedure for Deep Boltzmann Machines”   We can accomplish a lot, using multiple layers trained with backpropagation. However (as we all know), there are limits to how many layers that we can train at once, if we’re relying strictly on backpropagation (or any other gradient-descent learning rule). This is what stalled out the neural networks community, from the mid-1990’s to the mid-2000’s. The breakthrough came from Hinton and his group, with a…

Read More Read More

How Ontologies Fit Into AI

How Ontologies Fit Into AI

Comments 0 Comment

Roy, A. Park, Y.J. and Pan, S.(2017, Sept. 21). Domain-SpecificWord Embeddings from Sparse Cybersecurity Texts, arXiv 1709.07470v1 [cs.CL]. pdf, accessed Apr. 25, 2018 by A.J.M. Abstract: Classic word embedding methods such as Word2Vec and GloVe work well when they are given a large text corpus. When the input texts are sparse as in many specialized domains (e.g., cybersecurity), these methods often fail to produce high-quality vectors. In this paper, we describe a novel method to train domain-specific word embeddings from…

Read More Read More

Ontologies, Knowledge Graphs, and AI: Getting from “Here” to “There” (Part 2)

Ontologies, Knowledge Graphs, and AI: Getting from “Here” to “There” (Part 2)

Comments 0 Comment

A Principled Approach to AI: Representations and Transitions:   In the last post, on “Moving Between Representation Levels: The Key to an AI System (Part 1),” I re-introduced one of the most important and fundamental AI topics: how we can effectively use multiple representation levels. If we’re going to build (or gauge the properties of) an AI system, we need a framework. The notion of representations, and of moving between representation levels, is as fundamental as we can get. In…

Read More Read More

Moving Between Representation Levels – the Key to Making an AI System Work (Part 1)

Moving Between Representation Levels – the Key to Making an AI System Work (Part 1)

Comments 2 comments

Representation Levels: The Key to Understanding AI   “No computation without representation” Jerry Fodor (1975). The Language of Thought, p.34. online access.   One of the key notions underlying artificial intelligence (AI) systems is not only that of knowledge representation, but that a good AI system will successively move disparate pieces of low-level, or signal-level information up the abstraction ladder. For example, an image understanding system will have a low-level component that extracts edges and regions from the image (or…

Read More Read More

Statistical Mechanics, the Future of AI, and Personal Stories

Statistical Mechanics, the Future of AI, and Personal Stories

Comments 0 Comment

Statistical Mechanics and Personal Stories (On the Same Page!)   Yikes! It’s Thursday morning already. I haven’t written to you for three weeks. That’s long enough that I have to pause and search my memory for my username to get into the website. Thanksgiving was lovely. The Thursday after that was grading, all day – and for several days before and after. By now, I (and most of you) have had a few days of recovery, from what has been…

Read More Read More

Third Stage Boost – Part 2: Implications of Neuromorphic Computing

Third Stage Boost – Part 2: Implications of Neuromorphic Computing

Comments 6 comments

Neuromorphic Computing: Statistical Mechanics & Criticality   Last week, I suggested that we were on the verge of something new, and referenced an article by von Bubnoff: A brain built from atomic switches [that] can learn, together with the follow-on article Brain Built on Switches. The key innovation described in this article was a silver mesh, as shown in the following figure. This mesh is a “network of microscopically thin intersecting silver wires,” grown via a combination of electrochemical and…

Read More Read More

Third Stage Boost: Statistical Mechanics and Neuromorphic Computing – Part 1

Third Stage Boost: Statistical Mechanics and Neuromorphic Computing – Part 1

Comments 8 comments

Next-Generation Neural Network Architectures: More Brain-Like   Three generations of artificial intelligence.. The third generation is emerging … right about … now. That’s what is shown in this figure, presented in log-time scale. Brief history of AI in log-time scale The first generation of AI, symbolic AI, began conceptually around 1954, and lasted until 1986; 32 years. On the log-time scale shown in the figure above, this entire era takes place under the first curve; the black bell-shaped curve on…

Read More Read More

Neg-Log-Sum-Exponent-Neg-Energy – That’s the Easy Part!

Neg-Log-Sum-Exponent-Neg-Energy – That’s the Easy Part!

Comments 12 comments

The Surprising (Hidden) “Gotcha” in This Energy Equation: A couple of days ago, I was doing one of my regular weekly online “Synch” sessions with my Deep Learning students. In a sort of “Beware, here there be dragons!” moment, I showed them this energy equation from the Hinton et al. (2012) Nature review paper on acoustic speech modeling: One of my students pointed out, “That equation looks kind of simple.” Well, he’s right. And I kind of bungled the answer,…

Read More Read More

Neural Network Architectures: Determining the Number of Hidden Nodes

Neural Network Architectures: Determining the Number of Hidden Nodes

Comments 0 Comment

Figuring Out the Number of Hidden Nodes: Then and Now   One of the most demanding questions in developing neural networks (of any size or complexity) is determining the architecture: number of layers, nodes-per-layer, and other factors. This was an important question in the late 1980’s and early 1990’s, when neural networks first emerged. Deciding on the network architecture details is even more challenging today. In this post, we’re going to look at some strategies for deciding on the number…

Read More Read More