AI, Deep Learning, and Emerging Computing Technologies

 

This is a short history of AI and neural networks, with a purpose.

I think we’re poised for a major breakthrough – something different from and beyond deep learning.

Here’s why.

 

The Two Eras of AI

 

We’ve had two major forms of AI computing over the past 64 years. The following diagram illustrates it:

This shows, in with time given in a log-format on the x-axis, that we’ve had two major eras of AI so far:

1. Symbolic AI: 1952 – 1984 (32 years), and
2. Connectionist (neural networks) AI: 1985 – 2017 (32 years).

My gut tells me that we’re ready for a new paradigm, and that it will emerge … about now.

 

Symbolic AI

 

Here’s the essence: From 1952 to 1984 (approximately), the AI world was dominated by symbolic logic, with the notions that everything could be expressed using either declarative or procedural logic. This was based on early work in cognitive science (Simon & Newell, for example) along with breakthroughs in establishing the first major AI systems. This led to expert systems and knowledge extraction (the process of eliciting knowledge from experts). Expert systems hit a wall around 1983 – 1986, as we became aware that these systems were brittle and hard to update / maintain. There was sufficient unhappiness with these systems that people were receptive, as never before, to the notion of neural networks.

This does not mean that symbolic AI is dead; far from it. Witness Google’s Knowledge Graph. Witness IBM’s Watson. Witness the plethora of digital personal assistants (the “Siri’s” of the world) which purport to help us. These new AIs include high-level, ontology-based knowledge as well as lots of low-level signal and data processing. However, today’s “cognitive computing” is very different from the older systems, based strictly (and exclusively) on predicate-logic and similar symbolic AI methods. In fact, these newer systems often involve coupling symbolic knowledge with signal processing, using connectionist (neural network) algorithms to support massive data-handling.

 

Neural Networks, Deep Learning, and Other Connectionist Architectures

 

Neural networks became a practical option, starting in 1986 with the release of Rumelhart and McClelland’s Parallel Distributed Processing, which pulled together seminal neural networks papers and introduced the backpropagation and Boltmann machine (simulated annealing) learning algorithms. This made it possible to train neural networks; up until then, there was no realistic way to train them.

A huge flurry of work ensued, carrying well through the 1990’s. During the 2000’s, that work slowed down as the limitations of simple neural networks became apparent. There was a lot of work on practical applications, including knowledge discovery in large data corpora (following 9/11, when there was a lot of funding for intelligence-related work). The economic crisis of 2008 halted almost all work. A few dedicated persons, notably Geoffrey Hinton and his students, continued developing advanced neural network architectures, and working on the basic theory.

Their work achieved public acclaim when Hinton students created “deep learning” neural networks and when Yann LeCun created very effective CNNs (convolution neural networks), and these people began winning Kaggle competitions, as well as showing performance in other areas. The key to much of this was hugely more powerful processing systems. NVIDIA quickly became a leader, re-envisioning their powerful graphics cards as neural networks engines.

(For a very useful interview with LeCun, see HERE: LeCun Interview; also see LeCun’s Research Page.)

Here’s a detailed look at some major breakthroughs in the late 2000’s and 2010’s:

Through a combination methods, different neural networks began achieving human-like performance in tasks ranging from image classification to speech translation. In some tasks, the neural network could outperform a human (e.g., identification of a tumor in a medical image). Very recently, advances in haptic (touch) sensing, together with visual object recognition, have made robotics very effective for an increasing range of applications, portending a huge impact on smart manufacturing and other potential tasks, including agriculture. Also, the breakthroughs in computer vision (classifying objects in an image and other tasks) began to make autonomous vehicles a viable technology.

Aiding all of this, NVIDIA and other companies (Intel, QUALCOMM) are continuing to develop ever-more-powerful processors. NVIDIA CEO Jensen Huang has boldly stated that Moore’s Law (governing the evolution of processing power) is dead. Whether we believe that or not, there are clearly major advances in computing platforms that will power the next generation of computer-based intelligence.

In short, we’ve seen huge breakthroughs in neural networks capabilities, all within the past several years.

 

The Next AI Era – Brain-Based Computing

 

With all that said, and despite some very real fundamental shifts in machine learning algorithms, the algorithms underlying recent advances are ones that were invented about three decades ago; early in the neural network era. Backpropagation is still the most widely-used method for training. (LeCun has re-envisioned it as an energy-minimization method, and has created some major breakthroughs, but the basic idea is still there.) Neural network architectures such as RBMs (Restricted Boltzmann Machines), CNNs (convolutional neural networks), DBNs (Deep Belief Networks) and even autoencoders have been around for some time. Even the newest architecture – GANs (generative adversarial networks) represents a combination of two networks, each gaming each other, and each using a different learning method – supervised vs. unsupervised.

Thirty-two years. It’s about time for some new thoughts to emerge.

My guess is that they will, starting soon.

Exactly what I think these will be?

Let’s hold on that for now. More forthcoming.

AJM’s Note, Thursday, Aug. 31, 2017: Nice surprise! William Vorhies, in The Three Ages of AI – Figuring Out Where We Are (May 9, 2017 at 9:41am) agrees with me; we come at the same conclusion from different points of view, but our results are the same. Check out his blog also!

 

 

Here’s some related future forecasts reading:

• Our immediate future: The Master Algorithm (and other good reads); these are all readable by the intelligent lay reader, and includes a link to some good science fiction with AI themes,
• Technological advances, the singularity, and critical points: Technological singularities and accelerating change, and
• Carrying capacity of the earth: Timeframes and critical dependencies, including Paul Chefurka on the carrying capacity of the earth.