Statistical Mechanics and Personal Stories (On the Same Page!)

 

Yikes! It’s Thursday morning already. I haven’t written to you for three weeks. That’s long enough that I have to pause and search my memory for my username to get into the website.

Thanksgiving was lovely. The Thursday after that was grading, all day – and for several days before and after.

By now, I (and most of you) have had a few days of recovery, from what has been (always) a grueling quarter. Even if you haven’t been enrolled in classes, you’ve likely had an extra workload as the year comes to a close.

As promised, I’m writing to you again … I told you that it would be every Thursday, like clockwork … it’s just that the clock was a little pre-occupied over the preceding few weeks.

So, with Christmas (Solstice / Hanukkah / Kwanzaa) and the New Year rapidly approaching, let’s take a moment, gather together, and do a little checking in, shall we?

 

 

Where We Left Off the Last Time

 

The two weeks just before I took this impromptu (and unscheduled) little break from blog-writing, we were looking at a novel system constructed of ultra-fine silver mesh. The researchers who developed this mesh network noticed that it had some very unusual properties. The words that they used were straight out of the statistical mechanics vocabulary, and included phrases such as “power law” and “phase transition.” Even the term “criticality” popped up.

You can read the original article, and my previous two blogposts, using these links:

• von Bubnoff, A brain built from atomic switches [that] can learn, and a follow-on article (same material), Brain Built on Switches,
• Third Stage Boost: Statistical Mechanics and Neuromorphic Computing, and
• Third Stage Boost – Part 2: Implications of Neuromorphic Computing

Quoting from the previous blogpost:

This mesh is a “network of microscopically thin intersecting silver wires,” grown via a combination of electrochemical and chemical processes … resembling “a highly interconnected plate of noodles.” They are testing a device built around a 2mm x 2mm mesh of this material.

Again, picking up from the last post:

The device exhibits self-organized criticality.

As described by graduate student Audrius Avizienis,

“That was really jaw-dropping; [it was] the first [time] we pulled a power law out of this.”

 

 

What This Means for Us (In Real, Practical Terms)

 

I’ve now taught a deep learning course three times in Northwestern University’s Master of Science in Predictive Analytics program over this last year. (This program will soon become the Master of Science in Data Science program, so use this new link, and I’ll continue teaching Text Analytics and the new elective, AI and Deep Learning – which will be substantially the same as what I’ve taught as the Special Topics 490 in Deep Learning.)

Typically, many of students get further and deeper into the deep learning material than I do. They manage to watch more tutorial vids, read more papers and blogs, and download and experiment with more software than I can manage.

Yet, over this past year, I’ve realized three things:

1. There are really two kinds of deep learning; one is a fairly straightforward layering of different layers: varieties in the themes of architectures and procedural details, but more-or-less following a simple gradient-descent learning method; and the other kind of deep learning involves energy-based models; similar architectures with a different philosophy to the learning method,
2. Deep learning has just about reached its zenith; all the algorithms are essentially those created some 30 years ago, and
3. There’s going to be a next stage, and it will be as radically different from our current neural architectures as deep learning (and all other neural network approaches) differ from the earlier symbolic AI methods.

It’s that last point that deserves our attention.

My gut feeling is that this “something different” will draw largely from the realm of statistical mechanics, maybe with a bit of quantum mechanics (or quantum computing) thrown in.

This means that it behooves each of us to start studying these topics now, since they take a while to learn.

Now, I’m not at all suggesting that each of you pick up a book on statistical mechanics and try reading through it over the Christmas holidays. I’ve tried it, and even though my Ph.D. was in statistical thermodynamics, as I look back on these books – they’re not casual reading. They’re not at all good for self-study.

So, that puts us into a bit of a dilemma, doesn’t it?

Here’s what I suggest: for the next little while, focus on learning the vocabulary of statistical mechanics and (even) quantum mechanics. I’m sort of suggesting “stat-mech-light.”

This means, you could come across the term phase transition, and instead of it being uncomfortably weird, you’d have a frame of reference. You’d know that all sorts of things undergo phase transitions; water in its liquid form (one phase) can become ice (that’s one kind of phase transition), or it can become steam (a different phase transition). I’m suggesting a sort of Scientific American kind of literacy about these topics. Not a follow-the-equations kind of familiarity, but a follow-the-conversation sort of literacy.

That’s a reasonable goal for these next few months, isn’t it?

Now, while you’re doing that (and I know you; you are really good at following up on things once you set yourself to it; that’s why my students are so often ahead of me), I have a few personal goals that relate to you:

• Continue writing about statistical mechanics and how it plays with neural networks, artificial intelligence, and deep learning – so simply by following with these blogposts, you’ll naturally increase your stat-mech vocabulary and concept-building,
• Continue working on the book, Statistical Mechanics, Neural Networks, and Machine Learning, which will have the equations that we need (at least for introductory work), and
• Continue developing a new kind of statistical mechanics-based architecture I’m calling it a “computational engine”, which I believe can be really useful. And since it’s in its infancy, I don’t know how useful it will be yet; it will either be immensely useful or a total bust; too early to tell.

So, you’ll hear from me again. Most likely next Thursday.

However, if I don’t shift focus now, and go write some Christmas cards, there are a whole lot of people who will not be hearing from me again, and this year, I’m really resolved to be more in touch.

So … until Solstice, which is next week at this time, and is the turning point of the year –

 

 

Live free or die, my friend –

AJ Maren

Live free or die: Death is not the worst of evils.
Attr. to Gen. John Stark, American Revolutionary War

 

 

P.S. Those personal stories? Well yes, I’ll write them. But next week, ok? Till then!

References – Neuromorphic Computing

 

• von Bubnoff, A. Artificial synapses could lead to brainier super-efficient computers, Wired (Oct, 4, 2017). online article.
• von Bubnoff, A. A brain built from atomic switches can learn, Quanta (Sept. 20, 2017). online article.
• Gomes, L. Neuromorphic Chips Are Destined for Deep Learning—or Obscurity, IEEE Spectrum (29 May 2017), online article.
• Li, W.Y., Ovchinnikov, I.V., Chen, H.L., Wang, Z., Lee, A., Lee, H.C., Cepeda, C., Schwartz, R.N., Meier, K., and Wang, K.L., A neuronal dynamics study on a neuromorphic chip, arXiv 1703:03560 (2017). pdf.

 

 

Some Useful Background Reading on Statistical Mechanics

 

• Hermann, C. Statistical Physics – Including Applications to Condensed Matter, in Course Materials for Chemistry 480B – Physical Chemistry (New York: Springer Science+Business Media), 2005. pdf. Very well-written, however, for someone who is NOT a physicist or physical chemist, the approach may be too obscure.
• Maren, A.J. Statistical Thermodynamics: Basic Theory and Equations, THM TR2013-001(ajm) (Dec., 2013) Statistical Thermodynamics: Basic Theory and Equations.
• Salzman, R. Notes on Statistical Thermodynamics – Partition Functions, in Course Materials for Chemistry 480B – Physical Chemistry, 2004. Statistical Mechanics (chapter). Online book chapter. This is one of the best online resources for statistical mechanics; I’ve found it to be very useful and lucid.
• Tong, D. Chapter 1: Fundamentals of Statistical Mechanics, in Lectures on Statistical Physics (University of Cambridge Part II Mathematical Tripos), Preprint (2011). pdf.

 

 

Previous Related Posts

• Third Stage Boost – Part 2: Implications of Neuromorphic Computing
• Third Stage Boost: Statistical Mechanics and Neuromorphic Computing
• 2025 and Beyond
• Deep Learning: The Fast Evolution of Artificial Intelligence
• Neg-Log-Sum-Exponent-Neg-Energy – That’s the Easy Part!
• Seven Essential Machine Learning Equations: A Cribsheet (Really, the Précis)
• Statistical Mechanics of Machine Learning blogpost – the great “St. Crispin’s Day” introduction of statistical mechanics and machine learning.

 

 

Return to:

• Dr. A.J.’s Home Page
• Blog Page