The Conundrum – and How to Solve It:

We left off last week with a bit of a cliff-hanger; a puzzle with the 2-D CVM. (CVM stands for Cluster Variation Method; it’s a more complex form of a free energy equation that I discussed two weeks ago in this blogpost on The Big, Bad, Scary Free Energy Equation (and New Experimental Results); while not entirely unknown, it’s still not very common yet.)

We asked ourselves: which of the two grids (shown below) would be at equilibrium, and why?

Since I suggested that our first-glance answer was probably the wrong one, we’ve had a week in which to mull this over.

Here are the two grid configurations again:

We call the pattern on the left a “scale-free-like” system, and the one on the right an “extreme rich-club-like” system.

A scale-free system is one which looks the same at any level of resolution; fractal patterns are scale-free. In the case of this Left-Hand-side (LHS) grid, there’s an effort to create a couple of instances of a large (paisley-like) pattern, then several instances of the same pattern at a smaller scale, and then more instances at an even smaller scale.

In contrast, within a “rich club,” similar types of nodes are very densely connected to each other. This is based on the notion that in a rich club, all the members see each other all the time (at the yacht club, at the board meetings, at the benefits and galas), and that outside of their “club,” they don’t “see” anyone else.

Both the notions of “scale free” and “rich club” are known descriptions of different kinds of experimentally-observed neural activation configurations; I have lots of references in my 2016 paper.

The LHS pattern clearly has a greater variety of local patterns; the one on the Right-Hand Side (RHS) is much more constrained. So, our first guess would be: the one on the left is at equilibrium, and the one on the right isn’t.

But … if I’m saying that our first guess is likely wrong, then that means that the reverse is true. So what’s going on?

The grid on the right looks kind of like kids at a formal dance, with all the boys on one side and all the girls on the other. (OK, the “boys” are on the two sides – left and right, and the “girls” are in the center, but you get the idea.)

How can this possibly be an equilibrium situation?

The answer rests with our h-value, which is our interaction parameter.

When we think about it from this angle; the RHS grid configuration makes a lot of sense. It simply has a relatively high h-value, so we get a lot of “like-with-like” neighbor pairing.

This kind of thing happens in nature; here’s an example of a ferromagnetic system with an increasing applied magnetic field:

So, this makes sense. The way in which we get the RHS figure (b, the “extreme rich-club-like” example) is to have a relatively high interaction enthalpy parameter, or h-value. That’s a lot like increasing the magnetization in a ferromagnetic system. (And for fun, have a look at the Wiki page for magnetic domains; see the lovely “moving domain walls in a grain of silicon steel” about halfway through the Wiki, RHS. Fun!)

So … back to our conundrum. We can see how the RHS works out as possibly being an equilibrium situation.

But why not the LHS? What’s wrong there?

To understand why the LHS is NOT at equilibrium, we have to map the z(i) values (the triplet configurations) to the equilibrium curves for the z(i) versus h.

 
.

Counting the z’s

 

There’s a pretty neat little trick to figuring out whether either the LHS (“scale-free-like”) or the RHS (“extreme rich-club-like”) configurations are at equilibrium. This trick rests on the fact that when the distribution of on / off nodes is equal (when x1 = x2 = 0.5), we have an analytic solution for the free energy minimum, which is the equilibrium value for the system. This analytic solution ONLY works for the case of equiprobable distribution among the two states. However, both the LHS and RHS grid configurations were carefully designed so that X1 = X2 = 128, out of a 256-node system. This means that x1 = X1/256 = 0.5, and x2 = X2/256 = 0.5, and there is an equilibrium solution that is already figured out.

In fact, we don’t need the equilibrium solution of the free energy itself; all that we need are the z(i) values in terms of the h-value, where h is the interaction enthalpy parameter. These z’s in terms of h are precisely what we solve for, when we do the analytic solution for the 2-D CVM.

The z(i) values are triplets; that is, they are values for zigzag triplet combinations such as A–B–A. (In the grid configuration diagrams, the A states are black, and the B are white. Since the configuration wraps around like an envelope, the bottom row is shown again on the top, in shades of dark and light grey, and the left-hand-edge is shown again on the right, again in light and dark grey.) The A–A–A triplets are z1, and the A–B–A triplets are z3. The B–B–B triplets are z6, and the B–A–B triplets are z4. The remaining two kinds of triplets; A–A–B and B–B–A (z2 and z5, respectively) can be counted two different ways; they look different when counted left-to-right (e.g., A–A–B) as from right-to-left (B–A–A), but they’re the same thing; therefore we say that there is a degeneracy factor of two for each of these.

Here’s an example of counting the z1 and z3 values (in the horizontal direction only) for the extreme rich-club-like case on the RHS.

Since we have our triplet values, simply by counting them (whether manually or in computer code), all that we need to do is to see where these triplets fall in the graphs of various z(i) versus h. For the extreme rich-club-like case that we’re discussing here, the triplet values fall in the area of h=1.65, as shown in the following figure. (We could find the more precise h-value by doing a set of interpolations; that’s more than we need for today’s illustration.)

The convergence of the various z(i) versus h-values tells us that we’ve got an internally-consistent system; all the z(i)’s indicate the same h-values. Further, we’re at the upper extreme for h; this is a very strong like-with-like enthalpy. As we can see from the original RHS figure, we’ve pushed the grid configuration about as far as it can go; there are relatively few like-with-unlike triplets, and there are particularly few like-with-unlike-with-(original)-like; that means, very few A–B–A and B–A–B triplets. (Any fewer, and we wouldn’t have any of these at all, and then we’d be pushing our entropy to take the logarithm of zero, and that would put the whole system into very much a difficult state.)

So we can agree. The RHS grid configuration is indeed at equilibrium, and we even know it’s h-value; that means that we’ve characterized the entire grid with a single parameter (for the x1 = x2 = 0.5 constraint).

Now, what can we say about the LHS – the “scale-free-like” system?

 
.

How We Can Tell That a System is NOT at Equilibrium

 

We’re going to play exactly the same kind of game, this time with the LHS grid configuration.

First, we count up the various z(i) values, as shown in the following figure.

We can see that there is a more equitable distribution of z(i) values in this grid, especially as compared the “extreme rich-club-like” pattern in the previous discussion.

As before, we identify our z(i) values in the graph of z(i)’s vs h; this is shown in the following figure.

Wow! When we look at the z(i)-vs- h graphs this time, we see that the indicated h-values are all over!

For example, z4 = 0.121, which is close to the value it would have if h = 0. (The values for z1, z3, z4, and z6 = 0.125 when h = 0. Total values for z2 = z5 = 0.250, due to the degeneracy mentioned earlier in this post.)

In contrast to the z4 value, z3 = 0.047, z1 = 0.172, and z6 = 0.10. These each correspond to a different h-value.

This means that the system, while we can physically draw it out and compute it’s z(i) values, is not physically realistic; it is not at equilibrium.

In order to bring it to equilibrium, we’d have to let it undergo a free energy minimization process; various nodes would have to turn on and off (keeping the ratio of nodes in A and B equal all the time). Every time we did a (pair of) flip(s), we’d have to recompute all the z(i) values, and then compute the free energy, and see if it was lower. If it were, we’d keep the flips, and then try again. When we couldn’t lower the free energy any more, we’d look at the z(i)-vs.-h values again; very likely, they’d all line up as they did for the “extreme rich-club-like” case that we discussed above.

 
.

What This Gets Us

 

First of all, new toy! (I just love new toys, don’t you?)

Actually having the means to figure out whether a grid configuration represents an equilibrium situation or not is a brand new thing. The code that I described in my V&V document counts the z(i) values, computes the thermodynamic values (such as free energy), and – if the system is not at a free energy minimum – will do (pairs of) node-flips to bring it closer to equilibrium.

That’s new, interesting, different, and fun all in itself. (And while the code is not available yet, it will be soon, and you’ll be able to play with it yourself.)

More than that, it gives us a new way of thinking about things.

We’ll discuss some implications in next week’s post. See you then!

 
.
 

Live free or die, my friend –

AJ Maren

Live free or die: Death is not the worst of evils.
Attr. to Gen. John Stark, American Revolutionary War

 

 

The Essential References

 

If you’re going to follow along, there are now three valuable papers – the older, more tutorial paper on the 2-D Cluster Variation Method, and (just published this January, 2018) the Code V&V. Also, the derivation for the z(i) values in terms of h (where the solution is at the free energy minimum, or equilibrium) is given in the middle (2014) paper; I’ll tweak it a bit and put it up on arXiv soon, you can see the pdf below for now:

• Maren, A.J. (2018) Free Energy Minimization Using the 2-D Cluster Variation Method: Initial Code Verification and Validation, THM TR2018-001(ajm), arXiv:1801.08113 [cs.NE] arXiv abstract.
• Maren, A.J. (July, 2014) The Cluster Variation Method II: 2-D Grid of Zigzag Chains: Basic Theory, Analytic Solution and Free Energy Variable Distributions at Midpoint (x1 = x2 = 0.5). THM TR2014-003(ajm). DOI: 10.13140/2.1.4112.5446
• Maren, A.J. (2016) The Cluster Variation Method: A Primer for Neuroscientists. Brain Sciences, 6(4), 44. doi:10.3390/brainsci6040044 pdf

 

 

Previous Related Posts

• 2-D Cluster Variation Method: Code Verification and Validation
• The Big, Bad, Scary Free Energy Equation (and New Experimental Results)
• A “First Principles” Approach to Artificial General Intelligence
• A Hidden Layer Guiding Principle: What We Minimally Need
• How Getting to a Free Energy Bottom Helps Us Get to the Top
• What’s Next for AI: Beyond Deep Learning