Patent – Biologically-Based Sensor Fusion

Sensor Fusion in the Superior Colliculus Helps Solve Navy Air Traffic Control Challenges

Sensor Fusion: Why Is It So Important?

Sensor fusion is one of the biggest challenges facing our military. (See the Militarily Critical Technologies List.)

The reason that sensor fusion is so important is that many existing systems are stovepiped; they are self-contained. In order for human operators to connect information from one system to another, they must manually switch between different systems, and remember what they observed in one system and mentally map it into another. This has great risk potential, especially in high-stress situations.

Air Traffic Controller 2nd Class Joey Holloway and Air Traffic Controller Airman Nicholas Colvin monitor the SPN-43 Precision Approach Radar . — Naval air traffic control: Air Traffic Controller 2nd Class Joey Holloway and Air Traffic Controller Airman Nicholas Colvin monitor the SPN-43 Precision Approach Radar.

One specific example of such a sensor fusion challenge is that faced in carrier-based air traffic control. Every Navy carrier has its own Carrier Air Traffic Control Center (CATCC), and the specialists working in the CATCC need to coordinate information from multiple on-board radars.

There are typically two different kinds of radar systems for each of two distance coverages (long and short-range). The Digital Airport Surveillance Radar (DASR) ASR-11 provides primary surveillance radar coverage up to 60 miles. For short range, the AN/SPN-43 performs marshalling – helping CATCC personnel to organize incoming and outgoing air traffic; and to guide pilots for their final approach to the carrier deck.

As an example of how different radars are used together for a given task, the AN/SPN-43C is used in conjunction with the AATC DAIR system for the marshalling and tactical deployment, and in conjunction with the carrier-controlled-approach radar for aircraft landing operations. In addition, this same system serves as a backup short-range, air-search radar system.

USS Abraham Lincoln (CVN 72) is the fifth vessel of the Nimitz-Class nuclear-powered aircraft carriers.

Every carrier and destroyer is outfitted with a unique set of radar systems. For example, the Nimitz-class carrier USS Abraham Lincoln (CVN 72) is equipped with “AN/SPS-48E 3-D and AN/SPS-49(V)5 2-D air search radar, a AN/SPQ-9B target acquisition radar, AN/SPN-46 and AN/SPN-43C air traffic control radars, AN/SPN-41 landing aid radar, four Mk 91 NSSM guidance systems and four Mk 95 radars.”

What Can Cats Teach Us About Sensor Fusion?

Have you ever watched a cat prowling to hunt? She slinks to the area where a little critter might be found, and walks ever so softly. She hears a rustle in the leaves – but may ignore it. She sees a twitch in the grass, and may ignore this also.

A cat's biologically-based sensor fusion system causes it to pay attention to a target - the mouse! — Cat hunting mouse: a cat’s biologically-based sensor fusion system causes it to pay attention to a target – the mouse!

She hears a rustle and sees a twitch – in the same area – and she can’t ignore the two stimuli, coming from the same place at the same time.

Really, she has no choice. She can’t say, “Oh, I’m not hunting today.”

No, her brain-based target guidance and control system takes over.

The Superior Colliculus: The Brain’s Sensor Fusion Center

There is a specific area of your cat’s brain (and in any mammalian brain, including yours and mine) that is charged with performing biologically-based sensor fusion.

Let’s look at what happens in your cat’s behavior when the right stimuli occur – stimuli that occur in the same place, at the same time.

Her head turns to center on the target,
Her ears move – directional hearing – to get the best sound from the target area, and
Her eyes foveate – they center on the suspected target area so that they get the most visual information from the target stimulus.

All of this happens even before your cat has a chance to think about it. She’s not wondering, Am I interested in a mouse today?

The superior colliculus performs sensor fusion in all mammalian brains. (Photo credit: DrCink.net.)

No. A particular part of her brain – the superior colliculus – collects stimulus from her visual and auditory fields (and even from her skin) – organizing these stimuli in topographically-mapped layers. When two stimuli occur – from different sensor systems (visual and auditory, for example) – but from the same place in the environment, and at the same time – the superior colliculus automatically fuses the information.

Without asking the rest of your cat’s brain if she wants to hunt, the fused sensor stimulus in the cat’s superior colliculus acts like an advanced target acquisition and guidance system – it takes over.

Once it takes over, your cat has a chance to think through what she’s seeing and hearing – but only after her superior colliculus has directed her to attend.

The superior colliculus is an exceptionally effective biologically-based sensor fusion system.

First Breakthroughs: Understanding Sensor Fusion in the Cat Superior Colliculus

Merging of the Senses, by Barry E. Stein and M. Alex Meredith (Cambridge, MA: MIT Press, Cognitive Neuroscience Series, 1993). — *Merging of the Senses*, by Barry E. Stein and M. Alex Meredith (Cambridge, MA: MIT Press, Cognitive Neuroscience Series, 1993).

Researchers Barry Stein and M. Alex Meredith, at the Bowman Grey School of Medicine at Wake Forrest University, were the first to carefully elucidate the specific mechanisms of biological sensor fusion in the superior colliculus. (See their book, The Merging of the Senses, in Further Readings, below.)

Biologically-Based Sensor Fusion Inspires Award-Winning Patent

Biologically-based sensor fusion architecture.

Working with Dr. Stein as the Institutional Investigator, Dr. Alianna Maren, Principal Investigator for the ONR-sponsored contract on Signal Processing Chips/Electronics (Phase II STTR Contract N00014-95-C-0323) was able to create a computer-based algorithm for correlating and fusing point source targets. This architecture included several key features inspired by the brain.

This ONR-sponsored work was later patented (US Patent 5850625 A; Sensor Fusion Apparatus and Method), and received the New York Times Patent of the Week award in 1999.

Blog Posts on Biological Sensor Fusion

Biologically-based multisensory fusion for Brain-Computer Interfaces (BCIs)

Alianna J. Maren

Statistical Mechanics, Neural Networks, Artificial Intelligence