Multisensor Fusion for Brain-Computer Interfaces (BCIs)

More than 25 years ago, sensor fusion was identified as a militarily critical technology. (See blog post describing role of sensor fusion for Navy air traffic control.)

Since that time, both our knowledge of – and the importance of – sensor fusion has grown substantially.

Groundbreaking work by Barry Stein and M. Alex Meredith, at the Bowman Grey School of Medicine at Wake Forrest University, elucidated the specific mechanisms of biological sensor fusion in the superior colliculus. (See their book, The Merging of the Senses (1993), in Further Readings, below.)

A year later, this work on biologically-based sensor fusion was inspiration for a computer-based sensor fusion algorithm, designed to address the challenge of fusing radar target information for air traffic control.

Working with Dr. Stein as the Institutional Investigator, I was Principal Investigator for 1994/95 ONR-sponsored contracts on Signal Processing Chips/Electronics (Phase II STTR Contract N00014-95-C-0323), and was able to create a computer-based algorithm for correlating and fusing point source targets. This architecture included several key features inspired by the brain.

This ONR-sponsored work was later patented (US Patent 5850625 A; Sensor Fusion Apparatus and Method), and received the New York Times Patent of the Week award in 1999.

Now, twenty years after that initial work, biologically-based sensor fusion is more important than ever.

Biologically-based Sensor Fusion and Brain-Computer Interfaces

Brain-Computer Interfaces (BCIs) are becoming the most exciting technological breakthrough of this decade, and will substantially impact human experience over decades to come.

Even though our early, very-first-gen BCIs interact with only one area of the brain, we can see that sensor fusion will have a strong role to play – both in understanding how the brain itself will interact with externally (computer-driven) stimulus and how various devices can interact with multiple brain modalities.

Already, sensor fusion is recognized as essential in forming effective input modalities from brain-based measurements (EEG, etc.) to device controls. Pires et al. report that Bayesian-based sensor fusion of multiple EEG channels gives wheelchair users accurate EEG-based directional control, noting that “The 4 best channels (CPz, P3, PO7 and PO8) were used for Bayesian fusion… Fusion improves both the FP and FN rates … After 5 epochs average, the FP rate is about 1%.”

Suminski, Tkach, & Hatsopoulos (2009) show using mutual information and directional tuning analyses that the presence of multi-sensory feedback (i.e. vision and proprioception) during replay of movements evokes neural responses in MI that are almost indistinguishable from those responses measured during overt movement.

Clearly, multiple stimulus modes drawn from the brain itself will be more effective than single stimulus modes in providing not only distinguishable input, but also graceful degradation, in the operation of brain-controlled devices.

On the other hand, introducing stimulus directly into the brain will impact not only specific sensory and motor-cortex areas, but also the various integration pathways.

Recent Work in Biologically-based Sensor Fusion

Barry Stein, editor of The New Handbook of Multisensory Processing (MIT Press, 2012), has assembled a comprehensive set of chapters discussing the full range of multisensory fusion in the brain, ranging from individual neurons to neural systems, and including behavioral, perceptual, and cognitive correlates. This book goes far beyond the original 1993 work, showing how substantially the field has advanced in twenty years.

Nevertheless, as we seek to interact directly with the brain, what we know now is only a fraction of what will be needed as we move towards the next BCI generation.

Further Readings – Biological Sensor Fusion

• Calvert, G.A., Spence, C., & Stein, B.E. (Eds) (2004). The Handbook of Multisensory Processes (Cambridge, MA: MIT Press – Bradford Books) Amazon book link
• Maren, A.J., Akita, R.M., Colbert, B.D., Donovan, D.J., Glover, C.W., Mathia, K., Pap, R.M., Priddy, K.L., Robinson, T.W., & Saeks, R.E. (1998). U.S. Patent 5850625 A; Sensor Fusion Apparatus and Method. online
• Marino, R.A., Rodgers, C.K., Levy,R., & Munoz, D.P. (2008). Spatial relationships of visuomotor transformations in the superior colliculus map. J. Neurophys. 100 (5), 2564-2576 DOI: 10.1152/jn.90688.2008. online
• Robert-Ribes, J., Schwartz, J-L., & Escudier, P. (2005). A comparison of models for fusion of the auditory and visual sensors in speech perception. Artificial Intelligence Review 9 (4-5), 323-346. abstract and image of first two pages
• Stein, B.E., and Rowland, B.A. (2011). Organization and plasticity in multisensory integration: early and late experience affects its governing principles. Prog Brain Res. 191: 145–163. doi: 10.1016/B978-0-444-53752-2.00007-2. online
• Stein, B.E. (Ed.) (2012). The New Handbook of Multisensory Processing (Cambridge, MA: MIT – Bradford Press). Amazon book link
• Stein, B.E., & Meredith, M.A. (1993). The Merging of the Senses. (Cambridge, MA: MIT Press – Cognitive Neuroscience Series). Amazon book link

For a collection of good references, up through 2012, see: Multisensory-integration Wiki.

Sensor Fusion in Brain-Computer Interfaces

• Pires, G., Castelo-Branco, M., and Nunes, U. (2008). Visual P300-based BCI to steer a wheelchair: a Bayesian approach. Conf Proc IEEE Eng Med Biol Soc. 2008:658-61. doi: 10.1109/IEMBS.2008.4649238.pdf
• Suminski, A.J., Tkach, D.C., & Hatsopoulos, N.G. (2009). Exploiting multiple sensory modalities in brain-machine interfaces. Neural Networks 22, 1224- 1234. pdf

New readings, as acquired, will be added to: Biological Sensor Fusion and Brain-Computer Interfaces.