Brain-Inspired Complementary Vision Chip With Primitive-Based Representations

Introduction

Intelligent visual perception is leading a technological revolution. Traditional image sensors are limited by bandwidth and power consumption, with their core metrics mutually constraining each other, making it difficult to meet the demands of open environments. Drawing on human visual mechanisms, the team proposed an innovative complementary sensing paradigm comprising a primitive-based representation and developed the "Tianmouc," providing a new solution for efficient and robust visual perception.

A New Brain-Inspired Paradigm of Primitive-based Complementary Visual Perception

Traditional image sensors are constrained by power and bandwidth walls, making it difficult to simultaneously achieve high resolution, high frame rates, and high dynamic range, which cannot meet the demands of complex open environments.

The team drew inspiration from the human visual system and proposed a new brain-inspired paradigm of primitive-based complementary visual perception, developing the "Tianmouc". The team has achieved significant breakthroughs in theory, chip, algorithm, software, and system applications, with the results published as a cover article in Nature.

This paradigm disassembles visual information into primitive-based representations, forming two complementary, information-complete visual perception pathways of "cognition-oriented" and "action-oriented", thereby overcoming performance bottlenecks of traditional image sensors. Based on this paradigm, the Tianmouc was developed to achieve high-speed, high-precision, and high dynamic range perception under low bandwiath and power consumption, effectively tackling with extreme scenarios.

The team established a complete ecosystem including software, algorithms, data, and simulation platforms, realizing a novel perception-decision paradigm. The team constructed an automotive perception system that demonstrated low-latency, high-performance real-time perception reasoning on open roads, demonstrating enormous potential in the field of unmanned systems.

The team has applied for 68 patents, including international patents, and has received authorization for 37 of them.

Facing the Open World, Tianmouc Provides Efficient and Robust Solutions

Inspired by the human visual system, the team has developed a new paradigm of brain-inspired complementary perception based on primitive-based data representation, which decomposes visual information into different fundamental primitives and organically combines them to form complementary dual pathways. Based on this paradigm, they designed and implemented "Tianmouc". Through unique architectural designs, such as a hybrid pixel array, it achieves this paradigm, capable of capturing color information at high resolution while also sensing the sparse spatiotemporal difference with low latency. Tianmouc enables efficient complementary visual perception, effectively addressing the traditional challenges of capturing high-frame-rate, high-precision images over extended periods.

This breakthrough opens up new fields for brain-inspired perception, and can be extended to the development of sensors for auditory, tactile, olfactory, and other sensors, providing new perspectives for understanding human perception. It injects new vitality into the field of machine vision, promotes the development of key applications such as consumer imaging devices, industrial automation, autonomous driving, and intelligent robotics, and demonstrates enormous market potential.

Published as the Cover Paper of the Nature Journal, Receiving Praise from the Global Academic Community

The paper "A vision chip with complementary pathways for open-world sensing" relative to Tianmouc was published as a cover article in the Nature journal on May 30, 2024, attracting widespread attention from the academic community and mainstream media.

So far, there have been only about twenty cover articles published in Nature finished by academic institutions in mainland China as the primary institution. This marks a high recognition of this work by the academic community. One of the paper's reviewers, Craig Vineyard, head of the brain-inspired team at the Sandia National Laboratories in the United States, believes that resulting Tianmouc chip is an exciting and novel hybrid approach".

Mainstream media have also extensively reported on Tianmouc, including over 40 authoritative news outlets such as Xinhua News Agency, CCTV, People's Daily, Guangming Daily. According to Altmetric, an authoritative paper impact statistics organization, the impact of the paper ranks in the top 5%, while its attention is in the top 1%.

The team was also awarded the 2024 "Intel China Academic Achievement Award-Outstanding Research" by the Intel Technical Committee for their achievements in the Tianjic and Tianmouc, further demonstrating their outstanding contributions and influence in the field of brain-inspired intelligence.