End-to-end implementation of a convolutional neural network on a 3D-integrated image sensor with macropixel array

Maria Lepecq (1) , Thomas Dalgaty (1) , William Fabre (1) , Stéphane Chevobbe (1)

1 LIAE - Laboratoire Intelligence Artificielle Embarquée

3D-integrated focal-plane array image processor chips offer new opportunities to implement computer vision algorithms directly inside the sensor. Neural networks in particular can perform highly complex machine vision tasks, and therefore their efficient implementation in such imagers are of significant interest. However, studies with existing pixel-processor array chips have focused on the implementation of a subset of neural network components - notably convolutional kernels. In this work, we implement a continuous end-to-end pipeline for a convolutional neural network from the digitisation of incoming photons to the output prediction vector on a macropixel-processor (where a single processor acts on set of pixels) array chip. Our implementation performs inference between 265 and 310 frames per second directly inside of the sensor by exploiting the different levels of parallelism available.

Domains: Engineering Sciences [physics]; Electronics; Computer Science [cs]; Artificial Intelligence [cs.AI]; Engineering Sciences [physics]; Micro and nanotechnologies/Microelectronics

Maria Lepecq, Thomas Dalgaty, William Fabre, Stéphane Chevobbe. End-to-end implementation of a convolutional neural network on a 3D-integrated image sensor with macropixel array. Sensors, 2023, 23 (4), pp.1909. ⟨10.3390/s23041909⟩. ⟨cea-03983243⟩