Skip to Main content Skip to Navigation
Conference papers

Disentangled Loss for Low-Bit Quantization-Aware Training

Thibault Allenet 1 David Briand 1 Olivier Bichler 1 Olivier Sentieys 2 
1 LIAE - Laboratoire Intelligence Artificielle Embarquée
Université Paris-Saclay, DSCIN - Département Systèmes et Circuits Intégrés Numériques : DRT/LIST/DSCIN
2 TARAN - Architectures matérielles spécialisées pour l’ère post loi-de-Moore
Inria Rennes – Bretagne Atlantique , IRISA-D3 - ARCHITECTURE
Abstract : Quantization-Aware Training (QAT) has recently showed a lot of potential for low-bit settings in the context of image classification. Approaches based on QAT are using the Cross Entropy Loss function which is the reference loss function in this domain. We investigate quantization-aware training with disentangled loss functions. We qualify a loss to disentangle as it encourages the network output space to be easily discriminated with linear functions. We introduce a new method, Disentangled Loss Quantization Aware Training, as our tool to empirically demonstrate that the quantization procedure benefits from those loss functions. Results show that the proposed method substantially reduces the loss in top-1 accuracy for low-bit quantization on CIFAR10, CIFAR100 and ImageNet. Our best result brings the top-1 Accuracy of a Resnet-18 from 63.1\% to 64.0\% with binary weights and 2-bit activations when trained on ImageNet.
Document type :
Conference papers
Complete list of metadata

https://hal-cea.archives-ouvertes.fr/cea-03776535
Contributor : Contributeur MAP CEA Connect in order to contact the contributor
Submitted on : Tuesday, September 13, 2022 - 7:00:38 PM
Last modification on : Wednesday, September 21, 2022 - 3:51:32 AM

File

93.pdf
Files produced by the author(s)

Identifiers

Citation

Thibault Allenet, David Briand, Olivier Bichler, Olivier Sentieys. Disentangled Loss for Low-Bit Quantization-Aware Training. CVPR 2022 - IEEE / CVF Computer Vision and Pattern Recognition Conference, Jun 2022, New Orleans, United States. pp.2788-2792, ⟨10.1109/CVPRW56347.2022.00315⟩. ⟨cea-03776535⟩

Share

Metrics

Record views

0

Files downloads

0