We address the problem of network quantization, that is, reducing bit-widths of weights and/or activations to lighten network architectures. Quantization methods use a rounding function to map full-precision values to the nearest quantized ones, but this operation is not differentiable. There are mainly two approaches to training quantized networks with gradient-based optimizers. First, a straight-through estimator (STE) replaces the zero derivative of the rounding with that of an identity function, which causes a gradient mismatch problem. Second, soft quantizers approximate the rounding with continuous functions at training time, and exploit the rounding for quantization at test time. This alleviates the gradient mismatch, but causes a quantizer gap problem. We alleviate both problems in a unified framework. To this end, we introduce a novel quantizer, dubbed a distance-aware quantizer (DAQ), that mainly consists of a distance-aware soft rounding (DASR) and a temperature controller. To alleviate the gradient mismatch problem, DASR approximates the discrete rounding with the kernel soft argmax, which is based on our insight that the quantization can be formulated as a distance-based assignment problem between full-precision values and quantized ones. The controller adjusts the temperature parameter in DASR adaptively according to the input, addressing the quantizer gap problem. Experimental results on standard benchmarks show that DAQ outperforms the state of the art significantly for various bit-widths without bells and whistles.
|Title of host publication||Proceedings - 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021|
|Publisher||Institute of Electrical and Electronics Engineers Inc.|
|Number of pages||10|
|Publication status||Published - 2021|
|Event||18th IEEE/CVF International Conference on Computer Vision, ICCV 2021 - Virtual, Online, Canada|
Duration: 2021 Oct 11 → 2021 Oct 17
|Name||Proceedings of the IEEE International Conference on Computer Vision|
|Conference||18th IEEE/CVF International Conference on Computer Vision, ICCV 2021|
|Period||21/10/11 → 21/10/17|
Bibliographical noteFunding Information:
We have shown that network quantization can be formulated as an assignment problem between full-precision and quantized values, and introduced a novel quantizer, dubbed DAQ, that addresses both the gradient mismatch and quantizer gap problems in a unified framework. Specifically, DASR approximates a rounding function with a kernel soft argmax operator, together with a temperature controller adjusting the temperature parameter adaptively. We have shown that DAQ achieves the state of the art for various network architectures and bit-widths without bells and whistles. We have also verified the effectiveness of each component of DAQ with a detailed analysis. Acknowledgments. This research was supported by the Samsung Research Funding & Incubation Center for Future Technology (SRFC-IT1802-06).
© 2021 IEEE
All Science Journal Classification (ASJC) codes
- Computer Vision and Pattern Recognition