An adaptive outlier correction quantization method for vision Transformers

Abstract

Transformers have demonstrated considerable success across various domains but are constrained by their significant computational and memory requirements. This poses challenges for deployment on resource-constrained devices. Quantization, as an effective model compression method, can significantly reduce the operational time of Transformers on edge devices. Notably, Transformers display more substantial outliers than convolutional neural networks, leading to uneven feature distribution among different channels and tokens. To address this issue, we propose an adaptive outlier correction quantization (AOCQ) method for Transformers, which significantly alleviates the adverse effects of these outliers. AOCQ adjusts the notable discrepancies in channels and tokens across three levels: operator level, framework level, and loss level. We introduce a new operator that equivalently balances the activations across different channels and insert an extra stage to optimize the activation quantization step on the framework level. Additionally, we transfer the imbalanced activations across tokens and channels to the optimization of model weights on the loss level. Based on the theoretical study, our method can reduce the quantization error. The effectiveness of the proposed method is verified on various benchmark models and tasks. Surprisingly, DeiT-Base with 8-bit post-training quantization (PTQ) can achieve 81.57% accuracy with a 0.28 percentage point drop while enjoying 4× faster runtime. Furthermore, the weights of Swin and DeiT on several tasks, including classification and object detection, can be post-quantized to ultra-low 4 bits, with a minimal accuracy loss of 2%, while requiring nearly 8× less memory.

1. Introduction

Transformer-based architectures (Vaswani et al., 2017) have shown great power in natural language processing (NLP) tasks (Choi et al., 2018; Devlin et al., 2019). Increasingly, vision Transformers (ViTs) have also achieved competitive performance on many computer vision (CV) tasks including image classification, object detection, object segmentation, and other vision tasks recently (Carion et al., 2020; Chen ZS et al., 2021; Dosovitskiy et al., 2021; Graham et al., 2021; Yuan L et al., 2021; Touvron et al., 2021a; Dong et al., 2022; Liu Z et al., 2022b; Yu et al., 2022). Transformers consist of a number of blocks containing multi-head self-attention (MHSA) and feed-forward networks (FFNs), which enables the extraction of highly discriminative features. However, these Transformer-based models are notable for their substantial computational intensity and extensive memory requirements, posing significant challenges for deployment on resource-constrained devices (Alam et al., 2023; Chitty-Venkata et al., 2023). Consequently, there is an urgent industry requirement to compress and accelerate these Transformer-based models to facilitate broader application.

Much effort has been invested in facilitating the deployment of Transformers, including pruning (Han S et al., 2015; Zheng et al., 2022), distillation (Touvron et al., 2021b), quantization (Yao et al., 2022), and the direct design of more efficient Transformers (Choromanski et al., 2021; Yang et al., 2022). Among these methods, quantization employs low-bit precision for weight and activation values without altering the model architecture, making it particularly suitable for carefully designed efficient Transformers. There are primarily two types of quantization methods: post-training quantization (PTQ) and quantization-aware training (QAT) (Chitty-Venkata et al., 2023). Unlike the QAT method, which necessitates the entire training dataset, PTQ only requires unlabeled calibration images, thereby enabling rapid quantization and deployment. Consequently, our focus is on PTQ methods to compress Transformers. Besides, quantization and other techniques (Han S et al., 2015; Touvron et al., 2021b; Yang et al., 2022) are complementary for model acceleration.

Previous studies, such as ranking-aware (Liu ZH et al., 2021), adopt a ranking loss to make the order of the self-attention results after quantization as consistent as possible. Fully quantized vision Transformer (FQ-ViT) (Lin et al., 2022) observes serious inter-channel variation in LayerNorm inputs and extreme non-uniform distributions in attention maps, and thus uses power-of-two factor (PTF) and log-int-softmax (LIS) to reduce the performance degradation. Scale reparameterization for post-training quantization of vision Transformers (RepQ-ViT) (Li ZK et al., 2023) uses channel-wise quantization to deal with the imbalance between channels and uses a log√2 quantizer to compress the power-law features of the latter.