
## Quantization
| Title & Authors | Introduction | Links |
|:--|  :----: | :---:|
|[![Star](https://img.shields.io/github/stars/IST-DASLab/gptq.svg?style=social&label=Star)](https://github.com/IST-DASLab/gptq)[![Publish](https://img.shields.io/badge/Conference-ICLR'22-blue)]()<br>[GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://arxiv.org/abs/2210.17323) <br> Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh |<img width="202" alt="image" src="figures/GPTQ.png"> |[Github](https://github.com/IST-DASLab/gptq) <br> [Paper](https://arxiv.org/abs/2210.17323)|o
|[![Star](https://img.shields.io/github/stars/mit-han-lab/smoothquant.svg?style=social&label=Star)](https://github.com/mit-han-lab/smoothquant)[![Publish](https://img.shields.io/badge/Conference-ICML'23-blue)]() <br>[SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://arxiv.org/abs/2211.10438) <br> Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, Song Han |<img width="1002" alt="image" src="https://github.com/mit-han-lab/smoothquant/blob/main/figures/intuition.png"> |[Github](https://github.com/mit-han-lab/smoothquant) <br> [Paper](https://arxiv.org/abs/2211.10438)|
| [![Star](https://img.shields.io/github/stars/artidoro/qlora.svg?style=social&label=Star)](https://github.com/artidoro/qlora) [![Publish](https://img.shields.io/badge/Conference-NeurIPS'23-blue)]() <br>[QLoRA: Efficient Finetuning of Quantized LLMs](https://arxiv.org/abs/2305.14314) <br> Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer | ![](figures/qlora.png) | <br>[Github](https://github.com/artidoro/qlora)</br> [Paper](https://arxiv.org/abs/2305.14314) |
|[![Star](https://img.shields.io/github/stars/jerry-chee/QuIP.svg?style=social&label=Star)](https://github.com/jerry-chee/QuIP) [![Publish](https://img.shields.io/badge/Conference-NeurIPS'23-blue)]() <br>[QuIP: 2-Bit Quantization of Large Language Models With Guarantees](https://arxiv.org/abs/2307.13304) <br> Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, Christopher De SaXQ |<img width="302" alt="image" src="figures/QuIP.png"> |[Github](https://github.com/jerry-chee/QuIP) <br> [Paper](https://arxiv.org/abs/2307.13304)|
|[![Publish](https://img.shields.io/badge/Conference-NeurIPS'23-blue)]()<br>[Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization](https://arxiv.org/abs/2305.14152) <br> Jeonghoon Kim, Jung Hyun Lee, Sungdong Kim, Joonsuk Park, Kang Min Yoo, Se Jung Kwon, Dongsoo Lee |<img width="1002" alt="image" src="figures/PEQA.png"> |[Paper](https://arxiv.org/abs/2305.14152)|
| [![Star](https://img.shields.io/github/stars/Qualcomm-AI-research/outlier-free-transformers.svg?style=social&label=Star)](https://github.com/Qualcomm-AI-research/outlier-free-transformers)  [![Publish](https://img.shields.io/badge/Conference-NeurIPS'23-blue)]() <br>[Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing](https://arxiv.org/abs/2306.12929) <br> Yelysei Bondarenko, Markus Nagel, Tijmen Blankevoort | ![](figures/QT.png) | [Github](https://github.com/Qualcomm-AI-research/outlier-free-transformers) [Paper](https://arxiv.org/abs/2306.12929) |
|[![Star](https://img.shields.io/github/stars/nbasyl/LLM-FP4.svg?style=social&label=Star)](https://github.com/nbasyl/LLM-FP4)[![Publish](https://img.shields.io/badge/Conference-EMNLP'23-blue)]()<br>[LLM-FP4: 4-Bit Floating-Point Quantized Transformers](https://arxiv.org/abs/2310.16836) <br> Shih-yang Liu, Zechun Liu, Xijie Huang, Pingcheng Dong, Kwang-Ting Cheng |<img width="1002" alt="image" src="figures/LLM-FP4.png"> |[Github](https://github.com/nbasyl/LLM-FP4) <br> [Paper](https://arxiv.org/abs/2310.16836)|
|[![Publish](https://img.shields.io/badge/Conference-EMNLP'23-blue)]()<br>[Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization](https://arxiv.org/abs/2311.05161) <br> Jangwhan Lee, Minsoo Kim, Seungcheol Baek, Seok Joong Hwang, Wonyong Sung, Jungwook Choi |<img width="1002" alt="image" src="figures/AQAS.png"> |[Paper](https://arxiv.org/abs/2311.05161)|
|[![Publish](https://img.shields.io/badge/Conference-AAAI'24-blue)]()<br>[Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edge](https://arxiv.org/abs/2312.05693) <br> Xuan Shen, Peiyan Dong, Lei Lu, Zhenglun Kong, Zhengang Li, Ming Lin, Chao Wu, Yanzhi Wang |<img width="302" alt="image" src="figures/agile.png"> |[Paper](https://arxiv.org/abs/2312.05693)|
|[![Star](https://img.shields.io/github/stars/OpenGVLab/OmniQuant.svg?style=social&label=Star)](https://github.com/OpenGVLab/OmniQuant)[![Publish](https://img.shields.io/badge/Conference-ICLR'24-blue)]()<br>[OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models](https://arxiv.org/abs/2308.13137) <br> Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, Ping Luo |<img width="1002" alt="image" src="figures/omniquant.png"> |[Github](https://github.com/OpenGVLab/OmniQuant) <br> [Paper](https://arxiv.org/abs/2308.13137)|
|[![Star](https://img.shields.io/github/stars/bytedance/AffineQuant.svg?style=social&label=Star)](https://github.com/bytedance/AffineQuant)[![Publish](https://img.shields.io/badge/Conference-ICLR'24-blue)]()<br>[AffineQuant: Affine Transformation Quantization for Large Language Models](https://arxiv.org/abs/2403.12544) <br> Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang, Shilei Wen, Fei Chao, Rongrong Ji |<img width="1002" alt="image" src="https://github.com/bytedance/AffineQuant/blob/main/fig/overview.png"> |[Github](https://github.com/bytedance/AffineQuant) <br> [Paper](https://arxiv.org/abs/2403.12544)|
|[![Publish](https://img.shields.io/badge/Conference-ICML'23%20ES%20FOMO-blue)]()<br>[GPT-Zip: Deep Compression of Finetuned Large Language Models](https://openreview.net/forum?id=hO0c2tG2xL) <br> Berivan Isik, Hermann Kumbong, Wanyi Ning, Xiaozhe Yao, Sanmi Koyejo, Ce Zhang |<img width="1002" alt="image" src="figures/GPT-Zip.png"> |[Paper](https://openreview.net/forum?id=hO0c2tG2xL)|
|[![Star](https://img.shields.io/github/stars/Twilight92z/Quantize-Watermark.svg?style=social&label=Star)](https://github.com/Twilight92z/Quantize-Watermark)[![Publish](https://img.shields.io/badge/Conference-EMNLP'23%20Findings-blue)]()<br>[Watermarking LLMs with Weight Quantization](https://arxiv.org/abs/2310.11237) <br> Linyang Li, Botian Jiang, Pengyu Wang, Ke Ren, Hang Yan, Xipeng Qiu |<img width="1002" alt="image" src="figures/watermark_quant.png"> |[Github](https://github.com/Twilight92z/Quantize-Watermark) <br> [Paper](https://arxiv.org/abs/2310.11237)|
|[![Star](https://img.shields.io/github/stars/mit-han-lab/llm-awq.svg?style=social&label=Star)](https://github.com/mit-han-lab/llm-awq) <br>[AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://arxiv.org/abs/2306.00978) <br> Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, Song Han |<img width="1002" alt="image" src="https://github.com/mit-han-lab/llm-awq/blob/main/figures/overview.png"> |[Github](https://github.com/mit-han-lab/llm-awq) <br> [Paper](https://arxiv.org/abs/2306.00978)|
| [![Star](https://img.shields.io/github/stars/hahnyuan/RPTQ4LLM.svg?style=social&label=Star)](https://github.com/hahnyuan/RPTQ4LLM) <br>[RPTQ: Reorder-based Post-training Quantization for Large Language Models](https://arxiv.org/abs/2304.01089) <br> Zhihang Yuan and Lin Niu and Jiawei Liu and Wenyu Liu and Xinggang Wang and Yuzhang Shang and Guangyu Sun and Qiang Wu and Jiaxiang Wu and Bingzhe Wu | ![](https://github.com/hahnyuan/RPTQ4LLM/blob/master/ims/cover.png) | <br>[Github](https://github.com/hahnyuan/RPTQ4LLM)</br> [Paper](https://arxiv.org/abs/2304.01089) |
|[ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation](https://arxiv.org/abs/2303.08302) <br> Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, Yuxiong He |<img width="402" alt="image" src="figures/zeroquant-v2.png"> |[Paper](https://arxiv.org/abs/2303.08302)|
| [![Star](https://img.shields.io/github/stars/SqueezeAILab/SqueezeLLM.svg?style=social&label=Star)](https://github.com/SqueezeAILab/SqueezeLLM) <br>[SqueezeLLM: Dense-and-Sparse Quantization](https://arxiv.org/pdf/2306.07629.pdf) <br>Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, Kurt Keutzer | <img width="1102" alt="image" src="figures/SqueezeLLM.png"> |[Github](https://github.com/SqueezeAILab/SqueezeLLM) <br> [Paper](https://arxiv.org/pdf/2306.07629.pdf)|
| [Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling](https://arxiv.org/abs/2304.09145v1) <br> Xiuying Wei , Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, Xianglong Liu|  <img width="1102" alt="image" src="figures/outliersuppression.png"> | [Paper](https://arxiv.org/abs/2304.09145v1)|
|[Integer or Floating Point? New Outlooks for Low-Bit Quantization on Large Language Models](https://arxiv.org/abs/2305.12356) <br> Yijia Zhang, Lingran Zhao, Shijie Cao, Wenqiang Wang, Ting Cao, Fan Yang, Mao Yang, Shanghang Zhang, Ningyi Xu |<img width="1002" alt="image" src="figures/MoFQ.png"> |[Paper](https://arxiv.org/abs/2305.12356)|
|[LLM-QAT: Data-Free Quantization Aware Training for Large Language Models](https://arxiv.org/abs/2305.17888) <br> Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, Vikas Chandra |<img width="1002" alt="image" src="figures/LLM-QAT.png"> |[Paper](https://arxiv.org/abs/2305.17888)|
|[![Star](https://img.shields.io/github/stars/Vahe1994/SpQR.svg?style=social&label=Star)](https://github.com/Vahe1994/SpQR) <br>[SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression](https://arxiv.org/abs/2306.03078) <br> Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, Dan Alistarh |<img width="1002" alt="image" src="figures/SpQR.png"> |[Github](https://github.com/Vahe1994/SpQR) <br> [Paper](https://arxiv.org/abs/2306.03078)|
|[![Star](https://img.shields.io/github/stars/xvyaward/owq.svg?style=social&label=Star)](https://github.com/xvyaward/owq) <br>[OWQ: Lessons learned from activation outliers for weight quantization in large language models](https://arxiv.org/abs/2306.02272) <br> Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, Eunhyeok Park |<img width="1002" alt="image" src="figures/OWQ.png"> |[Github](https://github.com/xvyaward/owq) <br> [Paper](https://arxiv.org/abs/2306.02272)|
|[![Star](https://img.shields.io/github/stars/RUCAIBox/QuantizedEmpirical.svg?style=social&label=Star)](https://github.com/RUCAIBox/QuantizedEmpirical)<br>[Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Study](https://arxiv.org/abs/2307.08072) <br> Peiyu Liu, Zikang Liu, Ze-Feng Gao, Dawei Gao, Wayne Xin Zhao, Yaliang Li, Bolin Ding, Ji-Rong Wen |<img width="1002" alt="image" src="figures/QuantizedEmpirical.png"> |[Github](https://github.com/RUCAIBox/QuantizedEmpirical) <br> [Paper](https://arxiv.org/abs/2307.08072)|
|[ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats](https://arxiv.org/abs/2307.09782) <br> Xiaoxia Wu, Zhewei Yao, Yuxiong He |<img width="1002" alt="image" src="figures/ZeroQuant-FP.png"> |[Paper](https://arxiv.org/abs/2307.09782)|
|[FPTQ: Fine-grained Post-Training Quantization for Large Language Models](https://arxiv.org/abs/2308.15987) <br> Qingyuan Li, Yifan Zhang, Liang Li, Peng Yao, Bo Zhang, Xiangxiang Chu, Yerui Sun, Li Du, Yuchen Xie |<img width="1002" alt="image" src="figures/FPTQ.png"> |[Paper](https://arxiv.org/abs/2308.15987)|
|[QuantEase: Optimization-based Quantization for Language Models - An Efficient and Intuitive Algorithm](https://arxiv.org/abs/2309.01885) <br> Kayhan Behdin, Ayan Acharya, Aman Gupta, Qingquan Song, Siyu Zhu, Sathiya Keerthi, Rahul Mazumder |<img width="1002" alt="image" src="figures/QuantEase.png"> |[Github](https://github.com/linkedin/QuantEase) <br> [Paper](https://arxiv.org/abs/2309.01885)|
|[Norm Tweaking: High-performance Low-bit Quantization of Large Language Models](https://arxiv.org/abs/2309.02784) <br> Liang Li, Qingyuan Li, Bo Zhang, Xiangxiang Chu |<img width="302" alt="image" src="figures/NormTweaking.png"> |[Paper](https://arxiv.org/abs/2309.02784)|
|[Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs](https://arxiv.org/abs/2309.05516) <br> Wenhua Cheng, Weiwei Zhang, Haihao Shen, Yiyang Cai, Xin He, Kaokao Lv |<img width="1002" alt="image" src="figures/SignRound.png"> |[Github](https://github.com/intel/neural-compressor) <br> [Paper](https://arxiv.org/abs/2309.05516)|
|[![Star](https://img.shields.io/github/stars/yuhuixu1993/qa-lora.svg?style=social&label=Star)](https://github.com/yuhuixu1993/qa-lora)<br>[QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2309.14717) <br> Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhensu Chen, Xiaopeng Zhang, Qi Tian |<img width="1002" alt="image" src="https://github.com/yuhuixu1993/qa-lora/blob/main/image/qalora.png"> |[Github](https://github.com/yuhuixu1993/qa-lora) <br> [Paper](https://arxiv.org/abs/2309.14717)|
|[ModuLoRA: Finetuning 3-Bit LLMs on Consumer GPUs by Integrating with Modular Quantizers](https://arxiv.org/abs/2309.16119) <br> Junjie Yin, Jiahao Dong, Yingheng Wang, Christopher De Sa, Volodymyr Kuleshov |<img width="1002" alt="image" src="figures/ModuLoRA.png"> |[Paper](https://arxiv.org/abs/2309.16119)|
|[![Star](https://img.shields.io/github/stars/hahnyuan/PB-LLM.svg?style=social&label=Star)](https://github.com/hahnyuan/PB-LLM)<br>[PB-LLM: Partially Binarized Large Language Models](https://arxiv.org/abs/2310.00034) <br> Yuzhang Shang, Zhihang Yuan, Qiang Wu, Zhen Dong |<img width="1002" alt="image" src="figures/PB-LLM.png"> |[Github](https://github.com/hahnyuan/PB-LLM) <br> [Paper](https://arxiv.org/abs/2310.00034)|
|[Dual Grained Quantization: Efficient Fine-Grained Quantization for LLM](https://arxiv.org/abs/2310.04836) <br> Luoming Zhang, Wen Fei, Weijia Wu, Yefei He, Zhenyu Lou, Hong Zhou |<img width="1002" alt="image" src="figures/DGQ.png"> |[Paper](https://arxiv.org/abs/2310.04836)|
|[QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resources](https://arxiv.org/abs/2310.07147) <br> Zhikai Li, Xiaoxuan Liu, Banghua Zhu, Zhen Dong, Qingyi Gu, Kurt Keutzer |<img width="1002" alt="image" src="figures/QFT.png"> |[Paper](https://arxiv.org/abs/2310.07147)|
|[QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models](https://arxiv.org/abs/2310.08041) <br> Jing Liu, Ruihao Gong, Xiuying Wei, Zhiwei Dong, Jianfei Cai, Bohan Zhuang |<img width="1002" alt="image" src="figures/QLLM.png"> |[Paper](https://arxiv.org/abs/2310.08041)|
|[LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models](https://arxiv.org/abs/2310.08659) <br> Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, Tuo Zhao |<img width="1002" alt="image" src="figures/LoftQ.png"> |[Paper](https://arxiv.org/abs/2310.08659)|
|[TEQ: Trainable Equivalent Transformation for Quantization of LLMs](https://arxiv.org/abs/2310.10944) <br> Wenhua Cheng, Yiyang Cai, Kaokao Lv, Haihao Shen |<img width="1002" alt="image" src="figures/TEQ.png"> |[Github](https://github.com/intel/neural-compressor) <br> [Paper](https://arxiv.org/abs/2310.10944)|
|[BitNet: Scaling 1-bit Transformers for Large Language Models](https://arxiv.org/abs/2310.11453) <br> Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, Furu Wei |<img width="1002" alt="image" src="https://thegenerality.com/agi/assets/img/bitnet.png"> |[Paper](https://arxiv.org/abs/2310.11453)|
|[Atom: Low-bit Quantization for Efficient and Accurate LLM Serving](https://arxiv.org/abs/2310.19102) <br> Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, Baris Kasikci |<img width="302" alt="image" src="figures/atom.png"> |[Paper](https://arxiv.org/abs/2310.19102)|
|[AWEQ: Post-Training Quantization with Activation-Weight Equalization for Large Language Models](https://arxiv.org/abs/2311.01305) <br> Baisong Li, Xingwang Wang, Haixiao Xu |<img width="1002" alt="image" src="figures/AWE.png"> |[Paper](https://arxiv.org/abs/2311.01305)|
|[![Star](https://img.shields.io/github/stars/zhangsichengsjtu/AFPQ.svg?style=social&label=Star)](https://github.com/zhangsichengsjtu/AFPQ)<br>[AFPQ: Asymmetric Floating Point Quantization for LLMs](https://arxiv.org/abs/2311.01792) <br> Yijia Zhang, Sicheng Zhang, Shijie Cao, Dayou Du, Jianyu Wei, Ting Cao, Ningyi Xu |<img width="1002" alt="image" src="figures/AFPQ.png"> |[Github](https://github.com/zhangsichengsjtu/AFPQ) <br> [Paper](https://arxiv.org/abs/2311.01792)|
|[A Speed Odyssey for Deployable Quantization of LLMs](https://arxiv.org/abs/2311.09550) <br> Qingyuan Li, Ran Meng, Yiduo Li, Bo Zhang, Liang Li, Yifan Lu, Xiangxiang Chu, Yerui Sun, Yuchen Xie |<img width="1002" alt="image" src="figures/OdysseyLLM.png"> |[Paper](https://arxiv.org/abs/2311.09550)|
|[![Star](https://img.shields.io/github/stars/HanGuo97/lq-lora.svg?style=social&label=Star)](https://github.com/HanGuo97/lq-lora)<br>[LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning](https://arxiv.org/abs/2311.12023) <br> Han Guo, Philip Greengard, Eric P. Xing, Yoon Kim |<img width="1002" alt="image" src="figures/LQ-LoRA.png"> |[Github](https://github.com/HanGuo97/lq-lora) <br> [Paper](https://arxiv.org/abs/2311.12023)|
|[Enabling Fast 2-bit LLM on GPUs: Memory Alignment, Sparse Outlier, and Asynchronous Dequantization](https://arxiv.org/abs/2311.16442) <br> Jinhao Li, Shiyao Li, Jiaming Xu, Shan Huang, Yaoxiu Lian, Jun Liu, Yu Wang, Guohao Dai |<img width="1002" alt="image" src="figures/fast-2-bit.png"> |[Paper](https://arxiv.org/abs/2311.16442)|
|[![Star](https://img.shields.io/github/stars/adlik/smoothquantplus.svg?style=social&label=Star)](https://github.com/adlik/smoothquantplus)<br>[SmoothQuant+: Accurate and Efficient 4-bit Post-Training WeightQuantization for LLM](https://arxiv.org/abs/2312.03788) <br> Jiayi Pan, Chengcan Wang, Kaifu Zheng, Yangguang Li, Zhenyu Wang, Bin Feng |<img width="402" alt="image" src="figures/SmoothQuant+.png"> |[Github](https://github.com/adlik/smoothquantplus) <br> [Paper](https://arxiv.org/abs/2312.03788)|
|[ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Tasks](https://arxiv.org/abs/2312.08583) <br> Xiaoxia Wu, Haojun Xia, Stephen Youn, Zhen Zheng, Shiyang Chen, Arash Bakhtiari, Michael Wyatt, Yuxiong He, Olatunji Ruwase, Leon Song, Zhewei Yao |<img width="1002" alt="image" src="figures/zeroquant-6bit.png"> |[Github](https://github.com/microsoft/DeepSpeed) <br> [Paper](https://arxiv.org/abs/2312.08583)|
|[![Star](https://img.shields.io/github/stars/vahe1994/AQLM.svg?style=social&label=Star)](https://github.com/vahe1994/AQLM)<br>[Extreme Compression of Large Language Models via Additive Quantization](https://arxiv.org/abs/2401.06118) <br> Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, Dan Alistarh |<img width="1002" alt="image" src="figures/MCQ.png"> |[Github](https://github.com/vahe1994/AQLM) <br> [Paper](https://arxiv.org/abs/2401.06118)|
|[FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Design](https://arxiv.org/abs/2401.14112) <br> Haojun Xia, Zhen Zheng, Xiaoxia Wu, Shiyang Chen, Zhewei Yao, Stephen Youn, Arash Bakhtiari, Michael Wyatt, Donglin Zhuang, Zhongzhu Zhou, Olatunji Ruwase, Yuxiong He, Shuaiwen Leon Song |<img width="1002" alt="image" src="figures/FP6-LLM.png"> |[Paper](https://arxiv.org/abs/2401.14112)|
|[L4Q: Parameter Efficient Quantization-Aware Training on Large Language Models via LoRA-wise LSQ](https://arxiv.org/abs/2402.04902) <br> Hyesung Jeon, Yulhwa Kim, Jae-joon Kim |<img width="1002" alt="image" src="figures/L4Q.png"> |[Paper](https://arxiv.org/abs/2402.04902)|
|[![Star](https://img.shields.io/github/stars/Cornell-RelaxML/quip-sharp.svg?style=social&label=Star)](https://github.com/Cornell-RelaxML/quip-sharp)<br>[QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks](https://arxiv.org/abs/2402.04396) <br> Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, Christopher De Sa |<img width="1002" alt="image" src="figures/QuIP_sign.png"> |[Github](https://github.com/Cornell-RelaxML/quip-sharp) <br> [Paper](https://arxiv.org/abs/2402.04396)|
|[![Star](https://img.shields.io/github/stars/Aaronhuang-778/BiLLM.svg?style=social&label=Star)](https://github.com/Aaronhuang-778/BiLLM)<br>[BiLLM: Pushing the Limit of Post-Training Quantization for LLMs](https://arxiv.org/abs/2402.04291) <br> Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, Xiaojuan Qi |<img width="1002" alt="image" src="https://github.com/Aaronhuang-778/BiLLM/blob/main/imgs/main.png"> |[Github](https://github.com/Aaronhuang-778/BiLLM) <br> [Paper](https://arxiv.org/abs/2402.04291)|
|[![Star](https://img.shields.io/github/stars/htqin/ir-qlora.svg?style=social&label=Star)](https://github.com/htqin/ir-qlora)<br>[Accurate LoRA-Finetuning Quantization of LLMs via Information Retention](https://arxiv.org/abs/2402.05445) <br> Haotong Qin, Xudong Ma, Xingyu Zheng, Xiaoyang Li, Yang Zhang, Shouda Liu, Jie Luo, Xianglong Liu, Michele Magno |<img width="1002" alt="image" src="https://github.com/htqin/IR-QLoRA/blob/main/imgs/overview.png"> |[Github](https://github.com/htqin/ir-qlora) <br> [Paper](https://arxiv.org/abs/2402.05445)|
|[ApiQ: Finetuning of 2-Bit Quantized Large Language Model](https://arxiv.org/abs/2402.05147) <br> Baohao Liao, Christof Monz |<img width="302" alt="image" src="figures/ApiQ.png"> |[Paper](https://arxiv.org/abs/2402.05147)|
|[Towards Next-Level Post-Training Quantization of Hyper-Scale Transformers](https://arxiv.org/abs/2402.08958) <br> Junhan Kim, Kyungphil Park, Chungman Lee, Ho-young Kim, Joonyoung Kim, Yongkweon Jeon |<img width="1002" alt="image" src="https://arxiv.org/html/2402.08958v1/x1.png"> |[Paper](https://arxiv.org/abs/2402.08958)|
|[![Star](https://img.shields.io/github/stars/shawnricecake/EdgeQAT.svg?style=social&label=Star)](https://github.com/shawnricecake/EdgeQAT)<br>[EdgeQAT: Entropy and Distribution Guided Quantization-Aware Training for the Acceleration of Lightweight LLMs on the Edge](https://arxiv.org/abs/2402.10787) <br> Xuan Shen, Zhenglun Kong, Changdi Yang, Zhaoyang Han, Lei Lu, Peiyan Dong, Cheng Lyu, Chih-hsiang Li, Xuehang Guo, Zhihao Shu, Wei Niu, Miriam Leeser, Pu Zhao, Yanzhi Wang |<img width="1002" alt="image" src="figures/EdgeQAT.png"> |[Github](https://github.com/shawnricecake/EdgeQAT) <br> [Paper](https://arxiv.org/abs/2402.10787)|
|[![Star](https://img.shields.io/github/stars/DD-DuDa/BitDistiller.svg?style=social&label=Star)](https://github.com/DD-DuDa/BitDistiller)<br>[BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation](https://arxiv.org/abs/2402.10631) <br> Dayou Du, Yijia Zhang, Shijie Cao, Jiaqi Guo, Ting Cao, Xiaowen Chu, Ningyi Xu |<img width="202" alt="image" src="https://github.com/DD-DuDa/BitDistiller/raw/main/imgs/overview.jpg"> |[Github](https://github.com/DD-DuDa/BitDistiller) <br> [Paper](https://arxiv.org/abs/2402.10631)|
|[OneBit: Towards Extremely Low-bit Large Language Models](https://arxiv.org/abs/2402.11295) <br> Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, Wanxiang Che |<img width="1002" alt="image" src="figures/OneBit.png"> |[Paper](https://arxiv.org/abs/2402.11295)|
|[![Star](https://img.shields.io/github/stars/FasterDecoding/BitDelta.svg?style=social&label=Star)](https://github.com/FasterDecoding/BitDelta)<br>[BitDelta: Your Fine-Tune May Only Be Worth One Bit](https://arxiv.org/abs/2402.10193) <br> James Liu, Guangxuan Xiao, Kai Li, Jason D. Lee, Song Han, Tri Dao, Tianle Cai |<img width="1002" alt="image" src="https://github.com/FasterDecoding/BitDelta/raw/main/figures/BitDelta.png"> |[Github](https://github.com/FasterDecoding/BitDelta) <br> [Paper](https://arxiv.org/abs/2402.10193)|
|[Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs](https://arxiv.org/abs/2402.10517) <br> Yeonhong Park, Jake Hyun, SangLyul Cho, Bonggeun Sim, Jae W. Lee |<img width="1002" alt="image" src="figures/AnyPrecisionLLM.png"> |[Paper](https://arxiv.org/abs/2402.10517)|
|[![Publish](https://img.shields.io/badge/Conference-DAC'24-blue)]()<br>[APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Models](https://arxiv.org/abs/2402.14866) <br> Ziyi Guan, Hantao Huang, Yupeng Su, Hong Huang, Ngai Wong, Hao Yu |<img width="1002" alt="image" src="https://arxiv.org/html/2402.14866v1/x1.png"> |[Paper](https://arxiv.org/abs/2402.14866)|
|[![Star](https://img.shields.io/github/stars/qualcomm-ai-research/gptvq.svg?style=social&label=Star)](https://github.com/qualcomm-ai-research/gptvq)<br>[GPTVQ: The Blessing of Dimensionality for LLM Quantization](https://arxiv.org/abs/2402.15319) <br> Mart van Baalen, Andrey Kuzmin, Markus Nagel, Peter Couperus, Cedric Bastoul, Eric Mahurin, Tijmen Blankevoort, Paul Whatmough |<img width="1002" alt="image" src="https://arxiv.org/html/2402.15319v1/extracted/5412979/fig/new_fig1a_blue.png"> |[Github](https://github.com/qualcomm-ai-research/gptvq) <br> [Paper](https://arxiv.org/abs/2402.15319)|
|[A Comprehensive Evaluation of Quantization Strategies for Large Language Models](https://arxiv.org/abs/2402.16775) <br> Renren Jin, Jiangcun Du, Wuwei Huang, Wei Liu, Jian Luan, Bin Wang, Deyi Xiong |<img width="1002" alt="image" src="figures/QuantizationStrategies.png"> |[Paper](https://arxiv.org/abs/2402.16775)|
|[The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://arxiv.org/abs/2402.17764) <br> Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, Furu Wei |<img width="1002" alt="image" src="https://arxiv.org/html/2402.17764v1/x1.png"> |[Paper](https://arxiv.org/abs/2402.17764)|
|[![Star](https://img.shields.io/github/stars/thu-nics/qllm-eval.svg?style=social&label=Star)](https://github.com/thu-nics/qllm-eval)<br>[Evaluating Quantized Large Language Models](https://arxiv.org/abs/2402.18158) <br> Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, Yu Wang |<img width="302" alt="image" src="figures/qllm-eval.png"> |[Github](https://github.com/thu-nics/qllm-eval) <br> [Paper](https://arxiv.org/abs/2402.18158)|
|[FlattenQuant: Breaking Through the Inference Compute-bound for Large Language Models with Per-tensor Quantization](https://arxiv.org/abs/2402.17985) <br> Yi Zhang, Fei Yang, Shuang Peng, Fangyu Wang, Aimin Pan |<img width="1002" alt="image" src="https://arxiv.org/html/2402.17985v1/extracted/5426624/figure/Flatten_fig_new2.png"> |[Paper](https://arxiv.org/abs/2402.17985)|
|[What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbation](https://arxiv.org/abs/2403.06408) <br> Zhuocheng Gong, Jiahao Liu, Jingang Wang, Xunliang Cai, Dongyan Zhao, Rui Yan |<img width="1002" alt="image" src="https://arxiv.org/html/2403.06408v1/x1.png"> |[Paper](https://arxiv.org/abs/2403.06408)|
|[FrameQuant: Flexible Low-Bit Quantization for Transformers](https://arxiv.org/abs/2403.06082) <br> Harshavardhan Adepu, Zhanpeng Zeng, Li Zhang, Vikas Singh |<img width="1002" alt="image" src="https://arxiv.org/html/2403.06082v1/extracted/5460218/figures/mainFigure4.png"> |[Paper](https://arxiv.org/abs/2403.06082)|
|[![Star](https://img.shields.io/github/stars/spcl/QuaRot.svg?style=social&label=Star)](https://github.com/spcl/QuaRot)<br>[QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs](https://arxiv.org/abs/2404.00456) <br> Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, James Hensman |<img width="1002" alt="image" src="https://github.com/spcl/QuaRot/blob/main/img/fig1.png"> |[Github](https://github.com/spcl/QuaRot) <br> [Paper](https://arxiv.org/abs/2404.00456)|
|[Accurate Block Quantization in LLMs with Outliers](https://arxiv.org/abs/2403.20137) <br> Nikita Trukhanov, Ilya Soloveychik |<img width="1002" alt="image" src="https://arxiv.org/html/2403.20137v1/extracted/5504178/WXT.png"> |[Paper](https://arxiv.org/abs/2403.20137)|
|[Cherry on Top: Parameter Heterogeneity and Quantization in Large Language Models](https://arxiv.org/abs/2404.02837) <br> Wanyun Cui, Qianle Wang |<img width="1002" alt="image" src="https://arxiv.org/html/2404.02837v1/x1.png"> |[Paper](https://arxiv.org/abs/2404.02837)|
|[Increased LLM Vulnerabilities from Fine-tuning and Quantization](https://arxiv.org/abs/2404.04392) <br> Divyanshu Kumar, Anurakt Kumar, Sahil Agarwal, Prashanth Harshangi |<img width="1002" alt="image" src="https://arxiv.org/html/2404.04392v1/extracted/5518332/expt.png"> |[Paper](https://arxiv.org/abs/2404.04392)|
|[Quantization of Large Language Models with an Overdetermined Basis](https://arxiv.org/abs/2404.09737) <br> Daniil Merkulov, Daria Cherniuk, Alexander Rudikov, Ivan Oseledets, Ekaterina Muravleva, Aleksandr Mikhalev, Boris Kashin |<img width="302" alt="image" src="https://arxiv.org/html/2404.09737v1/x1.png"> |[Paper](https://arxiv.org/abs/2404.09737)|
|[![Star](https://img.shields.io/github/stars/bytedance/decoupleQ.svg?style=social&label=Star)](https://github.com/bytedance/decoupleQ)<br>[decoupleQ: Towards 2-bit Post-Training Uniform Quantization via decoupling Parameters into Integer and Floating Points](https://arxiv.org/abs/2404.12759) <br> Yi Guo, Fanliu Kong, Xiaoyang Li, Hui Li, Wei Chen, Xiaogang Tian, Jinping Cai, Yang Zhang, Shouda Liu |<img width="1002" alt="image" src="https://github.com/bytedance/decoupleQ/raw/main/imgs/img.png"> |[Github](https://github.com/bytedance/decoupleQ) <br> [Paper](https://arxiv.org/abs/2404.12759)|
|[Lossless and Near-Lossless Compression for Foundation Models](https://arxiv.org/abs/2404.15198) <br> Moshik Hershcovitch, Leshem Choshen, Andrew Wood, Ilias Enmouri, Peter Chin, Swaminathan Sundararaman, Danny Harnik |<img width="1002" alt="image" src="figures/compress_rank.png"> |[Paper](https://arxiv.org/abs/2404.15198)|
|[![Star](https://img.shields.io/github/stars/Macaronlin/LLaMA3-Quantization.svg?style=social&label=Star)](https://github.com/Macaronlin/LLaMA3-Quantization)<br>[How Good Are Low-bit Quantized LLaMA3 Models? An Empirical Study](https://arxiv.org/abs/2404.14047) <br> Wei Huang, Xudong Ma, Haotong Qin, Xingyu Zheng, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, Michele Magno |<img width="1002" alt="image" src="https://arxiv.org/html/2404.14047v1/x1.png"> |[Github](https://github.com/Macaronlin/LLaMA3-Quantization) <br> [Paper](https://arxiv.org/abs/2404.14047) <br> [Model](https://huggingface.co/LLMQ)|
|[![Star](https://img.shields.io/github/stars/upunaprosk/quantized-lm-confidence.svg?style=social&label=Star)](https://github.com/upunaprosk/quantized-lm-confidence)[![Publish](https://img.shields.io/badge/Conference-NAACL'24%20Findings-blue)]()<br>[When Quantization Affects Confidence of Large Language Models?](https://arxiv.org/abs/2405.00632) <br> Irina Proskurina, Luc Brun, Guillaume Metzler, Julien Velcin |<img width="1002" alt="image" src="figures/quantized-lm-confidence.png"> |[Github](https://github.com/upunaprosk/quantized-lm-confidence) <br> [Paper](https://arxiv.org/abs/2405.00632)|
|[![Star](https://img.shields.io/github/stars/mit-han-lab/qserve.svg?style=social&label=Star)](https://github.com/mit-han-lab/qserve)<br>[QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving](https://arxiv.org/abs/2405.04532) <br> Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, Song Han |<img width="1002" alt="image" src="https://github.com/mit-han-lab/qserve/blob/main/assets/figures/teaser.png"> |[Github](https://github.com/mit-han-lab/qserve) <br> [Paper](https://arxiv.org/abs/2405.04532)|
|[![Star](https://img.shields.io/github/stars/ModelTC/llmc.svg?style=social&label=Star)](https://github.com/ModelTC/llmc)<br>[LLM-QBench: A Benchmark Towards the Best Practice for Post-training Quantization of Large Language Models](https://arxiv.org/abs/2405.06001) <br> Ruihao Gong, Yang Yong, Shiqiao Gu, Yushi Huang, Yunchen Zhang, Xianglong Liu, Dacheng Tao |<img width="1002" alt="image" src="https://github.com/ModelTC/llmc/raw/main/imgs/best_practice.png"> |[Github](https://github.com/ModelTC/llmc) <br> [Paper](https://arxiv.org/abs/2405.06001)|
|[![Star](https://img.shields.io/github/stars/eth-sri/llm-quantization-attack.svg?style=social&label=Star)](https://github.com/eth-sri/llm-quantization-attack)<br>[Exploiting LLM Quantization](https://arxiv.org/abs/2405.18137) <br> Kazuki Egashira, Mark Vero, Robin Staab, Jingxuan He, Martin Vechev |<img width="1002" alt="image" src="figures/exploiting_llm_quantization.png"> |[Github](https://github.com/eth-sri/llm-quantization-attack) <br> [Paper](https://arxiv.org/abs/2405.18137)|
|[CLAQ: Pushing the Limits of Low-Bit Post-Training Quantization for LLMs](https://arxiv.org/abs/2405.17233) <br> Haoyu Wang, Bei Liu, Hang Shao, Bo Xiao, Ke Zeng, Guanglu Wan, Yanmin Qian |<img width="1002" alt="image" src="https://arxiv.org/html/2405.17233v1/x1.png"> |[Paper](https://arxiv.org/abs/2405.17233)|
|[SpinQuant -- LLM quantization with learned rotations](https://arxiv.org/abs/2405.16406) <br> Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, Tijmen Blankevoort |<img width="1002" alt="image" src="figures/spinquant.png"> |[Paper](https://arxiv.org/abs/2405.16406)|
|[![Star](https://img.shields.io/github/stars/Aaronhuang-778/SliM-LLM.svg?style=social&label=Star)](https://github.com/Aaronhuang-778/SliM-LLM)<br>[SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models](https://arxiv.org/abs/2405.14917) <br> Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Xianglong Liu, Luca Benini, Michele Magno, Xiaojuan Qi |<img width="1002" alt="image" src="https://github.com/Aaronhuang-778/SliM-LLM/blob/main/imgs/WX20240527-155305@2x.png"> |[Github](https://github.com/Aaronhuang-778/SliM-LLM) <br> [Paper](https://arxiv.org/abs/2405.14917)|
|[![Star](https://img.shields.io/github/stars/tree/pv-tuning.svg?style=social&label=Star)](https://github.com/tree/pv-tuning)<br>[PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression](https://arxiv.org/abs/2405.14852) <br> Vladimir Malinovskii, Denis Mazur, Ivan Ilin, Denis Kuznedelev, Konstantin Burlachenko, Kai Yi, Dan Alistarh, Peter Richtarik |<img width="1002" alt="image" src="figures/pv-tuning.png"> |[Github](https://github.com/Vahe1994/AQLM/tree/pv-tuning) <br> [Paper](https://arxiv.org/abs/2405.14852)|
|[Integer Scale: A Free Lunch for Faster Fine-grained Quantization of LLMs](https://arxiv.org/abs/2405.14597) <br> Qingyuan Li, Ran Meng, Yiduo Li, Bo Zhang, Yifan Lu, Yerui Sun, Lin Ma, Yuchen Xie |<img width="1002" alt="image" src="https://arxiv.org/html/2405.14597v2/x2.png"> |[Paper](https://arxiv.org/abs/2405.14597)|
|[I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models](https://arxiv.org/abs/2405.17849) <br> Xing Hu, Yuan Chen, Dawei Yang, Sifan Zhou, Zhihang Yuan, Jiangyong Yu, Chen Xu |<img width="1002" alt="image" src="figures/I-LLM.png"> |[Paper](https://arxiv.org/abs/2405.17849)|
|[![Star](https://img.shields.io/github/stars/pilancilab/caldera.svg?style=social&label=Star)](https://github.com/pilancilab/caldera)<br>[Compressing Large Language Models using Low Rank and Low Precision Decomposition](https://arxiv.org/abs/2405.18886) <br> Rajarshi Saha, Naomi Sagan, Varun Srivastava, Andrea J. Goldsmith, Mert Pilanci |<img width="1002" alt="image" src="https://arxiv.org/html/2405.18886v1/x1.png"> |[Github](https://github.com/pilancilab/caldera) <br> [Paper](https://arxiv.org/abs/2405.18886)|
|[MagR: Weight Magnitude Reduction for Enhancing Post-Training Quantization](https://arxiv.org/abs/2406.00800) <br> Aozhong Zhang, Naigang Wang, Yanxia Deng, Xin Li, Zi Yang, Penghang Yin |<img width="1002" alt="image" src="https://arxiv.org/html/2406.00800v1/extracted/5638265/figure/magr.png"> |[Paper](https://arxiv.org/abs/2406.00800)|[//]: #06/05
|[LCQ: Low-Rank Codebook based Quantization for Large Language Models](https://arxiv.org/abs/2405.20973) <br> Wen-Pu Cai, Wu-Jun Li |<img width="1002" alt="image" src="https://arxiv.org/html/2405.20973v1/x5.png"> |[Paper](https://arxiv.org/abs/2405.20973)|[//]: #06/05
|[Outliers and Calibration Sets have Diminishing Effect on Quantization of Modern LLMs](https://arxiv.org/abs/2405.20835) <br> Davide Paglieri, Saurabh Dash, Tim Rocktäschel, Jack Parker-Holder | |[Paper](https://arxiv.org/abs/2405.20835)|[//]: #06/05
|[Low-Rank Quantization-Aware Training for LLMs](https://arxiv.org/abs/2406.06385) <br> Yelysei Bondarenko, Riccardo Del Chiaro, Markus Nagel |<img width="1002" alt="image" src="https://arxiv.org/html/2406.06385v1/extracted/5656645/img/01_lora_qat_proposed_palette.png"> |[Paper](https://arxiv.org/abs/2406.06385)|[//]: #06/11
|[![Star](https://img.shields.io/github/stars/GATECH-EIC/ShiftAddLLM.svg?style=social&label=Star)](https://github.com/GATECH-EIC/ShiftAddLLM)<br>[ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization](https://arxiv.org/abs/2406.05981) <br> Haoran You, Yipin Guo, Yichao Fu, Wei Zhou, Huihong Shi, Xiaofan Zhang, Souvik Kundu, Amir Yazdanbakhsh, Yingyan Lin |<img width="1002" alt="image" src="https://github.com/GATECH-EIC/ShiftAddLLM/raw/main/assets/overview.jpg"> |[Github](https://github.com/GATECH-EIC/ShiftAddLLM) <br> [Paper](https://arxiv.org/abs/2406.05981)|[//]: #06/11
|[![Star](https://img.shields.io/github/stars/HandH1998/QQQ.svg?style=social&label=Star)](https://github.com/HandH1998/QQQ)<br>[QQQ: Quality Quattuor-Bit Quantization for Large Language Models](https://arxiv.org/abs/2406.09904) <br> Ying Zhang, Peng Zhang, Mincong Huang, Jingyang Xiang, Yujie Wang, Chao Wang, Yineng Zhang, Lei Yu, Chuan Liu, Wei Lin |<img width="202" alt="image" src="https://arxiv.org/html/2406.09904v1/x1.png"> |[Github](https://github.com/HandH1998/QQQ) <br> [Paper](https://arxiv.org/abs/2406.09904)|[//]: #06/18
|[Attention-aware Post-training Quantization without Backpropagation](https://arxiv.org/abs/2406.13474) <br> Junhan Kim, Ho-young Kim, Eulrang Cho, Chungman Lee, Joonyoung Kim, Yongkweon Jeon |<img width="1002" alt="image" src="https://arxiv.org/html/2406.13474v1/x1.png"> |[Paper](https://arxiv.org/abs/2406.13474)|[//]: #06/24
|[Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantization](https://arxiv.org/abs/2406.12016) <br> Seungwoo Son, Wonpyo Park, Woohyun Han, Kyuyeun Kim, Jaeho Lee |<img width="1002" alt="image" src="https://arxiv.org/html/2406.12016v1/extracted/5669665/figures/mainfig.png"> |[Paper](https://arxiv.org/abs/2406.12016)|[//]: #06/24
|[SDQ: Sparse Decomposed Quantization for LLM Inference](https://arxiv.org/abs/2406.13868) <br> Geonhwa Jeong, Po-An Tsai, Stephen W. Keckler, Tushar Krishna |<img width="1002" alt="image" src="https://arxiv.org/html/2406.13868v1/x3.png"> |[Paper](https://arxiv.org/abs/2406.13868)|[//]: #06/24
|[CDQuant: Accurate Post-training Weight Quantization of Large Pre-trained Models using Greedy Coordinate Descent](https://arxiv.org/abs/2406.17542) <br> Pranav Ajit Nair, Arun Sai Suggala |<img width="1002" alt="image" src="figures/CD.png"> |[Paper](https://arxiv.org/abs/2406.17542)|[//]: #06/26
|[![Star](https://img.shields.io/github/stars/RazvanDu/LayerwiseQuant.svg?style=social&label=Star)](https://github.com/RazvanDu/LayerwiseQuant)<br>[Variable Layer-Wise Quantization: A Simple and Effective Approach to Quantize LLMs](https://arxiv.org/abs/2406.17415) <br> Razvan-Gabriel Dumitru, Vikas Yadav, Rishabh Maheshwary, Paul-Ioan Clotan, Sathwik Tejaswi Madhusudhan, Mihai Surdeanu |<img width="202" alt="image" src="https://arxiv.org/html/2406.17415v1/x1.png"> |[Github](https://github.com/RazvanDu/LayerwiseQuant) <br> [Paper](https://arxiv.org/abs/2406.17415)|[//]: #06/26
|[![Star](https://img.shields.io/github/stars/microsoft/T-MAC.svg?style=social&label=Star)](https://github.com/microsoft/T-MAC)<br>[T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge](https://arxiv.org/abs/2407.00088) <br> Jianyu Wei, Shijie Cao, Ting Cao, Lingxiao Ma, Lei Wang, Yanyong Zhang, Mao Yang |<img width="1002" alt="image" src="https://arxiv.org/html/2407.00088v1/x2.png"> |[Github](https://github.com/microsoft/T-MAC) <br> [Paper](https://arxiv.org/abs/2407.00088)|[//]: #07/03
|[![Publish](https://img.shields.io/badge/Conference-CIS-RAM'24-blue)]()<br>[GPTQT: Quantize Large Language Models Twice to Push the Efficiency](https://arxiv.org/abs/2407.02891) <br> Yipin Guo, Yilin Lang, Qinyuan Ren |<img width="1002" alt="image" src="https://arxiv.org/html/2407.02891v1/x1.png"> |[Paper](https://arxiv.org/abs/2407.02891)|[//]: #07/05
|[![Star](https://img.shields.io/github/stars/LiqunMa/FBI-LLM.svg?style=social&label=Star)](https://github.com/LiqunMa/FBI-LLM)<br>[FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillation](https://arxiv.org/abs/2407.07093) <br> Liqun Ma, Mingjie Sun, Zhiqiang Shen |<img width="1002" alt="image" src="https://github.com/LiqunMa/FBI-LLM/blob/main/figures/structure_and_training_procedure.png"> |[Github](https://github.com/LiqunMa/FBI-LLM) <br> [Paper](https://arxiv.org/abs/2407.07093)|[//]: #07/10
|[![Star](https://img.shields.io/github/stars/HuangOwen/RoLoRA.svg?style=social&label=Star)](https://github.com/HuangOwen/RoLoRA)<br>[RoLoRA: Fine-tuning Rotated Outlier-free LLMs for Effective Weight-Activation Quantization](https://arxiv.org/abs/2407.08044) <br> Xijie Huang, Zechun Liu, Shih-Yang Liu, Kwang-Ting Cheng |<img width="1002" alt="image" src="https://arxiv.org/html/2407.08044v1/x1.png"> |[Github](https://github.com/HuangOwen/RoLoRA) <br> [Paper](https://arxiv.org/abs/2407.08044)|[//]: #07/12
|[Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantization](https://arxiv.org/abs/2406.12016) <br> Seungwoo Son, Wonpyo Park, Woohyun Han, Kyuyeun Kim, Jaeho Lee |<img width="1002" alt="image" src="https://arxiv.org/html/2406.12016v1/extracted/5669665/figures/mainfig.png"> |[Paper](https://arxiv.org/abs/2406.12016)|[//]: #07/16
|[LeanQuant: Accurate Large Language Model Quantization with Loss-Error-Aware Grid](https://arxiv.org/abs/2407.10032) <br> Tianyi Zhang, Anshumali Shrivastava |<img width="1002" alt="image" src="https://arxiv.org/html/2407.10032v1/x2.png"> |[Paper](https://arxiv.org/abs/2407.10032)|[//]: #07/16
|[![Star](https://img.shields.io/github/stars/HanGuo97/flute.svg?style=social&label=Star)](https://github.com/HanGuo97/flute)<br>[Fast Matrix Multiplications for Lookup Table-Quantized LLMs](https://arxiv.org/abs/2407.10960) <br> Han Guo, William Brandon, Radostin Cholakov, Jonathan Ragan-Kelley, Eric P. Xing, Yoon Kim |<img width="302" alt="image" src="https://arxiv.org/html/2407.10960v1/x1.png"> |[Github](https://github.com/HanGuo97/flute) <br> [Paper](https://arxiv.org/abs/2407.10960)|[//]: #07/16
|[![Star](https://img.shields.io/github/stars/NolanoOrg/SpectraSuite.svg?style=social&label=Star)](https://github.com/NolanoOrg/SpectraSuite)<br>[Spectra: A Comprehensive Study of Ternary, Quantized, and FP16 Language Models](https://arxiv.org/abs/2407.12327) <br> Ayush Kaushal, Tejas Pandey, Tejas Vaidhya, Aaryan Bhagat, Irina Rish |<img width="1002" alt="image" src="https://arxiv.org/html/2407.11722v1/x1.png"> |[Github](https://github.com/NolanoOrg/SpectraSuite) <br> [Paper](https://arxiv.org/abs/2407.12327)|[//]: #07/21
|[![Star](https://img.shields.io/github/stars/onliwad101/FlexRound_LRQ.svg?style=social&label=Star)](https://github.com/onliwad101/FlexRound_LRQ)<br>[LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices](https://arxiv.org/abs/2407.11534) <br> Jung Hyun Lee, Jeonghoon Kim, June Yong Yang, Se Jung Kwon, Eunho Yang, Kang Min Yoo, Dongsoo Lee |<img width="1002" alt="image" src="https://arxiv.org/html/2407.11534v1/extracted/5734567/Figures/Fig_ablation_samplesize_flexround.png"> |[Github](https://github.com/onliwad101/FlexRound_LRQ) <br> [Paper](https://arxiv.org/abs/2407.11534)|[//]: #07/21
|[![Star](https://img.shields.io/github/stars/OpenGVLab/EfficientQAT.svg?style=social&label=Star)](https://github.com/OpenGVLab/EfficientQAT)<br>[EfficientQAT: Efficient Quantization-Aware Training for Large Language Models](https://arxiv.org/abs/2407.11062) <br> Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, Yu Qiao, Ping Luo |<img width="1002" alt="image" src="https://arxiv.org/html/2407.11062v1/x5.png"> |[Github](https://github.com/OpenGVLab/EfficientQAT) <br> [Paper](https://arxiv.org/abs/2407.11062)|[//]: #07/21
|[![Star](https://img.shields.io/github/stars/graphcore-research/jax-scalify.svg?style=social&label=Star)](https://github.com/graphcore-research/jax-scalify)[![Publish](https://img.shields.io/badge/Conference-ICML'24%20WANT-blue)]()<br>[Scalify: scale propagation for efficient low-precision LLM training](https://arxiv.org/abs/2407.17353) <br> Paul Balança, Sam Hosegood, Carlo Luschi, Andrew Fitzgibbon | |[Github](https://github.com/graphcore-research/jax-scalify) <br> [Paper](https://arxiv.org/abs/2407.17353)|[//]: #07/26
|[![Star](https://img.shields.io/github/stars/xiaocaigou/qbaraqahira.svg?style=social&label=Star)](https://github.com/xiaocaigou/qbaraqahira)<br>[Accurate and Efficient Fine-Tuning of Quantized Large Language Models Through Optimal Balance](https://arxiv.org/abs/2407.17029) <br> Ao Shen, Qiang Wang, Zhiquan Lai, Xionglve Li, Dongsheng Li |<img width="1002" alt="image" src="figures/Q-BaRA.png"> |[Github](https://github.com/xiaocaigou/qbaraqahira) <br> [Paper](https://arxiv.org/abs/2407.17029)|[//]: #07/26
|[STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMs](https://arxiv.org/abs/2408.01803) <br> Peijie Dong, Lujun Li, Dayou Du, Yuhan Chen, Zhenheng Tang, Qiang Wang, Wei Xue, Wenhan Luo, Qifeng Liu, Yike Guo, Xiaowen Chu |<img width="1002" alt="image" src="https://arxiv.org/html/2408.01803v1/extracted/5772020/pic/basic_block.png"> |[Paper](https://arxiv.org/abs/2408.01803)|[//]: #08/08
|[![Star](https://img.shields.io/github/stars/bytedance/ABQ-LLM.svg?style=social&label=Star)](https://github.com/bytedance/ABQ-LLM)<br>[ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models](https://arxiv.org/abs/2408.08554) <br> Chao Zeng, Songwei Liu, Yusheng Xie, Hong Liu, Xiaojian Wang, Miao Wei, Shu Yang, Fangmin Chen, Xing Mei |<img width="1002" alt="image" src="figures/abq-llm.png"> |[Github](https://github.com/bytedance/ABQ-LLM) <br> [Paper](https://arxiv.org/abs/2408.08554)|[//]: #08/20
|[Matmul or No Matmal in the Era of 1-bit LLMs](https://arxiv.org/abs/2408.11939) <br> Jinendra Malekar, Mohammed E. Elbtity, Ramtin Zand Co |<img width="1002" alt="image" src="https://arxiv.org/html/2408.11939v1/extracted/5805924/figures/matmul.png"> |[Paper](https://arxiv.org/abs/2408.11939)|[//]: #08/27
|[![Star](https://img.shields.io/github/stars/saic-fi/MobileQuant.svg?style=social&label=Star)](https://github.com/saic-fi/MobileQuant)<br>[MobileQuant: Mobile-friendly Quantization for On-device Language Models](https://arxiv.org/abs/2408.13933) <br> Fuwen Tan, Royson Lee, Łukasz Dudziak, Shell Xu Hu, Sourav Bhattacharya, Timothy Hospedales, Georgios Tzimiropoulos, Brais Martinez |<img width="1002" alt="image" src="https://arxiv.org/html/2408.13933v1/x1.png"> |[Github](https://github.com/saic-fi/MobileQuant) <br> [Paper](https://arxiv.org/abs/2408.13933)|[//]: #08/27
|[The Uniqueness of LLaMA3-70B with Per-Channel Quantization: An Empirical Study](https://arxiv.org/abs/2408.15301) <br> Minghai Qin |<img width="1002" alt="image" src="https://arxiv.org/html/2408.15301v1/extracted/5797059/LaTeX/figures/llama3-70b-series-accuracy.png"> |[Paper](https://arxiv.org/abs/2408.15301)|[//]: #09/02
|[A Comprehensive Evaluation of Quantized Instruction-Tuned Large Language Models: An Experimental Analysis up to 405B](https://arxiv.org/abs/2409.11055) <br> Jemin Lee, Sihyeong Park, Jinse Kwon, Jihun Oh, Yongin Kwon |<img width="1002" alt="image" src="https://arxiv.org/html/2409.11055v1/x1.png"> |[Paper](https://arxiv.org/abs/2409.11055)|[//]: #09/21
|[![Star](https://img.shields.io/github/stars/Hsu1023/DuQuant?tab=readme-ov-file.svg?style=social&label=Star)](https://github.com/Hsu1023/DuQuant?tab=readme-ov-file)[![Publish](https://img.shields.io/badge/Conference-NeurIPS'24-blue)]()<br>[DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs](https://arxiv.org/abs/2406.01721) <br> Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, Ying Wei |<img width="1002" alt="image" src="https://github.com/Hsu1023/DuQuant/blob/main/imgs/duquant.png"> |[Github](https://github.com/Hsu1023/DuQuant?tab=readme-ov-file) <br> [Paper](https://arxiv.org/abs/2406.01721)|[//]: #09/27
|[Accumulator-Aware Post-Training Quantization](https://arxiv.org/abs/2409.17092) <br> Ian Colbert, Fabian Grob, Giuseppe Franco, Jinjie Zhang, Rayan Saab |<img width="1002" alt="image" src="https://arxiv.org/html/2409.17092v1/x2.png"> |[Paper](https://arxiv.org/abs/2409.17092)|[//]: #09/27
|[![Star](https://img.shields.io/github/stars/INT-FlashAttention2024/INT-FlashAttention.svg?style=social&label=Star)](https://github.com/INT-FlashAttention2024/INT-FlashAttention)<br>[INT-FlashAttention: Enabling Flash Attention for INT8 Quantization](https://arxiv.org/abs/2409.16997) <br> Shimao Chen, Zirui Liu, Zhiying Wu, Ce Zheng, Peizhuang Cong, Zihan Jiang, Yuhan Wu, Lei Su, Tong Yang |<img width="1002" alt="image" src="https://arxiv.org/html/2409.16997v2/x1.png"> |[Github](https://github.com/INT-FlashAttention2024/INT-FlashAttention) <br> [Paper](https://arxiv.org/abs/2409.16997)|[//]: #09/27
|[![Star](https://img.shields.io/github/stars/microsoft/VPTQ.svg?style=social&label=Star)](https://github.com/microsoft/VPTQ)<br>[VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models](https://arxiv.org/abs/2409.17066) <br> Yifei Liu, Jicheng Wen, Yang Wang, Shengyu Ye, Li Lyna Zhang, Ting Cao, Cheng Li, Mao Yang |<img width="1002" alt="image" src="figures/VPTQ.png"> |[Github](https://github.com/microsoft/VPTQ) <br> [Paper](https://arxiv.org/abs/2409.17066)|[//]: #09/27
|[Addition is All You Need for Energy-efficient Language Models](https://arxiv.org/abs/2410.00907) <br> Hongyin Luo, Wei Sun |<img width="1002" alt="image" src="https://arxiv.org/html/2410.00907v1/x2.png"> |[Paper](https://arxiv.org/abs/2410.00907)|[//]: #10/02
|[SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration](https://arxiv.org/abs/2410.02367) <br> Jintao Zhang, Jia wei, Pengle Zhang, Jun Zhu, Jianfei Chen |<img width="1002" alt="image" src="https://arxiv.org/html/2410.02367v1/x5.png"> |[Paper](https://arxiv.org/abs/2410.02367)|[//]: #10/04
|[CrossQuant: A Post-Training Quantization Method with Smaller Quantization Kernel for Precise Large Language Model Compression](https://arxiv.org/abs/2410.07505) <br> Wenyuan Liu, Xindian Ma, Peng Zhang, Yan Wang |<img width="1002" alt="image" src="https://arxiv.org/html/2410.07505v1/x1.png"> |[Paper](https://arxiv.org/abs/2410.07505)|[//]: #10/13
|[PalmBench: A Comprehensive Benchmark of Compressed Large Language Models on Mobile Platforms](https://arxiv.org/abs/2410.05315) <br> Yilong Li, Jingyu Liu, Hao Zhang, M Badri Narayanan, Utkarsh Sharma, Shuai Zhang, Pan Hu, Yijing Zeng, Jayaram Raghuram, Suman Banerjee |<img width="1002" alt="image" src="figures/PalmBench.png"> |[Paper](https://arxiv.org/abs/2410.05315)|[//]: #10/14
|[Scaling Laws for Mixed quantization in Large Language Models](https://arxiv.org/abs/2410.06722) <br> Zeyu Cao, Cheng Zhang, Pedro Gimenes, Jianqiao Lu, Jianyi Cheng, Yiren Zhao |<img width="1002" alt="image" src="figures/LLM-MPQ.png"> |[Paper](https://arxiv.org/abs/2410.06722)|[//]: #10/14
|[![Star](https://img.shields.io/github/stars/ChenMnZ/PrefixQuant.svg?style=social&label=Star)](https://github.com/ChenMnZ/PrefixQuant)<br>[PrefixQuant: Static Quantization Beats Dynamic through Prefixed Outliers in LLMs](https://arxiv.org/abs/2410.05265) <br> Mengzhao Chen, Yi Liu, Jiahao Wang, Yi Bin, Wenqi Shao, Ping Luo |<img width="1002" alt="image" src="https://arxiv.org/html/2410.05265v1/x1.png"> |[Github](https://github.com/ChenMnZ/PrefixQuant) <br> [Paper](https://arxiv.org/abs/2410.05265)|[//]: #10/14
|[![Star](https://img.shields.io/github/stars/Anonymous1252022/EXAQ.svg?style=social&label=Star)](https://github.com/Anonymous1252022/EXAQ)<br>[EXAQ: Exponent Aware Quantization For LLMs Acceleration](https://arxiv.org/abs/2410.03185) <br> Moran Shkolnik, Maxim Fishman, Brian Chmiel, Hilla Ben-Yaacov, Ron Banner, Kfir Yehuda Levy |<img width="1002" alt="image" src="figures/EXAQ.png"> |[Github](https://github.com/Anonymous1252022/EXAQ) <br> [Paper](https://arxiv.org/abs/2410.03185)|[//]: #10/14
|[![Star](https://img.shields.io/github/stars/enyac-group/Quamba.svg?style=social&label=Star)](https://github.com/enyac-group/Quamba)<br>[Quamba: A Post-Training Quantization Recipe for Selective State Space Models](https://arxiv.org/abs/2410.13229) <br> Hung-Yueh Chiang, Chi-Chih Chang, Natalia Frumkin, Kai-Chiang Wu, Diana Marculescu |<img width="1002" alt="image" src="https://arxiv.org/html/2410.13229v1/extracted/5933363/figures/outliers.png"> |[Github](https://github.com/enyac-group/Quamba) <br> [Paper](https://arxiv.org/abs/2410.13229)|[//]: #10/21
|[AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurations](https://arxiv.org/abs/2410.13212) <br> Qian Tao, Wenyuan Yu, Jingren Zhou |<img width="1002" alt="image" src="https://arxiv.org/html/2410.13212v1/extracted/5933292/figures/kvmix.png"> |[Paper](https://arxiv.org/abs/2410.13212)|[//]: #10/21
|[Channel-Wise Mixed-Precision Quantization for Large Language Models](https://arxiv.org/abs/2410.13056) <br> Zihan Chen, Bike Xie, Jundong Li, Cong Shen |<img width="1002" alt="image" src="https://arxiv.org/html/2410.13056v1/x1.png"> |[Paper](https://arxiv.org/abs/2410.13056)|[//]: #10/21
|[Progressive Mixed-Precision Decoding for Efficient LLM Inference](https://arxiv.org/abs/2410.13461) <br> Hao Mark Chen, Fuwen Tan, Alexandros Kouris, Royson Lee, Hongxiang Fan, Stylianos I. Venieris |<img width="1002" alt="image" src="https://arxiv.org/html/2410.13461v1/x4.png"> |[Paper](https://arxiv.org/abs/2410.13461)|[//]: #10/21
|[![Star](https://img.shields.io/github/stars/LuoYingSong/DAQ.svg?style=social&label=Star)](https://github.com/LuoYingSong/DAQ)<br>[DAQ: Density-Aware Post-Training Weight-Only Quantization For LLMs](https://arxiv.org/abs/2410.12187) <br> Yingsong Luo, Ling Chen |<img width="1002" alt="image" src="https://arxiv.org/html/2410.12187v2/x1.png"> |[Github](https://github.com/LuoYingSong/DAQ) <br> [Paper](https://arxiv.org/abs/2410.12187)|[//]: #10/21
|[Continuous Approximations for Improving Quantization Aware Training of LLMs](https://arxiv.org/abs/2410.10849) <br> He Li, Jianhang Hong, Yuanzhuo Wu, Snehal Adbol, Zonglin Li | |[Paper](https://arxiv.org/abs/2410.10849)|[//]: #10/21
|[Scaling laws for post-training quantized large language models](https://arxiv.org/abs/2410.12119) <br> Zifei Xu, Alexander Lan, Wanzin Yazar, Tristan Webb, Sayeh Sharify, Xin Wang |<img width="202" alt="image" src="https://arxiv.org/html/2410.12119v1/extracted/5929616/figures/fig_12.png"> |[Paper](https://arxiv.org/abs/2410.12119)|[//]: #10/21
|[![Star](https://img.shields.io/github/stars/Mohammad-Mozaffari/slim.svg?style=social&label=Star)](https://github.com/Mohammad-Mozaffari/slim)<br>[SLiM: One-shot Quantized Sparse Plus Low-rank Approximation of LLMs](https://arxiv.org/abs/2410.09615) <br> Mohammad Mozaffari, Maryam Mehri Dehnavi |<img width="1002" alt="image" src="https://arxiv.org/html/2410.09615v1/x1.png"> |[Github](https://github.com/Mohammad-Mozaffari/slim) <br> [Paper](https://arxiv.org/abs/2410.09615)|[//]: #10/21
|[SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators](https://arxiv.org/abs/2410.10714) <br> Rasoul Shafipour, David Harrison, Maxwell Horton, Jeffrey Marker, Houman Bedayat, Sachin Mehta, Mohammad Rastegari, Mahyar Najibi, Saman Naderiparizi |<img width="1002" alt="image" src="https://arxiv.org/html/2410.10714v2/x1.png"> |[Paper](https://arxiv.org/abs/2410.10714)|[//]: #10/21
|[![Star](https://img.shields.io/github/stars/ruikangliu/FlatQuant.svg?style=social&label=Star)](https://github.com/ruikangliu/FlatQuant)<br>[FlatQuant: Flatness Matters for LLM Quantization](https://arxiv.org/abs/2410.09426) <br> Yuxuan Sun, Ruikang Liu, Haoli Bai, Han Bao, Kang Zhao, Yuening Li, Jiaxin Hu, Xianzhi Yu, Lu Hou, Chun Yuan, Xin Jiang, Wulong Liu, Jun Yao |<img width="1002" alt="image" src="https://arxiv.org/html/2410.09426v1/x11.png"> |[Github](https://github.com/ruikangliu/FlatQuant) <br> [Paper](https://arxiv.org/abs/2410.09426)|[//]: #10/21
|[Pyramid Vector Quantization for LLMs](https://arxiv.org/abs/2410.16926) <br> Tycho F. A. van der Ouderaa, Maximilian L. Croci, Agrin Hilmkil, James Hensman |<img width="1002" alt="image" src="https://arxiv.org/html/2410.16926v1/x1.png"> |[Paper](https://arxiv.org/abs/2410.16926)|[//]: #10/29
|[Evaluating Quantized Large Language Models for Code Generation on Low-Resource Language Benchmarks](https://arxiv.org/abs/2410.14766) <br> Enkhbold Nyamsuren | |[Paper](https://arxiv.org/abs/2410.14766)|[//]: #10/30
|[QuAILoRA: Quantization-Aware Initialization for LoRA](https://arxiv.org/abs/2410.14713) <br> Neal Lawton, Aishwarya Padmakumar, Judith Gaspers, Jack FitzGerald, Anoop Kumar, Greg Ver Steeg, Aram Galstyan | |[Paper](https://arxiv.org/abs/2410.14713)|[//]: #10/30
|[![Star](https://img.shields.io/github/stars/microsoft/BitNet.svg?style=social&label=Star)](https://github.com/microsoft/BitNet)<br>[1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs](https://arxiv.org/abs/2410.16144) <br> Jinheng Wang, Hansong Zhou, Ting Song, Shaoguang Mao, Shuming Ma, Hongyu Wang, Yan Xia, Furu Wei |<img width="1002" alt="image" src="https://arxiv.org/html/2410.16144v2/x1.png"> |[Github](https://github.com/microsoft/BitNet) <br> [Paper](https://arxiv.org/abs/2410.16144)|[//]: #10/30
|[Understanding the difficulty of low-precision post-training quantization of large language models](https://arxiv.org/abs/2410.14570) <br> Zifei Xu, Sayeh Sharify, Wanzin Yazar, Tristan Webb, Xin Wang |<img width="1002" alt="image" src="https://arxiv.org/html/2410.14570v1/extracted/5935973/figures/fig1.png"> |[Paper](https://arxiv.org/abs/2410.14570)|[//]: #10/30
|[The Impact of Inference Acceleration Strategies on Bias of LLMs](https://arxiv.org/abs/2410.22118) <br> Elisabeth Kirsten, Ivan Habernal, Vedant Nanda, Muhammad Bilal Zafar | |[Paper](https://arxiv.org/abs/2410.22118)|[//]: #11/17
|[![Star](https://img.shields.io/github/stars/xinghaow99/BitStack.svg?style=social&label=Star)](https://github.com/xinghaow99/BitStack)<br>[BitStack: Fine-Grained Size Control for Compressed Large Language Models in Variable Memory Environments](https://arxiv.org/abs/2410.23918) <br> Xinghao Wang, Pengyu Wang, Bo Wang, Dong Zhang, Yunhua Zhou, Xipeng Qiu |<img width="1002" alt="image" src="https://github.com/xinghaow99/BitStack/raw/main/assets/bitstack.png"> |[Github](https://github.com/xinghaow99/BitStack) <br> [Paper](https://arxiv.org/abs/2410.23918)|[//]: #11/17
|[![Star](https://img.shields.io/github/stars/Intelligent-Computing-Lab-Yale/TesseraQ.svg?style=social&label=Star)](https://github.com/Intelligent-Computing-Lab-Yale/TesseraQ)<br>[TesseraQ: Ultra Low-Bit LLM Post-Training Quantization with Block Reconstruction](https://arxiv.org/abs/2410.19103) <br> Yuhang Li, Priyadarshini Panda |<img width="1002" alt="image" src="https://github.com/Intelligent-Computing-Lab-Yale/TesseraQ/raw/main/imgs/tesseraq.png"> |[Github](https://github.com/Intelligent-Computing-Lab-Yale/TesseraQ) <br> [Paper](https://arxiv.org/abs/2410.19103)|[//]: #11/17
|[BitNet a4.8: 4-bit Activations for 1-bit LLMs](https://arxiv.org/abs/2411.04965) <br> Hongyu Wang, Shuming Ma, Furu Wei |<img width="1002" alt="image" src="https://arxiv.org/html/2411.04965v1/x1.png"> |[Paper](https://arxiv.org/abs/2411.04965)|[//]: #11/18
|["Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization](https://arxiv.org/abs/2411.02355) <br> Eldar Kurtic, Alexandre Marques, Shubhra Pandit, Mark Kurtz, Dan Alistarh | |[Paper](https://arxiv.org/abs/2411.02355)|[//]: #11/18
|[GWQ: Gradient-Aware Weight Quantization for Large Language Models](https://arxiv.org/abs/2411.00850) <br> Yihua Shao, Siyu Liang, Xiaolin Lin, Zijian Ling, Zixian Zhu et al  |<img width="1002" alt="image" src="https://arxiv.org/html/2411.00850v1/x2.png"> |[Paper](https://arxiv.org/abs/2411.00850)|[//]: #11/18
|[A Comprehensive Study on Quantization Techniques for Large Language Models](https://arxiv.org/abs/2411.02530) <br> Jiedong Lang, Zhehao Guo, Shuyu Huang | |[Paper](https://arxiv.org/abs/2411.02530)|[//]: #11/18
|[Bi-Mamba: Towards Accurate 1-Bit State Space Models](https://arxiv.org/abs/2411.11843) <br> Shengkun Tang, Liqun Ma, Haonan Li, Mingjie Sun, Zhiqiang Shen |<img width="1002" alt="image" src="https://arxiv.org/html/2411.11843v1/x2.png"> |[Paper](https://arxiv.org/abs/2411.11843)|[//]: #11/24
|[AMXFP4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM Inference](https://arxiv.org/abs/2411.09909) <br> Janghwan Lee, Jiwoong Park, Jinseok Kim, Yongjik Kim, Jungju Oh, Jinwook Oh, Jungwook Choi |<img width="1002" alt="image" src="figures/AMXFP4.png"> |[Paper](https://arxiv.org/abs/2411.09909)|[//]: #11/24
|[![Star](https://img.shields.io/github/stars/abdelfattah-lab/BitMoD-HPCA-25.svg?style=social&label=Star)](https://github.com/abdelfattah-lab/BitMoD-HPCA-25)[![Publish](https://img.shields.io/badge/Conference-HPCA'25-blue)]()<br>[BitMoD: Bit-serial Mixture-of-Datatype LLM Acceleration](https://arxiv.org/abs/2411.11745) <br> Yuzong Chen, Ahmed F. AbouElhamayed, Xilai Dai, Yang Wang, Marta Andronic, George A. Constantinides, Mohamed S. Abdelfattah |<img width="1002" alt="image" src="https://arxiv.org/html/2411.11745v1/x5.png"> |[Github](https://github.com/abdelfattah-lab/BitMoD-HPCA-25) <br> [Paper](https://arxiv.org/abs/2411.11745)|[//]: #11/24
|[MixPE: Quantization and Hardware Co-design for Efficient LLM Inference](https://arxiv.org/abs/2411.16158) <br> Yu Zhang, Mingzi Wang, Lancheng Zou, Wulong Liu, Hui-Ling Zhen, Mingxuan Yuan, Bei Yu |<img width="1002" alt="image" src="https://arxiv.org/html/2411.16158v1/x5.png"> |[Paper](https://arxiv.org/abs/2411.16158)|[//]: #12/03
|[![Publish](https://img.shields.io/badge/Conference-HPCA'25-blue)]()<br>[Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format](https://arxiv.org/abs/2411.15982) <br> Chao Fang, Man Shi, Robin Geens, Arne Symons, Zhongfeng Wang, Marian Verhelst |<img width="1002" alt="image" src="https://arxiv.org/html/2411.15982v1/x1.png"> |[Paper](https://arxiv.org/abs/2411.15982)|[//]: #12/03
|[SKIM: Any-bit Quantization Pushing The Limits of Post-Training Quantization](https://arxiv.org/abs/2412.04180) <br> Runsheng Bai, Qiang Liu, Bo Liu |<img width="1002" alt="image" src="https://arxiv.org/html/2412.04180v1/x2.png"> |[Paper](https://arxiv.org/abs/2412.04180)|[//]: #12/09
|[CPTQuant -- A Novel Mixed Precision Post-Training Quantization Techniques for Large Language Models](https://arxiv.org/abs/2412.03599) <br> Amitash Nanda, Sree Bhargavi Balija, Debashis Sahoo |<img width="1002" alt="image" src="https://arxiv.org/html/2412.03599v1/x3.png"> |[Paper](https://arxiv.org/abs/2412.03599)|[//]: #12/09
|[LSAQ: Layer-Specific Adaptive Quantization for Large Language Model Deployment](https://arxiv.org/abs/2412.18135) <br> Binrui Zeng, Bin Ji, Xiaodong Liu, Jie Yu, Shasha Li, Jun Ma, Xiaopeng Li, Shangwen Wang, Xinran Hong |<img width="1002" alt="image" src="https://arxiv.org/html/2412.18135v1/x1.png"> |[Paper](https://arxiv.org/abs/2412.18135)|[//]: #12/30
|[GQSA: Group Quantization and Sparsity for Accelerating Large Language Model Inference](https://arxiv.org/abs/2412.17560) <br> Chao Zeng, Songwei Liu, Shu Yang, Fangmin Chen, Xing Mei, Lean Fu |<img width="1002" alt="image" src="https://arxiv.org/html/2412.17560v1/extracted/6090667/GQS_block.png"> |[Paper](https://arxiv.org/abs/2412.17560)|[//]: #12/30
|[MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Design](https://arxiv.org/abs/2412.14590) <br> Zhen Zheng, Xiaonan Song, Chuanjie Liu |<img width="1002" alt="image" src="https://arxiv.org/html/2412.14590v1/x1.png"> |[Paper](https://arxiv.org/abs/2412.14590)|[//]: #12/30
|[![Star](https://img.shields.io/github/stars/utkarsh-dmx/project-resq.svg?style=social&label=Star)](https://github.com/utkarsh-dmx/project-resq)<br>[ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals](https://arxiv.org/abs/2412.14363) <br> Utkarsh Saxena, Sayeh Sharify, Kaushik Roy, Xin Wang |<img width="1002" alt="image" src="figures/ResQ.png"> |[Github](https://github.com/utkarsh-dmx/project-resq) <br> [Paper](https://arxiv.org/abs/2412.14363)|[//]: #12/30