
## KV Cache Compression
| Title & Authors | Introduction | Links |
|:--|  :----: | :---:|
|[![Star](https://img.shields.io/github/stars/KVQuant/.svg?style=social&label=Star)](https://github.com/KVQuant/)<br>[KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://arxiv.org/abs/2401.18079) <br> Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, Amir Gholami |<img width="1002" alt="image" src="figures/KVQuant.png"> |[Github](https://github.com/SqueezeAILab/KVQuant/) <br> [Paper](https://arxiv.org/abs/2401.18079)|
|[WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More](https://arxiv.org/abs/2402.12065) <br> Yuxuan Yue, Zhihang Yuan, Haojie Duanmu, Sifan Zhou, Jianlong Wu, Liqiang Nie |<img width="302" alt="image" src="figures/WKVQuant.png"> |[Paper](https://arxiv.org/abs/2402.12065)|
|[DB-LLM: Accurate Dual-Binarization for Efficient LLMs](https://arxiv.org/abs/2402.11960) <br> Hong Chen, Chengtao Lv, Liang Ding, Haotong Qin, Xiabin Zhou, Yifu Ding, Xuebo Liu, Min Zhang, Jinyang Guo, Xianglong Liu, Dacheng Tao |<img width="1002" alt="image" src="figures/DB-LLM.png"> |[Paper](https://arxiv.org/abs/2402.11960)|
|[No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization](https://arxiv.org/abs/2402.18096) <br> June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, Dongsoo Lee |<img width="302" alt="image" src="https://arxiv.org/html/2402.18096v1/x1.png"> |[Paper](https://arxiv.org/abs/2402.18096)|
|[![Star](https://img.shields.io/github/stars/ClubieDong/QAQ-KVCacheQuantization.svg?style=social&label=Star)](https://github.com/ClubieDong/QAQ-KVCacheQuantization)<br>[QAQ: Quality Adaptive Quantization for LLM KV Cache](https://arxiv.org/abs/2403.04643) <br> Shichen Dong, Wen Cheng, Jiayu Qin, Wei Wang |<img width="1002" alt="image" src="https://arxiv.org/html/2403.04643v1/x1.png"> |[Github](https://github.com/ClubieDong/QAQ-KVCacheQuantization) <br> [Paper](https://arxiv.org/abs/2403.04643)|
|[![Star](https://img.shields.io/github/stars/cat538/SKVQ.svg?style=social&label=Star)](https://github.com/cat538/SKVQ)<br>[SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models](https://arxiv.org/abs/2405.06219) <br> Haojie Duanmu, Zhihang Yuan, Xiuhong Li, Jiangfei Duan, Xingcheng Zhang, Dahua Lin |<img width="1002" alt="image" src="figures/SKVQ.png"> |[Github](https://github.com/cat538/SKVQ) <br> [Paper](https://arxiv.org/abs/2405.06219)|
|[![Publish](https://img.shields.io/badge/Conference-NeurIPS'23-blue)]()<br>[Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time](https://arxiv.org/abs/2305.17118) <br> Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, Anshumali Shrivastava |<img width="302" alt="image" src="figures/Scissorhands.png"> |[Paper](https://arxiv.org/abs/2305.17118)|
|[Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs](https://arxiv.org/abs/2310.01801) <br> Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, Jianfeng Gao |<img width="1002" alt="image" src="figures/FastGen.png"> |[Paper](https://arxiv.org/abs/2310.01801)|
|[ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition](https://arxiv.org/abs/2402.15220) <br> Lu Ye, Ze Tao, Yong Huang, Yang Li |<img width="1002" alt="image" src="figures/ChunkAttention.png"> |[Paper](https://arxiv.org/abs/2402.15220)|
|[![Star](https://img.shields.io/github/stars/HaoKang-Timmy/GEAR.svg?style=social&label=Star)](https://github.com/HaoKang-Timmy/GEAR)<br>[GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM](https://arxiv.org/abs/2403.05527) <br> Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, Tuo Zhao |<img width="1002" alt="image" src="https://github.com/HaoKang-Timmy/GEAR/raw/main/Fig/overview.png"> |[Github](https://github.com/HaoKang-Timmy/GEAR) <br> [Paper](https://arxiv.org/abs/2403.05527)|
|[Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference](https://arxiv.org/abs/2403.09054) <br> Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant J. Nair, Ilya Soloveychik, Purushotham Kamath |<img width="1002" alt="image" src="https://arxiv.org/html/2403.09054v1/x2.png"> |[Paper](https://arxiv.org/abs/2403.09054)|
|[ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching](https://arxiv.org/abs/2403.17312) <br> Youpeng Zhao, Di Wu, Jun Wang |<img width="1002" alt="image" src="https://arxiv.org/html/2403.17312v1/extracted/5495383/imgs/background_imgs/figure2_revised.png"> |[Paper](https://arxiv.org/abs/2403.17312)|
|[![Star](https://img.shields.io/github/stars/hdong920/LESS.svg?style=social&label=Star)](https://github.com/hdong920/LESS)<br>[Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference](https://arxiv.org/abs/2402.09398) <br> Harry Dong, Xinyu Yang, Zhenyu Zhang, Zhangyang Wang, Yuejie Chi, Beidi Chen |<img width="1002" alt="image" src="figures/LESS.png"> |[Github](https://github.com/hdong920/LESS) <br> [Paper](https://arxiv.org/abs/2402.09398)|
|[MiniCache: KV Cache Compression in Depth Dimension for Large Language Models](https://arxiv.org/abs/2405.14366) <br> Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, Bohan Zhuang |<img width="1002" alt="image" src="figures/minicache.png"> |[Paper](https://arxiv.org/abs/2405.14366)|
|[![Star](https://img.shields.io/github/stars/lpyhdzx/DecoQuant_code.svg?style=social&label=Star)](https://github.com/lpyhdzx/DecoQuant_code)<br>[Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression](https://arxiv.org/abs/2405.12591) <br> Peiyu Liu, Ze-Feng Gao, Wayne Xin Zhao, Yipeng Ma, Tao Wang, Ji-Rong Wen |<img width="1002" alt="image" src="figures/DecoQuant.png"> |[Github](https://github.com/lpyhdzx/DecoQuant_code) <br> [Paper](https://arxiv.org/abs/2405.12591)|
|[![Star](https://img.shields.io/github/stars/mutonix/pyramidinfer.svg?style=social&label=Star)](https://github.com/mutonix/pyramidinfer)[![Publish](https://img.shields.io/badge/Conference-ACL'24-blue)]()<br>[PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference](https://arxiv.org/abs/2405.12532) <br> Dongjie Yang, XiaoDong Han, Yan Gao, Yao Hu, Shilin Zhang, Hai Zhao |<img width="1002" alt="image" src="figures/PyramidInfer.png"> |[Github](https://github.com/mutonix/pyramidinfer) <br> [Paper](https://arxiv.org/abs/2405.12532)|
|[Reducing Transformer Key-Value Cache Size with Cross-Layer Attention](https://arxiv.org/abs/2405.12981) <br> William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, Jonathan Ragan Kelly |<img width="1002" alt="image" src="figures/CLA.png"> |[Paper](https://arxiv.org/abs/2405.12981)|
|[![Star](https://img.shields.io/github/stars/whyNLP/LCKV.svg?style=social&label=Star)](https://github.com/whyNLP/LCKV)[![Publish](https://img.shields.io/badge/Conference-ACL'24-blue)]()<br>[Layer-Condensed KV Cache for Efficient Inference of Large Language Models](https://arxiv.org/abs/2405.10637) <br> Haoyi Wu, Kewei Tu |<img width="1002" alt="image" src="figures/LCKV.png"> |[Github](https://github.com/whyNLP/LCKV) <br> [Paper](https://arxiv.org/abs/2405.10637)|
|[ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification](https://arxiv.org/abs/2405.14256) <br> Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, Bohan Zhuang |<img width="1002" alt="image" src="figures/zipcache.png"> |[Paper](https://arxiv.org/abs/2405.14256)|
|[![Star](https://img.shields.io/github/stars/amirzandieh/QJL.svg?style=social&label=Star)](https://github.com/amirzandieh/QJL)<br>[QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead](https://arxiv.org/abs/2406.03482) <br> Amir Zandieh, Majid Daliri, Insu Han |<img width="1002" alt="image" src="figures/QJL.png"> |[Github](https://github.com/amirzandieh/QJL) <br> [Paper](https://arxiv.org/abs/2406.03482)|[//]: #06/11
|[Loki: Low-Rank Keys for Efficient Sparse Attention](https://arxiv.org/abs/2406.02542) <br> Prajwal Singhania, Siddharth Singh, Shwai He, Soheil Feizi, Abhinav Bhatele |<img width="1002" alt="image" src="https://arxiv.org/html/2406.02542v1/x2.png"> |[Paper](https://arxiv.org/abs/2406.02542)|[//]: #06/12
|[![Star](https://img.shields.io/github/stars/zaydzuhri/pythia-mlkv.svg?style=social&label=Star)](https://github.com/zaydzuhri/pythia-mlkv)<br>[MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding](https://arxiv.org/abs/2406.09297) <br> Zayd Muhammad Kawakibi Zuhri, Muhammad Farid Adilazuarda, Ayu Purwarianti, Alham Fikri Aji |<img width="1002" alt="image" src="https://arxiv.org/html/2406.09297v1/extracted/5665367/resources/mlkv-All_KV.png"> |[Github](https://github.com/zaydzuhri/pythia-mlkv) <br> [Paper](https://arxiv.org/abs/2406.09297)|[//]: #06/18
|[![Star](https://img.shields.io/github/stars/henryzhongsc/longctx_bench.svg?style=social&label=Star)](https://github.com/henryzhongsc/longctx_bench)<br>[KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches](https://arxiv.org/abs/2407.01527) <br> Jiayi Yuan, Hongyi Liu, Shaochen (Henry)Zhong, Yu-Neng Chuang, Songchen Li, Guanchu Wang, Duy Le, Hongye Jin, Vipin Chaudhary, Zhaozhuo Xu, Zirui Liu, Xia Hu |<img width="1002" alt="image" src="figures/longctx_bench.png"> |[Github](https://github.com/henryzhongsc/longctx_bench) <br> [Paper](https://arxiv.org/abs/2407.01527)|[//]: #07/03
|[![Star](https://img.shields.io/github/stars/WHUIR/ADORE.svg?style=social&label=Star)](https://github.com/WHUIR/ADORE)[![Publish](https://img.shields.io/badge/Conference-ACL'24%20Findings-blue)]()<br>[Efficient Sparse Attention needs Adaptive Token Release](https://arxiv.org/abs/2407.02328) <br> Chaoran Zhang, Lixin Zou, Dan Luo, Min Tang, Xiangyang Luo, Zihao Li, Chenliang Li |<img width="1002" alt="image" src="https://arxiv.org/html/2407.02328v1/x1.png"> |[Github](https://github.com/WHUIR/ADORE) <br> [Paper](https://arxiv.org/abs/2407.02328)|[//]: #07/05
|[![Star](https://img.shields.io/github/stars/recursal/GoldFinch-paper.svg?style=social&label=Star)](https://github.com/recursal/GoldFinch-paper)<br>[GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill and Extreme KV-Cache Compression](https://arxiv.org/abs/2407.12077) <br> Daniel Goldstein, Fares Obeid, Eric Alcaide, Guangyu Song, Eugene Cheah |<img width="202" alt="image" src="https://github.com/recursal/GoldFinch-paper/raw/main/assets/architecture.png"> |[Github](https://github.com/recursal/GoldFinch-paper) <br> [Paper](https://arxiv.org/abs/2407.12077)|[//]: #07/21
|[PQCache: Product Quantization-based KVCache for Long Context LLM Inference](https://arxiv.org/abs/2407.12820) <br> Hailin Zhang, Xiaodong Ji, Yilin Chen, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, Weipeng Chen, Bin Cui |<img width="1002" alt="image" src="https://arxiv.org/html/2407.12820v1/extracted/5702744/Figures/transformer.png"> |[Paper](https://arxiv.org/abs/2407.12820)|[//]: #07/21
|[RazorAttention: Efficient KV Cache Compression Through Retrieval Heads](https://arxiv.org/abs/2407.15891) <br> Hanlin Tang, Yang Lin, Jing Lin, Qingsen Han, Shikuan Hong, Yiwu Yao, Gongyi Wang |<img width="1002" alt="image" src="https://arxiv.org/html/2407.15891v1/x3.png"> |[Paper](https://arxiv.org/abs/2407.15891)|[//]: #07/24
|[ThinK: Thinner Key Cache by Query-Driven Pruning](https://arxiv.org/abs/2407.21018) <br> Yuhui Xu, Zhanming Jie, Hanze Dong, Lei Wang, Xudong Lu, Aojun Zhou, Amrita Saha, Caiming Xiong, Doyen Sahoo |<img width="1002" alt="image" src="https://arxiv.org/html/2407.21018v1/x1.png"> |[Paper](https://arxiv.org/abs/2407.21018)|[//]: #08/08
|[![Star](https://img.shields.io/github/stars/shadowpa0327/Palu.svg?style=social&label=Star)](https://github.com/shadowpa0327/Palu)<br>[Palu: Compressing KV-Cache with Low-Rank Projection](https://arxiv.org/abs/2407.21118) <br> Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Kai-Chiang Wu |<img width="1002" alt="image" src="https://github.com/shadowpa0327/Palu/blob/master/img/palu_idea.png"> |[Github](https://github.com/shadowpa0327/Palu) <br> [Paper](https://arxiv.org/abs/2407.21118)|[//]: #08/08
|[Finch: Prompt-guided Key-Value Cache Compression](https://arxiv.org/abs/2408.00167) <br> Giulio Corallo, Paolo Papotti |<img width="1002" alt="image" src="https://arxiv.org/html/2408.00167v1/extracted/5763688/assets/diagram_finch.png"> |[Paper](https://arxiv.org/abs/2408.00167)|[//]: #08/08
|[Zero-Delay QKV Compression for Mitigating KV Cache and Network Bottlenecks in LLM Inference](https://arxiv.org/abs/2408.04107) <br> Zeyu Zhang,Haiying Shen |<img width="1002" alt="image" src="https://arxiv.org/html/2408.04107v1/x15.png"> |[Paper](https://arxiv.org/abs/2408.04107)|[//]: #08/09
|[![Star](https://img.shields.io/github/stars/UtkarshSaxena1/EigenAttn.svg?style=social&label=Star)](https://github.com/UtkarshSaxena1/EigenAttn)<br>[Eigen Attention: Attention in Low-Rank Space for KV Cache Compression](https://arxiv.org/abs/2408.05646) <br> Utkarsh Saxena, Gobinda Saha, Sakshi Choudhary, Kaushik Roy |<img width="1002" alt="image" src="https://arxiv.org/html/2408.05646v1/x1.png"> |[Github](https://github.com/UtkarshSaxena1/EigenAttn) <br> [Paper](https://arxiv.org/abs/2408.05646)|[//]: #08/13
|[![Star](https://img.shields.io/github/stars/andy-yang-1/DoubleSparse.svg?style=social&label=Star)](https://github.com/andy-yang-1/DoubleSparse)<br>[Post-Training Sparse Attention with Double Sparsity](https://arxiv.org/abs/2408.07092) <br> Shuo Yang, Ying Sheng, Joseph E. Gonzalez, Ion Stoica, Lianmin Zheng |<img width="302" alt="image" src="https://github.com/andy-yang-1/DoubleSparse/raw/main/assets/double-sparsity-gif-v2.gif"> |[Github](https://github.com/andy-yang-1/DoubleSparse) <br> [Paper](https://arxiv.org/abs/2408.07092)|[//]: #08/20
|[A First Look At Efficient And Secure On-Device LLM Inference Against KV Leakage](https://arxiv.org/abs/2409.04040) <br> Huan Yang, Deyu Zhang, Yudong Zhao, Yuanchun Li, Yunxin Liu |<img width="1002" alt="image" src="https://arxiv.org/html/2409.04040v1/x3.png"> |[Paper](https://arxiv.org/abs/2409.04040)|[//]: #09/13
|[CSKV: Training-Efficient Channel Shrinking for KV Cache in Long-Context Scenarios](https://arxiv.org/abs/2409.10593) <br> Luning Wang, Shiyao Li, Xuefei Ning, Zhihang Yuan, Shengen Yan, Guohao Dai, Yu Wang |<img width="1002" alt="image" src="https://arxiv.org/html/2409.10593v1/x1.png"> |[Paper](https://arxiv.org/abs/2409.10593)|[//]: #09/21
|[![Star](https://img.shields.io/github/stars/AlignedQuant/AlignedKV.svg?style=social&label=Star)](https://github.com/AlignedQuant/AlignedKV)<br>[AlignedKV: Reducing Memory Access of KV-Cache with Precision-Aligned Quantization](https://arxiv.org/abs/2409.16546) <br> Yifan Tan, Haoze Wang, Chao Yan, Yangdong Deng |<img width="1002" alt="image" src="https://arxiv.org/html/2409.16546v1/extracted/5867591/Figure6.png"> |[Github](https://github.com/AlignedQuant/AlignedKV) <br> [Paper](https://arxiv.org/abs/2409.16546)|[//]: #09/27
|[KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head](https://arxiv.org/abs/2410.00161) <br> Isaac Rehg |<img width="1002" alt="image" src="https://arxiv.org/html/2410.00161v1/x5.png"> |[Paper](https://arxiv.org/abs/2410.00161)|[//]: #10/02
|[![Publish](https://img.shields.io/badge/Conference-ICML'24-blue)]()<br>[Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference](https://arxiv.org/abs/2403.09636) <br> Piotr Nawrot, Adrian Łańcucki, Marcin Chochowski, David Tarjan, Edoardo M. Ponti |<img width="1002" alt="image" src="figures/DMC.png"> |[Paper](https://arxiv.org/abs/2403.09636)|[//]: #10/02
|[![Star](https://img.shields.io/github/stars/FFY0/AdaKV.svg?style=social&label=Star)](https://github.com/FFY0/AdaKV)<br>[Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference](https://arxiv.org/abs/2407.11550) <br> Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, S. Kevin Zhou |<img width="1002" alt="image" src="figures/adakv.png"> |[Github](https://github.com/FFY0/AdaKV) <br> [Paper](https://arxiv.org/abs/2407.11550)|[//]: #10/13
|[SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformation](https://arxiv.org/abs/2410.03960) <br> Aurick Qiao, Zhewei Yao, Samyam Rajbhandari, Yuxiong He |<img width="1002" alt="image" src="https://arxiv.org/html/2410.03960v1/x1.png"> |[Paper](https://arxiv.org/abs/2410.03960)|[//]: #10/14
|[LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy](https://arxiv.org/abs/2410.03111) <br> Rongzhi Zhang, Kuang Wang, Liyuan Liu, Shuohang Wang, Hao Cheng, Chao Zhang, Yelong Shen |<img width="1002" alt="image" src="figures/LoRC.png"> |[Paper](https://arxiv.org/abs/2410.03111)|[//]: #10/14
|[![Star](https://img.shields.io/github/stars/LMCache/LMCache.svg?style=social&label=Star)](https://github.com/LMCache/LMCache)[![Publish](https://img.shields.io/badge/Conference-SIGCOMM'24-red)]()<br>[CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving](https://arxiv.org/abs/2310.07240) <br> Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, Junchen Jiang |<img width="1002" alt="image" src="figures/cachegen-snapshot.png"> |[Github](https://github.com/LMCache/LMCache) <br> [Paper](https://arxiv.org/abs/2310.07240)|[//]: #10/23
|[![Star](https://img.shields.io/github/stars/yangyifei729/KVSharer.svg?style=social&label=Star)](https://github.com/yangyifei729/KVSharer)<br>[KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing](https://arxiv.org/abs/2410.18517) <br> Yifei Yang, Zouying Cao, Qiguang Chen, Libo Qin, Dongjie Yang, Hai Zhao, Zhi Chen |<img width="1002" alt="image" src="https://github.com/yangyifei729/KVSharer/raw/main/img/main_fig.jpg"> |[Github](https://github.com/yangyifei729/KVSharer) <br> [Paper](https://arxiv.org/abs/2410.18517)|[//]: #10/29
|[![Star](https://img.shields.io/github/stars/iankur/vqllm.svg?style=social&label=Star)](https://github.com/iankur/vqllm)<br>[Residual vector quantization for KV cache compression in large language model](https://arxiv.org/abs/2410.15704) <br> Ankur Kumar | |[Github](https://github.com/iankur/vqllm) <br> [Paper](https://arxiv.org/abs/2410.15704)|[//]: #10/30
|[MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection](https://arxiv.org/abs/2410.14731) <br> Bokai Lin, Zihao Zeng, Zipeng Xiao, Siqi Kou, Tianqi Hou, Xiaofeng Gao, Hao Zhang, Zhijie Deng |<img width="1002" alt="image" src="https://arxiv.org/html/2410.14731v1/x2.png"> |[Paper](https://arxiv.org/abs/2410.14731)|[//]: #10/30
|[Lossless KV Cache Compression to 2%](https://arxiv.org/abs/2410.15252) <br> Zhen Yang, J.N.Han, Kan Wu, Ruobing Xie, An Wang, Xingwu Sun, Zhanhui Kang |<img width="1002" alt="image" src="https://arxiv.org/html/2410.15252v1/extracted/5937225/images/CLLA_Overview.png"> |[Paper](https://arxiv.org/abs/2410.15252)|[//]: #10/30
|[![Star](https://img.shields.io/github/stars/whyNLP/LCKV.svg?style=social&label=Star)](https://github.com/whyNLP/LCKV)<br>[A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference](https://arxiv.org/abs/2410.14442) <br> You Wu, Haoyi Wu, Kewei Tu |<img width="1002" alt="image" src="figures/cross-layer-kv.png"> |[Github](https://github.com/whyNLP/LCKV) <br> [Paper](https://arxiv.org/abs/2410.14442)|[//]: #10/30
|[![Star](https://img.shields.io/github/stars/JunqiZhao888/buzz-llm.svg?style=social&label=Star)](https://github.com/JunqiZhao888/buzz-llm)<br>[BUZZ: Beehive-structured Sparse KV Cache with Segmented Heavy Hitters for Efficient LLM Inference](https://arxiv.org/abs/2410.23079) <br> Junqi Zhao, Zhijin Fang, Shu Li, Shaohui Yang, Shichao He |<img width="1002" alt="image" src="https://arxiv.org/html/2410.23079v1/x1.png"> |[Github](https://github.com/JunqiZhao888/buzz-llm) <br> [Paper](https://arxiv.org/abs/2410.23079)|[//]: #11/17
|[![Star](https://img.shields.io/github/stars/FYYFU/HeadKV.svg?style=social&label=Star)](https://github.com/FYYFU/HeadKV)<br>[Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning](https://arxiv.org/abs/2410.19258) <br> Yu Fu, Zefan Cai, Abedelkadir Asi, Wayne Xiong, Yue Dong, Wen Xiao |<img width="1002" alt="image" src="https://github.com/FYYFU/HeadKV/raw/main/main.png"> |[Github](https://github.com/FYYFU/HeadKV) <br> [Paper](https://arxiv.org/abs/2410.19258)|[//]: #11/17
|[TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection](https://arxiv.org/abs/2411.02886) <br> Wei Wu, Zhuoshi Pan, Chao Wang, Liyi Chen, Yunchu Bai, Kun Fu, Zheng Wang, Hui Xiong |<img width="1002" alt="image" src="https://arxiv.org/html/2411.02886v1/x1.png"> |[Paper](https://arxiv.org/abs/2411.02886)|[//]: #11/18
|[MiniKV: Pushing the Limits of LLM Inference via 2-Bit Layer-Discriminative KV Cache](https://arxiv.org/abs/2411.18077) <br> Akshat Sharma, Hangliang Ding, Jianping Li, Neel Dani, Minjia Zhang |<img width="1002" alt="image" src="https://arxiv.org/html/2411.18077v2/x1.png"> |[Paper](https://arxiv.org/abs/2411.18077)|[//]: #12/07
|[ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression](https://arxiv.org/abs/2412.03213) <br> Guangda Liu, Chengwei Li, Jieru Zhao, Chenqi Zhang, Minyi Guo |<img width="1002" alt="image" src="https://arxiv.org/html/2412.03213v1/x1.png"> |[Paper](https://arxiv.org/abs/2412.03213)|[//]: #12/09
|[Unifying KV Cache Compression for Large Language Models with LeanKV](https://arxiv.org/abs/2412.03131) <br> Yanqi Zhang, Yuwei Hu, Runyuan Zhao, John C.S. Lui, Haibo Chen |<img width="1002" alt="image" src="https://arxiv.org/html/2412.03131v1/x2.png"> |[Paper](https://arxiv.org/abs/2412.03131)|[//]: #12/09
|[Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity](https://arxiv.org/abs/2412.02252) <br> Da Ma, Lu Chen, Situo Zhang, Yuxun Miao, Su Zhu, Zhi Chen, Hongshen Xu, Hanqi Li, Shuai Fan, Lei Pan, Kai Yu |<img width="1002" alt="image" src="https://arxiv.org/html/2412.02252v1/extracted/6041612/figs/intro.png"> |[Paper](https://arxiv.org/abs/2412.02252)|[//]: #12/09