
## Inference Acceleration
| Title & Authors | Introduction | Links |
|:--|  :----: | :---:|
|[![Star](https://img.shields.io/github/stars/FMInference/DejaVu.svg?style=social&label=Star)](https://github.com/FMInference/DejaVu)[![Publish](https://img.shields.io/badge/Conference-ICML'23%20Oral-blue)]()<br>[Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time](https://openreview.net/forum?id=wIPIhHd00i) <br> Zichang Liu, Jue WANG, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, Beidi Chen |<img width="202" alt="image" src="figures/DajeVu.png"> |[Github](https://github.com/FMInference/DejaVu) <br> [Paper](https://openreview.net/forum?id=wIPIhHd00i)|
|[![Publish](https://img.shields.io/badge/Conference-NeurIPS'23-blue)]()<br>[Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers](https://arxiv.org/abs/2305.15805) <br> Sotiris Anagnostidis, Dario Pavllo, Luca Biggio, Lorenzo Noci, Aurelien Lucchi, Thomas Hofmann |<img width="1602" alt="image" src="figures/DCP.png"> |[Paper](https://arxiv.org/abs/2305.15805)|
|[![Star](https://img.shields.io/github/stars/FMInference/H2O.svg?style=social&label=Star)](https://github.com/FMInference/H2O)[![Publish](https://img.shields.io/badge/Conference-NeurIPS'23-blue)]()<br>[H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://arxiv.org/abs/2306.14048) <br> Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, Beidi Chen |<img width="1002" alt="image" src="https://github.com/FMInference/H2O/blob/main/Figs/h2o.jpg"> |[Github](https://github.com/FMInference/H2O) <br> [Paper](https://arxiv.org/abs/2306.14048)|
|[![Star](https://img.shields.io/github/stars/raymin0223/fast_robust_early_exit.svg?style=social&label=Star)](https://github.com/raymin0223/fast_robust_early_exit)[![Publish](https://img.shields.io/badge/Conference-EMNLP'23-blue)]()<br>[Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding](https://arxiv.org/abs/2310.05424) <br> Sangmin Bae, Jongwoo Ko, Hwanjun Song, Se-Young Yun |<img width="1002" alt="image" src="figures/FREE.png"> |[Github](https://github.com/raymin0223/fast_robust_early_exit) <br> [Paper](https://arxiv.org/abs/2310.05424)|
|[![Star](https://img.shields.io/github/stars/liyucheng09/Selective_Context.svg?style=social&label=Star)](https://github.com/liyucheng09/Selective_Context)[![Publish](https://img.shields.io/badge/Conference-EMNLP'23-blue)]()<br>[Compressing Context to Enhance Inference Efficiency of Large Language Models](https://arxiv.org/abs/2310.06201) <br> Yucheng Li, Bo Dong, Chenghua Lin, Frank Guerin |<img width="1002" alt="image" src="figures/selective_context.png"> |[Github](https://github.com/liyucheng09/Selective_Context) <br> [Paper](https://arxiv.org/abs/2310.06201)|
|[![Publish](https://img.shields.io/badge/Conference-AAAI'24-blue)]()<br>[ConsistentEE: A Consistent and Hardness-Guided Early Exiting Method for Accelerating Language Models Inference](https://arxiv.org/abs/2312.11882) <br> Ziqian Zeng, Yihuai Hong, Hongliang Dai, Huiping Zhuang, Cen Chen |<img width="1002" alt="image" src="figures/ConsistentEE.png"> |[Paper](https://arxiv.org/abs/2312.11882)|
|[![Publish](https://img.shields.io/badge/Conference-ICML'23%20ES%20FOMO-blue)]()<br>[Accelerating LLM Inference with Staged Speculative Decoding](https://arxiv.org/abs/2308.04623) <br> Benjamin Spector, Chris Re |<img width="202" alt="image" src="figures/StagedSpec.png"> |[Paper](https://arxiv.org/abs/2308.04623)|
|[![Publish](https://img.shields.io/badge/Conference-EMNLP'23%20Findings-blue)]()<br>[TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction](https://arxiv.org/abs/2310.15556) <br> Junyi Liu, Liangzhi Li, Tong Xiang, Bowen Wang, Yiming Qian |<img width="1002" alt="image" src="figures/TCRA-LLM.png"> |[Paper](https://arxiv.org/abs/2310.15556)|
| [Inference with Reference: Lossless Acceleration of Large Language Models](https://arxiv.org/abs/2304.04487) <br> Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, Furu Wei | <img width="600" alt="image" src="figures/llma.png"> | [Github](https://github.com/microsoft/LMOps/tree/main/llma) <br> [paper](https://arxiv.org/abs/2304.04487) |
| [![Star](https://img.shields.io/github/stars/flexflow/FlexFlow.svg?style=social&label=Star)](https://github.com/flexflow/FlexFlow/tree/inference) <br> [SpecInfer: Accelerating Generative LLM Serving with Speculative Inference and Token Tree Verification](https://arxiv.org/abs/2305.09781) <br> Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Rae Ying Yee Wong, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, Zhihao Jia| <img width="600" alt="image" src="https://github.com/flexflow/FlexFlow/blob/inference/img/overview.png">| [Github](https://github.com/flexflow/FlexFlow/tree/inference) <br> [paper](https://arxiv.org/abs/2305.09781) |
|[SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference](https://arxiv.org/abs/2307.02628) <br> Luciano Del Corro, Allie Del Giorno, Sahaj Agarwal, Bin Yu, Ahmed Awadallah, Subhabrata Mukherjee |<img width="1002" alt="image" src="figures/SkipDecode.png"> |[Paper](https://arxiv.org/abs/2307.02628)|
|[Skeleton-of-Thought: Large Language Models Can Do Parallel Decoding](https://arxiv.org/abs/2307.15337) <br> Xuefei Ning, Zinan Lin, Zixuan Zhou, Huazhong Yang, Yu Wang |<img width="1002" alt="image" src="figures/SoT.png"> |[Paper](https://arxiv.org/abs/2307.15337)|
|[![Star](https://img.shields.io/github/stars/dilab-zju/self-speculative-decoding.svg?style=social&label=Star)](https://github.com/dilab-zju/self-speculative-decoding)<br>[Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding](https://arxiv.org/abs/2309.08168) <br> Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, Sharad Mehrotra |<img width="1002" alt="image" src="https://github.com/dilab-zju/self-speculative-decoding/blob/main/assets/intro.png"> |[Github](https://github.com/dilab-zju/self-speculative-decoding) <br> [Paper](https://arxiv.org/abs/2309.08168)|
|[![Star](https://img.shields.io/github/stars/mit-han-lab/streaming-llm.svg?style=social&label=Star)](https://github.com/mit-han-lab/streaming-llm)<br>[Efficient Streaming Language Models with Attention Sinks](https://arxiv.org/abs/2309.17453) <br> Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis |<img width="1002" alt="image" src="https://github.com/mit-han-lab/streaming-llm/blob/main/figures/schemes.png"> |[Github](https://github.com/mit-han-lab/streaming-llm) <br> [Paper](https://arxiv.org/abs/2309.17453)|
|[(Dynamic) Prompting might be all you need to repair Compressed LLMs](https://arxiv.org/abs/2310.00867) <br> Duc N.M Hoang, Minsik Cho, Thomas Merth, Mohammad Rastegari, Zhangyang Wang |<img width="1002" alt="image" src="figures/IDP.png"> |[Paper](https://arxiv.org/abs/2310.00867)|
|[![Star](https://img.shields.io/github/stars/MurongYue/LLM_MoT_cascade.svg?style=social&label=Star)](https://github.com/MurongYue/LLM_MoT_cascade)<br>[Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoning](https://arxiv.org/abs/2310.03094) <br> Murong Yue, Jie Zhao, Min Zhang, Liang Du, Ziyu Yao |<img width="1002" alt="image" src="figures/LLM_MoT_cascade.png"> |[Github](https://github.com/MurongYue/LLM_MoT_cascade) <br> [Paper](https://arxiv.org/abs/2310.03094)|
|[CacheGen: Fast Context Loading for Language Model Applications](https://arxiv.org/abs/2310.07240) <br> Yuhan Liu, Hanchen Li, Kuntai Du, Jiayi Yao, Yihua Cheng, Yuyang Huang, Shan Lu, Michael Maire, Henry Hoffmann, Ari Holtzman, Ganesh Ananthanarayanan, Junchen Jiang |<img width="1002" alt="image" src="figures/CacheGen.png"> |[Paper](https://arxiv.org/abs/2310.07240)|
|[![Star](https://img.shields.io/github/stars/DRSY/KV_Compression.svg?style=social&label=Star)](https://github.com/DRSY/KV_Compression)[![Publish](https://img.shields.io/badge/Conference-EMNLP'23-blue)]()<br>[Context Compression for Auto-regressive Transformers with Sentinel Tokens](https://arxiv.org/abs/2310.08152) <br> Siyu Ren, Qi Jia, Kenny Q. Zhu |<img width="1002" alt="image" src="figures/KV_compression.png"> |[Github](https://github.com/DRSY/KV_Compression) <br> [Paper](https://arxiv.org/abs/2310.08152)|
|[![Star](https://img.shields.io/github/stars/ielab/llm-rankers.svg?style=social&label=Star)](https://github.com/ielab/llm-rankers)<br>[A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models](https://arxiv.org/abs/2310.09497) <br> Shengyao Zhuang, Honglei Zhuang, Bevan Koopman, Guido Zuccon |<img width="1002" alt="image" src="figures/Setwise.png"> |[Github](https://github.com/ielab/llm-rankers) <br> [Paper](https://arxiv.org/abs/2310.09497)|
|[SPEED: Speculative Pipelined Execution for Efficient Decoding](https://arxiv.org/abs/2310.12072) <br> Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Hasan Genc, Kurt Keutzer, Amir Gholami, Sophia Shao |<img width="1002" alt="image" src="figures/SPEED.png"> |[Paper](https://arxiv.org/abs/2310.12072)|
|[Accelerating LLM Inference by Enabling Intermediate Layer Decoding](https://arxiv.org/abs/2310.18581) <br> Neeraj Varshney, Agneet Chatterjee, Mihir Parmar, Chitta Baral |<img width="252" alt="image" src="figures/LITE.png"> |[Paper](https://arxiv.org/abs/2310.18581)|
|[Fast Chain-of-Thought: A Glance of Future from Parallel Decoding Leads to Answers Faster](https://arxiv.org/abs/2311.08263) <br> Hongxuan Zhang, Zhining Liu, Jiaqi Zheng, Chenyi Zhuang, Jinjie Gu, Guihai Chen |<img width="1002" alt="image" src="figures/FastCoT.png"> |[Paper](https://arxiv.org/abs/2311.08263)|
|[![Star](https://img.shields.io/github/stars/snu-mllab/context-memory.svg?style=social&label=Star)](https://github.com/snu-mllab/context-memory)<br>[Compressed Context Memory For Online Language Model Interaction](https://arxiv.org/abs/2312.03414) <br> Jang-Hyun Kim, Junyoung Yeom, Sangdoo Yun, Hyun Oh Song |<img width="1002" alt="image" src="https://github.com/snu-mllab/Context-Memory/blob/main/image/main.png"> |[Github](https://github.com/snu-mllab/context-memory) <br> [Paper](https://arxiv.org/abs/2312.03414)|
|[SparQ Attention: Bandwidth-Efficient LLM Inference](https://arxiv.org/abs/2312.04985) <br> Luka Ribar, Ivan Chelombiev, Luke Hudlass-Galley, Charlie Blake, Carlo Luschi, Douglas Orr |<img width="1002" alt="image" src="figures/SparQ.png"> |[Paper](https://arxiv.org/abs/2312.04985)|
|[Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracy](https://arxiv.org/abs/2312.12728) <br> Yao Zhao, Zhitian Xie, Chenyi Zhuang, Jinjie Gu |<img width="1002" alt="image" src="figures/Lookahead.png"> |[Paper](https://arxiv.org/abs/2312.12728)|
|[Cascade Speculative Drafting for Even Faster LLM Inference](https://arxiv.org/abs/2312.11462) <br> Ziyi Chen, Xiaocong Yang, Jiacheng Lin, Chenkai Sun, Jie Huang, Kevin Chen-Chuan Chang |<img width="1002" alt="image" src="figures/CSDrafting.png"> |[Paper](https://arxiv.org/abs/2312.11462)|
|[![Star](https://img.shields.io/github/stars/SafeAILab/EAGLE.svg?style=social&label=Star)](https://github.com/SafeAILab/EAGLE)<br>[EAGLE: Lossless Acceleration of LLM Decoding by Feature Extrapolation](https://sites.google.com/view/eagle-llm) <br> Yuhui Li, Chao Zhang, and Hongyang Zhang |<img width="302" alt="image" src="https://github.com/SafeAILab/EAGLE/blob/main/figs/fig1.png"> |[Github](https://github.com/SafeAILab/EAGLE) <br> [Blog](https://sites.google.com/view/eagle-llm)|
|[LoMA: Lossless Compressed Memory Attention](https://arxiv.org/abs/2401.09486) <br> Yumeng Wang, Zhenyang Xiao |<img width="102" alt="image" src="figures/LoMA.png"> |[Paper](https://arxiv.org/abs/2401.09486)|
|[![Star](https://img.shields.io/github/stars/FasterDecoding/Medusa.svg?style=social&label=Star)](https://github.com/FasterDecoding/Medusa)<br>[Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://arxiv.org/abs/2401.10774) <br> Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, Tri Dao |<img width="1002" alt="image" src="https://arxiv.org/html/2401.10774v1/x1.png"> |[Github](https://github.com/FasterDecoding/Medusa) <br> [Paper](https://arxiv.org/abs/2401.10774)|
|[APAR: LLMs Can Do Auto-Parallel Auto-Regressive Decoding](https://arxiv.org/abs/2401.06761) <br> Mingdao Liu, Aohan Zeng, Bowen Wang, Peng Zhang, Jie Tang, Yuxiao Dong |<img width="1002" alt="image" src="https://arxiv.org/html/2401.06761v1/x1.png"> |[Paper](https://arxiv.org/abs/2401.06761)|
|[![Star](https://img.shields.io/github/stars/linfeng93/BiTA.svg?style=social&label=Star)](https://github.com/linfeng93/BiTA)<br>[BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Models](https://arxiv.org/abs/2401.12522) <br> Feng Lin, Hanling Yi, Hongbin Li, Yifan Yang, Xiaotian Yu, Guangming Lu, Rong Xiao |<img width="1002" alt="image" src="figures/BiTA.png"> |[Github](https://github.com/linfeng93/BiTA) <br> [Paper](https://arxiv.org/abs/2401.12522)|
|[Speculative Streaming: Fast LLM Inference without Auxiliary Models](https://arxiv.org/abs/2402.11131) <br> Nikhil Bhendawade, Irina Belousova, Qichen Fu, Henry Mason, Mohammad Rastegari, Mahyar Najibi |<img width="1002" alt="image" src="figures/SpeculativeStreaming.png"> |[Paper](https://arxiv.org/abs/2402.11131)|
|[RelayAttention for Efficient Large Language Model Serving with Long System Prompts](https://arxiv.org/abs/2402.14808) <br> Lei Zhu, Xinjiang Wang, Wayne Zhang, Rynson W.H. Lau |<img width="1002" alt="image" src="https://arxiv.org/html/2402.14808v1/x3.png"> |[Paper](https://arxiv.org/abs/2402.14808)|
|[Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement](https://arxiv.org/abs/2402.14160) <br> Wonseok Jeon, Mukul Gagrani, Raghavv Goel, Junyoung Park, Mingu Lee, Christopher Lott |<img width="1002" alt="image" src="figures/RSD.png"> |[Paper](https://arxiv.org/abs/2402.14160)|
|[![Star](https://img.shields.io/github/stars/kafkayu/Chimera.svg?style=social&label=Star)](https://github.com/kafkayu/Chimera)<br>[Chimera: A Lossless Decoding Method for Accelerating Large Language Models Inference by Fusing all Tokens](https://arxiv.org/abs/2402.15758) <br> Ziqian Zeng, Jiahong Yu, Qianshi Pang, Zihao Wang, Huiping Zhuang, Cen Chen |<img width="1002" alt="image" src="https://arxiv.org/html/2402.15758v1/x1.png"> |[Github](https://github.com/kafkayu/Chimera) <br> [Paper](https://arxiv.org/abs/2402.15758)|
|[CHAI: Clustered Head Attention for Efficient LLM Inference](https://arxiv.org/abs/2403.08058) <br> Saurabh Agarwal, Bilge Acun, Basil Homer, Mostafa Elhoushi, Yejin Lee, Shivaram Venkataraman, Dimitris Papailiopoulos, Carole-Jean Wu |<img width="1002" alt="image" src="figures/chai.png"> |[Paper](https://arxiv.org/abs/2403.08058)|
|[Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference](https://arxiv.org/abs/2403.09636) <br> Piotr Nawrot, Adrian Łańcucki, Marcin Chochowski, David Tarjan, Edoardo M. Ponti |<img width="1002" alt="image" src="https://arxiv.org/html/2403.09636v1/x3.png"> |[Paper](https://arxiv.org/abs/2403.09636)|
|[Recurrent Drafter for Fast Speculative Decoding in Large Language Models](https://arxiv.org/abs/2403.09919) <br> Aonan Zhang, Chong Wang, Yi Wang, Xuanyu Zhang, Yunfei Cheng |<img width="1002" alt="image" src="https://arxiv.org/html/2403.09919v1/x2.png"> |[Paper](https://arxiv.org/abs/2403.09919)|
|[Optimal Block-Level Draft Verification for Accelerating Speculative Decoding](https://arxiv.org/abs/2403.10444) <br> Ziteng Sun, Jae Hun Ro, Ahmad Beirami, Ananda Theertha Suresh |<img width="1002" alt="image" src="https://arxiv.org/html/2403.10444v1/x1.png"> |[Paper](https://arxiv.org/abs/2403.10444)|
|[Hierarchical Skip Decoding for Efficient Autoregressive Text Generation](https://arxiv.org/abs/2403.14919) <br> Yunqi Zhu, Xuebing Yang, Yuanyuan Wu, Wensheng Zhang |<img width="1002" alt="image" src="https://arxiv.org/html/2403.14919v1/x2.png"> |[Paper](https://arxiv.org/abs/2403.14919)|
|[![Star](https://img.shields.io/github/stars/hasuoshenyun/SDSAT.svg?style=social&label=Star)](https://github.com/hasuoshenyun/SDSAT)<br>[SDSAT: Accelerating LLM Inference through Speculative Decoding with Semantic Adaptive Tokens](https://arxiv.org/abs/2403.18647) <br> Chengbo Liu, Yong Zhu |<img width="1002" alt="image" src="https://arxiv.org/html/2403.18647v1/extracted/5495606/sample_struture.png"> |[Github](https://github.com/hasuoshenyun/SDSAT) <br> [Paper](https://arxiv.org/abs/2403.18647)|
|[![Star](https://img.shields.io/github/stars/siyan-zhao/prepacking.svg?style=social&label=Star)](https://github.com/siyan-zhao/prepacking)<br>[Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models](https://arxiv.org/abs/2404.09529) <br> Siyan Zhao, Daniel Israel, Guy Van den Broeck, Aditya Grover |<img width="302" alt="image" src="https://github.com/siyan-zhao/prepacking/raw/main/figures/prepacking_gif_final.gif"> |[Github](https://github.com/siyan-zhao/prepacking) <br> [Paper](https://arxiv.org/abs/2404.09529)|
|[Exploring and Improving Drafts in Blockwise Parallel Decoding](https://arxiv.org/abs/2404.09221) <br> Taehyeon Kim, Ananda Theertha Suresh, Kishore Papineni, Michael Riley, Sanjiv Kumar, Adrian Benton |<img width="602" alt="image" src="figures/BPD.png"> |[Paper](https://arxiv.org/abs/2404.09221)|
|[![Star](https://img.shields.io/github/stars/oujieww/ANPD.svg?style=social&label=Star)](https://github.com/oujieww/ANPD)<br>[Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decoding](https://arxiv.org/abs/2404.08698) <br> Jie Ou, Yueming Chen, Wenhong Tian |<img width="602" alt="image" src="figures/ANPD.png"> |[Github](https://github.com/oujieww/ANPD) <br> [Paper](https://arxiv.org/abs/2404.08698)|
|[Self-Selected Attention Span for Accelerating Large Language Model Inference](https://arxiv.org/abs/2404.09336) <br> Tian Jin, Wanzin Yazar, Zifei Xu, Sayeh Sharify, Xin Wang |<img width="202" alt="image" src="https://arxiv.org/html/2404.09336v1/x1.png"> |[Paper](https://arxiv.org/abs/2404.09336)|
|[Parallel Decoding via Hidden Transfer for Lossless Large Language Model Acceleration](https://arxiv.org/abs/2404.12022) <br> Pengfei Wu, Jiahao Liu, Zhuocheng Gong, Qifan Wang, Jinpeng Li, Jingang Wang, Xunliang Cai, Dongyan Zhao |<img width="1002" alt="image" src="https://arxiv.org/html/2404.12022v1/extracted/5529882/figure/main_pic.png"> |[Paper](https://arxiv.org/abs/2404.12022)|
|[XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference](https://arxiv.org/abs/2404.15420) <br> João Monteiro, Étienne Marcotte, Pierre-André Noël, Valentina Zantedeschi, David Vázquez, Nicolas Chapados, Christopher Pal, Perouz Taslakian |<img width="1002" alt="image" src="https://arxiv.org/html/2404.15420v1/x2.png"> |[Paper](https://arxiv.org/abs/2404.15420)|
|[![Star](https://img.shields.io/github/stars/m365-core/hybrid_llm_routing.svg?style=social&label=Star)](https://github.com/m365-core/hybrid_llm_routing)[![Publish](https://img.shields.io/badge/Conference-ICLR'24-blue)]()<br>[Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing](https://arxiv.org/abs/2404.14618) <br> Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks V.S. Lakshmanan, Ahmed Hassan Awadallah |<img width="1002" alt="image" src="figures/hybridLLM.png"> |[Github](https://github.com/m365-core/hybrid_llm_routing) <br> [Paper](https://arxiv.org/abs/2404.14618)|
|[Efficient LLM Inference with Kcache](https://arxiv.org/abs/2404.18057) <br> Qiaozhi He, Zhihua Wu |<img width="1002" alt="image" src="figures/kcache.png"> |[Paper](https://arxiv.org/abs/2404.18057)|
|[Better & Faster Large Language Models via Multi-token Prediction](https://arxiv.org/abs/2404.19737) <br> Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, Gabriel Synnaeve |<img width="1002" alt="image" src="figures/MBPP.png"> |[Paper](https://arxiv.org/abs/2404.19737)|
|[![Publish](https://img.shields.io/badge/Conference-ICML'24-blue)]()<br>[KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generation](https://arxiv.org/abs/2405.05329) <br> Minsik Cho, Mohammad Rastegari, Devang Naik |<img width="1002" alt="image" src="https://arxiv.org/html/2405.05329v1/extracted/5585389/figure/llm_inference.png"> |[Paper](https://arxiv.org/abs/2405.05329)|
|[You Only Cache Once: Decoder-Decoder Architectures for Language Models](https://arxiv.org/abs/2405.05254) <br> Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, Furu Wei |<img width="1002" alt="image" src="figures/YOCO.png"> |[Github](https://github.com/microsoft/unilm/tree/master/YOCO) <br> [Paper](https://arxiv.org/abs/2405.05254)|
|[![Star](https://img.shields.io/github/stars/Equationliu/Kangaroo.svg?style=social&label=Star)](https://github.com/Equationliu/Kangaroo)<br>[Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting](https://arxiv.org/abs/2404.18911) <br> Fangcheng Liu, Yehui Tang, Zhenhua Liu, Yunsheng Ni, Kai Han, Yunhe Wang |<img width="1002" alt="image" src="https://github.com/Equationliu/Kangaroo/blob/main/imgs/kangaroo.png"> |[Github](https://github.com/Equationliu/Kangaroo) <br> [Paper](https://arxiv.org/abs/2404.18911)|
|[Accelerating Speculative Decoding using Dynamic Speculation Length](https://arxiv.org/abs/2405.04304) <br> Jonathan Mamou, Oren Pereg, Daniel Korat, Moshe Berchansky, Nadav Timor, Moshe Wasserblat, Roy Schwartz |<img width="1002" alt="image" src="figures/DISCO_2.png"> |[Paper](https://arxiv.org/abs/2405.04304)|
|[Clover: Regressive Lightweight Speculative Decoding with Sequential Knowledge](https://arxiv.org/abs/2405.00263) <br> Bin Xiao, Chunan Shi, Xiaonan Nie, Fan Yang, Xiangwei Deng, Lei Su, Weipeng Chen, Bin Cui |<img width="1002" alt="image" src="figures/clover.png"> |[Paper](https://arxiv.org/abs/2405.00263)|
|[![Star](https://img.shields.io/github/stars/niyunsheng/EMS-SD.svg?style=social&label=Star)](https://github.com/niyunsheng/EMS-SD)<br>[EMS-SD: Efficient Multi-sample Speculative Decoding for Accelerating Large Language Models](https://arxiv.org/abs/2405.07542) <br> Yunsheng Ni, Chuanjian Liu, Yehui Tang, Kai Han, Yunhe Wang |<img width="202" alt="image" src="https://github.com/niyunsheng/EMS-SD/raw/main/assets/fig2-method.png"> |[Github](https://github.com/niyunsheng/EMS-SD) <br> [Paper](https://arxiv.org/abs/2405.07542)|
|[Distributed Speculative Inference of Large Language Models](https://arxiv.org/abs/2405.14105) <br> Nadav Timor, Jonathan Mamou, Daniel Korat, Moshe Berchansky, Oren Pereg, Moshe Wasserblat, Tomer Galanti, Michal Gordon, David Harel |<img width="1002" alt="image" src="figures/DSI.png"> |[Paper](https://arxiv.org/abs/2405.14105)|
|[![Star](https://img.shields.io/github/stars/Zoeyyao27/SirLLM.svg?style=social&label=Star)](https://github.com/Zoeyyao27/SirLLM)<br>[SirLLM: Streaming Infinite Retentive LLM](https://arxiv.org/abs/2405.12528) <br> Yao Yao, Zuchao Li, Hai Zhao |<img width="1002" alt="image" src="https://github.com/Zoeyyao27/SirLLM/blob/main/image/sirllm.png"> |[Github](https://github.com/Zoeyyao27/SirLLM) <br> [Paper](https://arxiv.org/abs/2405.12528)|
|[![Star](https://img.shields.io/github/stars/hmarkc/parallel-prompt-decoding.svg?style=social&label=Star)](https://github.com/hmarkc/parallel-prompt-decoding)<br>[Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference](https://arxiv.org/abs/2405.18628) <br> Hao (Mark)Chen, Wayne Luk, Ka Fai Cedric Yiu, Rui Li, Konstantin Mishchenko, Stylianos I. Venieris, Hongxiang Fan |<img width="1002" alt="image" src="https://github.com/hmarkc/parallel-prompt-decoding/raw/main/assets/Overview.png"> |[Github](https://github.com/hmarkc/parallel-prompt-decoding) <br> [Paper](https://arxiv.org/abs/2405.18628)|
|[Faster Cascades via Speculative Decoding](https://arxiv.org/abs/2405.19261) <br> Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Seungyeon Kim, Neha Gupta, Aditya Krishna Menon, Sanjiv Kumar |<img width="1002" alt="image" src="figures/speccascade.png"> |[Paper](https://arxiv.org/abs/2405.19261)|
|[![Publish](https://img.shields.io/badge/Conference-ACL'24%20Findings-blue)]()<br>[Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism](https://arxiv.org/abs/2406.03853) <br> Jiahao Liu, Qifan Wang, Jingang Wang, Xunliang Cai |<img width="1002" alt="image" src="https://arxiv.org/html/2406.03853v1/x3.png"> |[Paper](https://arxiv.org/abs/2406.03853)|[//]: #06/12
|[![Star](https://img.shields.io/github/stars/dvlab-research/Q-LLM.svg?style=social&label=Star)](https://github.com/dvlab-research/Q-LLM)<br>[QuickLLaMA: Query-aware Inference Acceleration for Large Language Models](https://arxiv.org/abs/2406.07528) <br> Jingyao Li, Han Shi, Xin Jiang, Zhenguo Li, Hong Xu, Jiaya Jia |<img width="1002" alt="image" src="https://github.com/dvlab-research/Q-LLM/raw/master/img/framework.png"> |[Github](https://github.com/dvlab-research/Q-LLM) <br> [Paper](https://arxiv.org/abs/2406.07528)|[//]: #06/12
|[![Star](https://img.shields.io/github/stars/GATECH-EIC/Linearized-LLM.svg?style=social&label=Star)](https://github.com/GATECH-EIC/Linearized-LLM)[![Publish](https://img.shields.io/badge/Conference-ICML'24-blue)]()<br>[When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models](https://arxiv.org/abs/2406.07368) <br> Haoran You, Yichao Fu, Zheng Wang, Amir Yazdanbakhsh, Yingyan (Celine)Lin |<img width="1002" alt="image" src="https://arxiv.org/html/2406.07368v1/x5.png"> |[Github](https://github.com/GATECH-EIC/Linearized-LLM) <br> [Paper](https://arxiv.org/abs/2406.07368)|[//]: #06/12
|[HiP Attention: Sparse Sub-Quadratic Attention with Hierarchical Attention Pruning](https://arxiv.org/abs/2406.09827) <br> Heejun Lee, Geon Park, Youngwan Lee, Jina Kim, Wonyoung Jeong, Myeongjae Jeon, Sung Ju Hwang |<img width="1002" alt="image" src="https://arxiv.org/html/2406.09827v1/x1.png"> |[Paper](https://arxiv.org/abs/2406.09827)|[//]: #06/18
|[Optimized Speculative Sampling for GPU Hardware Accelerators](https://arxiv.org/abs/2406.11016) <br> Dominik Wagner, Seanie Lee, Ilja Baumann, Philipp Seeberger, Korbinian Riedhammer, Tobias Bocklet |<img width="1002" alt="image" src="https://arxiv.org/html/2406.11016v1/x1.png"> |[Paper](https://arxiv.org/abs/2406.11016)|[//]: #06/24
|[![Star](https://img.shields.io/github/stars/thu-nics/MoA.svg?style=social&label=Star)](https://github.com/thu-nics/MoA)<br>[MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression](https://arxiv.org/abs/2406.14909) <br> Tianyu Fu, Haofeng Huang, Xuefei Ning, Genghan Zhang, Boju Chen, Tianqi Wu, Hongyi Wang, Zixiao Huang, Shiyao Li, Shengen Yan, Guohao Dai, Huazhong Yang, Yu Wang |<img width="1002" alt="image" src="https://github.com/thu-nics/MoA/blob/master/assets/workflow.png"> |[Github](https://github.com/thu-nics/MoA) <br> [Paper](https://arxiv.org/abs/2406.14909)|[//]: #06/26
|[Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attention](https://arxiv.org/abs/2406.15486) <br> Qianchao Zhu, Jiangfei Duan, Chang Chen, Siran Liu, Xiuhong Li, Guanyu Feng, Xin Lv, Huanqi Cao, Xiao Chuanfu, Xingcheng Zhang, Dahua Lin, Chao Yang |<img width="1002" alt="image" src="https://arxiv.org/html/2406.15486v1/x1.png"> |[Paper](https://arxiv.org/abs/2406.15486)|[//]: #06/26
|[Interpreting Attention Layer Outputs with Sparse Autoencoders](https://arxiv.org/abs/2406.17759) <br> Connor Kissane, Robert Krzyzanowski, Joseph Isaac Bloom, Arthur Conmy, Neel Nanda |<img width="1002" alt="image" src="https://arxiv.org/html/2406.17759v1/x1.png"> |[Paper](https://arxiv.org/abs/2406.17759)|[//]: #06/26
|[![Star](https://img.shields.io/github/stars/SafeAILab/EAGLE.svg?style=social&label=Star)](https://github.com/SafeAILab/EAGLE)<br>[EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://arxiv.org/abs/2406.16858) <br> Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang |<img width="1002" alt="image" src="https://arxiv.org/html/2406.16858v1/x4.png"> |[Github](https://github.com/SafeAILab/EAGLE) <br> [Paper](https://arxiv.org/abs/2406.16858)|[//]: #06/26
|[Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers](https://arxiv.org/abs/2406.16747) <br> Chao Lou, Zixia Jia, Zilong Zheng, Kewei Tu |<img width="1002" alt="image" src="https://arxiv.org/html/2406.16747v1/x1.png"> |[Paper](https://arxiv.org/abs/2406.16747)|[//]: #06/26
|[S2D: Sorted Speculative Decoding For More Efficient Deployment of Nested Large Language Models](https://arxiv.org/abs/2407.01955) <br> Parsa Kavehzadeh, Mohammadreza Pourreza, Mojtaba Valipour, Tinashu Zhu, Haoli Bai, Ali Ghodsi, Boxing Chen, Mehdi Rezagholizadeh |<img width="1002" alt="image" src="https://arxiv.org/html/2407.01955v1/x1.png"> |[Paper](https://arxiv.org/abs/2407.01955)|[//]: #07/05
|[![Star](https://img.shields.io/github/stars/ChuangtaoChen-TUM/LiveMind.svg?style=social&label=Star)](https://github.com/ChuangtaoChen-TUM/LiveMind)<br>[LiveMind: Low-latency Large Language Models with Simultaneous Inference](https://arxiv.org/abs/2406.14319) <br> Chuangtao Chen, Grace Li Zhang, Xunzhao Yin, Cheng Zhuo, Ulf Schlichtmann, Bing Li |<img width="1002" alt="image" src="https://arxiv.org/html/2406.14319v1/x1.png"> |[Github](https://github.com/ChuangtaoChen-TUM/LiveMind) <br> [Paper](https://arxiv.org/abs/2406.14319)|[//]: #07/05
|[Multi-Token Joint Speculative Decoding for Accelerating Large Language Model Inference](https://arxiv.org/abs/2407.09722) <br> Zongyue Qin, Ziniu Hu, Zifan He, Neha Prakriya, Jason Cong, Yizhou Sun |<img width="1002" alt="image" src="https://arxiv.org/html/2407.09722v1/x1.png"> |[Paper](https://arxiv.org/abs/2407.09722)|[//]: #07/16
|[Adaptive Draft-Verification for Efficient Large Language Model Decoding](https://arxiv.org/abs/2407.12021) <br> Xukun Liu, Bowen Lei, Ruqi Zhang, Dongkuan Xu |<img width="1002" alt="image" src="https://arxiv.org/html/2407.12021v1/x1.png"> |[Paper](https://arxiv.org/abs/2407.12021)|[//]: #07/21
|[LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference](https://arxiv.org/abs/2407.14057) <br> Qichen Fu, Minsik Cho, Thomas Merth, Sachin Mehta, Mohammad Rastegari, Mahyar Najibi |<img width="1002" alt="image" src="https://arxiv.org/html/2407.14057v1/x1.png"> |[Paper](https://arxiv.org/abs/2407.14057)|[//]: #07/24
|[![Publish](https://img.shields.io/badge/Conference-IVUS'24-blue)]()<br>[Inference acceleration for large language models using "stairs" assisted greedy generation](https://arxiv.org/abs/2407.19947) <br> Domas Grigaliūnas, Mantas Lukoševičius |<img width="1002" alt="image" src="https://arxiv.org/html/2407.19947v1/extracted/5761251/assist_inf_2.png"> |[Paper](https://arxiv.org/abs/2407.19947)|[//]: #08/08
|[An Efficient Inference Framework for Early-exit Large Language Models](https://arxiv.org/abs/2407.20272) <br> Ruijie Miao, Yihan Yan, Xinshuo Yao, Tong Yang ||[Paper](https://arxiv.org/abs/2407.20272)|[//]: #08/08
|[Accelerating Large Language Model Inference with Self-Supervised Early Exits](https://arxiv.org/abs/2407.21082) <br> Florian Valade | |[Paper](https://arxiv.org/abs/2407.21082)|[//]: #08/08
|[![Star](https://img.shields.io/github/stars/XiaoBin1992/clover.svg?style=social&label=Star)](https://github.com/XiaoBin1992/clover)<br>[Clover-2: Accurate Inference for Regressive Lightweight Speculative Decoding](https://arxiv.org/abs/2408.00264) <br> Bin Xiao, Lujun Gui, Lei Su, Weipeng Chen |<img width="1002" alt="image" src="https://github.com/XiaoBin1992/clover/raw/v1/figs/structure.png"> |[Github](https://github.com/XiaoBin1992/clover) <br> [Paper](https://arxiv.org/abs/2408.00264)|[//]: #08/08
|[Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion](https://arxiv.org/abs/2408.05636) <br> Jacob K Christopher, Brian R Bartoldson, Bhavya Kailkhura, Ferdinando Fioretto |<img width="1002" alt="image" src="https://arxiv.org/html/2408.05636v1/x1.png"> |[Paper](https://arxiv.org/abs/2408.05636)|[//]: #08/13
|[Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling](https://arxiv.org/abs/2408.08696) <br> Xianzhen Luo, Yixuan Wang, Qingfu Zhu, Zhiming Zhang, Xuanyu Zhang, Qing Yang, Dongliang Xu, Wanxiang Che |<img width="202" alt="image" src="https://arxiv.org/html/2408.08696v1/x1.png"> |[Paper](https://arxiv.org/abs/2408.08696)|[//]: #08/20
|[Boosting Lossless Speculative Decoding via Feature Sampling and Partial Alignment Distillation](https://arxiv.org/abs/2408.15562) <br> Lujun Gui, Bin Xiao, Lei Su, Weipeng Chen |<img width="1002" alt="image" src="https://arxiv.org/html/2408.15562v1/extracted/5818109/structure_0.png"> |[Paper](https://arxiv.org/abs/2408.15562)|[//]: #09/02
|[Path-Consistency: Prefix Enhancement for Efficient Inference in LLM](https://arxiv.org/abs/2409.01281) <br> Jiace Zhu, Yingtao Shen, Jie Zhao, An Zou |<img width="1002" alt="image" src="https://arxiv.org/html/2409.01281v1/x1.png"> |[Paper](https://arxiv.org/abs/2409.01281)|[//]: #09/06
|[![Star](https://img.shields.io/github/stars/zjunlp/OneGen.svg?style=social&label=Star)](https://github.com/zjunlp/OneGen)<br>[OneGen: Efficient One-Pass Unified Generation and Retrieval for LLMs](https://arxiv.org/abs/2409.05152) <br> Jintian Zhang, Cheng Peng, Mengshu Sun, Xiang Chen, Lei Liang, Zhiqiang Zhang, Jun Zhou, Huajun Chen, Ningyu Zhang |<img width="1002" alt="image" src="https://github.com/zjunlp/OneGen/blob/main/assets/train.jpg"> |[Github](https://github.com/zjunlp/OneGen) <br> [Paper](https://arxiv.org/abs/2409.05152)|[//]: #09/13
|[![Star](https://img.shields.io/github/stars/Infini-AI-Lab/Sirius.svg?style=social&label=Star)](https://github.com/Infini-AI-Lab/Sirius)<br>[Sirius: Contextual Sparsity with Correction for Efficient LLMs](https://arxiv.org/abs/2409.03856) <br> Yang Zhou, Zhuoming Chen, Zhaozhuo Xu, Victoria Lin, Beidi Chen |<img width="1002" alt="image" src="https://infini-ai-lab.github.io/Sirius/static/images/methodsillustration.png"> |[Github](https://github.com/Infini-AI-Lab/Sirius) <br> [Paper](https://arxiv.org/abs/2409.03856)|[//]: #09/13
|[RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval](https://arxiv.org/abs/2409.10516) <br> Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chengruidong Zhang, Bailu Ding, Kai Zhang, Chen Chen, Fan Yang, Yuqing Yang, Lili Qiu |<img width="1002" alt="image" src="https://arxiv.org/html/2409.10516v2/x4.png"> |[Paper](https://arxiv.org/abs/2409.10516)|[//]: #09/21
|[![Star](https://img.shields.io/github/stars/66RING/CritiPrefill.svg?style=social&label=Star)](https://github.com/66RING/CritiPrefill)<br>[CritiPrefill: A Segment-wise Criticality-based Approach for Prefilling Acceleration in LLMs](https://arxiv.org/abs/2409.12490) <br> Junlin Lv, Yuan Feng, Xike Xie, Xin Jia, Qirong Peng, Guiming Xie |<img width="1002" alt="image" src="https://arxiv.org/html/2409.12490v1/x2.png"> |[Github](https://github.com/66RING/CritiPrefill) <br> [Paper](https://arxiv.org/abs/2409.12490)|[//]: #09/21
|[Dynamic-Width Speculative Beam Decoding for Efficient LLM Inference](https://arxiv.org/abs/2409.16560) <br> Zongyue Qin, Zifan He, Neha Prakriya, Jason Cong, Yizhou Sun |<img width="1002" alt="image" src="https://arxiv.org/html/2409.16560v1/x6.png"> |[Paper](https://arxiv.org/abs/2409.16560)|[//]: #09/27
|[![Star](https://img.shields.io/github/stars/SalesforceAIResearch/GemFilter.svg?style=social&label=Star)](https://github.com/SalesforceAIResearch/GemFilter)<br>[Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction](https://arxiv.org/abs/2409.17422) <br> Zhenmei Shi, Yifei Ming, Xuan-Phi Nguyen, Yingyu Liang, Shafiq Joty |<img width="1002" alt="image" src="https://arxiv.org/html/2409.17422v1/x1.png"> |[Github](https://github.com/SalesforceAIResearch/GemFilter) <br> [Paper](https://arxiv.org/abs/2409.17422)|[//]: #09/27
|[Mnemosyne: Parallelization Strategies for Efficiently Serving Multi-Million Context Length LLM Inference Requests Without Approximations](https://arxiv.org/abs/2409.17264) <br> Amey Agrawal, Junda Chen, Íñigo Goiri, Ramachandran Ramjee, Chaojie Zhang, Alexey Tumanov, Esha Choukse |<img width="1002" alt="image" src="https://arxiv.org/html/2409.17264v1/x14.png"> |[Paper](https://arxiv.org/abs/2409.17264)|[//]: #09/27
|[A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts](https://arxiv.org/abs/2410.01485) <br> Suyu Ge, Xihui Lin, Yunan Zhang, Jiawei Han, Hao Peng |<img width="1002" alt="image" src="https://arxiv.org/html/2410.01485v1/extracted/5895696/figures/model_architecture.png"> |[Paper](https://arxiv.org/abs/2410.01485)|[//]: #10/04
|[![Star](https://img.shields.io/github/stars/MooreThreads/TurboRAG.svg?style=social&label=Star)](https://github.com/MooreThreads/TurboRAG)<br>[TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text](https://arxiv.org/abs/2410.07590) <br> Songshuo Lu, Hua Wang, Yutian Rong, Zhi Chen, Yaohua Tang |<img width="1002" alt="image" src="https://github.com/MooreThreads/TurboRAG/raw/main/assets/image/TurboRAG.png"> |[Github](https://github.com/MooreThreads/TurboRAG) <br> [Paper](https://arxiv.org/abs/2410.07590)|[//]: #10/13
|[![Star](https://img.shields.io/github/stars/hemingkx/SWIFT.svg?style=social&label=Star)](https://github.com/hemingkx/SWIFT)<br>[SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration](https://arxiv.org/abs/2410.06916) <br> Heming Xia, Yongqi Li, Jun Zhang, Cunxiao Du, Wenjie Li |<img width="1002" alt="image" src="https://github.com/hemingkx/SWIFT/raw/main/assets/swift.png"> |[Github](https://github.com/hemingkx/SWIFT) <br> [Paper](https://arxiv.org/abs/2410.06916)|[//]: #10/14
|[ParallelSpec: Parallel Drafter for Efficient Speculative Decoding](https://arxiv.org/abs/2410.05589) <br> Zilin Xiao, Hongming Zhang, Tao Ge, Siru Ouyang, Vicente Ordonez, Dong Yu |<img width="1002" alt="image" src="https://arxiv.org/html/2410.05589v1/x1.png"> |[Paper](https://arxiv.org/abs/2410.05589)|[//]: #10/14
|[TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attention](https://arxiv.org/abs/2410.05076) <br> Lijie Yang, Zhihao Zhang, Zhuofu Chen, Zikun Li, Zhihao Jia |<img width="1002" alt="image" src="https://arxiv.org/html/2410.05076v1/x2.png"> |[Paper](https://arxiv.org/abs/2410.05076)|[//]: #10/14
|[DySpec: Faster Speculative Decoding with Dynamic Token Tree Structure](https://arxiv.org/abs/2410.11744) <br> Yunfan Xiong, Ruoyu Zhang, Yanzeng Li, Tianhao Wu, Lei Zou |<img width="1002" alt="image" src="https://arxiv.org/html/2410.11744v1/extracted/5913908/figures/tree_bold.png"> |[Paper](https://arxiv.org/abs/2410.11744)|[//]: #10/21
|[QSpec: Speculative Decoding with Complementary Quantization Schemes](https://arxiv.org/abs/2410.11305) <br> Juntao Zhao, Wenhao Lu, Sheng Wang, Lingpeng Kong, Chuan Wu |<img width="1002" alt="image" src="https://arxiv.org/html/2410.11305v1/x1.png"> |[Paper](https://arxiv.org/abs/2410.11305)|[//]: #10/21
|[![Star](https://img.shields.io/github/stars/mit-han-lab/duo-attention.svg?style=social&label=Star)](https://github.com/mit-han-lab/duo-attention)<br>[DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads](https://arxiv.org/abs/2410.10819) <br> Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, Song Han |<img width="1002" alt="image" src="https://github.com/mit-han-lab/duo-attention/raw/main/figures/method1.jpg"> |[Github](https://github.com/mit-han-lab/duo-attention) <br> [Paper](https://arxiv.org/abs/2410.10819)|[//]: #10/21
|[![Star](https://img.shields.io/github/stars/LMCache/LMCache.svg?style=social&label=Star)](https://github.com/LMCache/LMCache)[![Publish](https://img.shields.io/badge/Conference-EuroSys'25-red)]()<br>[CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion](https://arxiv.org/abs/2405.16444) <br> Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, Junchen Jiang |<img width="600" alt="image" src="figures/cacheblend-pic.png"> |[Github](https://github.com/LMCache/LMCache) <br> [Paper](https://arxiv.org/abs/2405.16444)|[//]: #10/23
|[![Star](https://img.shields.io/github/stars/wangqinsi1/CoreInfer.svg?style=social&label=Star)](https://github.com/wangqinsi1/CoreInfer)<br>[CoreInfer: Accelerating Large Language Model Inference with Semantics-Inspired Adaptive Sparse Activation](https://arxiv.org/abs/2410.18311#) <br> Qinsi Wang, Saeed Vahidian, Hancheng Ye, Jianyang Gu, Jianyi Zhang, Yiran Chen |<img width="1002" alt="image" src="https://wangqinsi1.github.io/coreinfer_page/static/images/overview.png"> |[Github](https://github.com/wangqinsi1/CoreInfer) <br> [Paper](https://arxiv.org/abs/2410.18311#)|[//]: #10/29
|[![Star](https://img.shields.io/github/stars/MatteoNulli/Vocabulary_pruning.svg?style=social&label=Star)](https://github.com/MatteoNulli/Vocabulary_pruning)<br>[Dynamic Vocabulary Pruning in Early-Exit LLMs](https://arxiv.org/abs/2410.18952) <br> Jort Vincenti, Karim Abdel Sadek, Joan Velja, Matteo Nulli, Metod Jazbec |<img width="1002" alt="image" src="https://github.com/MatteoNulli/Vocabulary_pruning/raw/main/src/images/final_nips.svg"> |[Github](https://github.com/MatteoNulli/Vocabulary_pruning) <br> [Paper](https://arxiv.org/abs/2410.18952)|[//]: #10/29
|[Efficient Inference for Augmented Large Language Models](https://arxiv.org/abs/2410.18248) <br> Rana Shahout, Cong Liang, Shiji Xin, Qianru Lao, Yong Cui, Minlan Yu, Michael Mitzenmacher |<img width="1002" alt="image" src="https://arxiv.org/html/2410.18248v1/extracted/5949546/figures/illustrations/api_example_png.png"> |[Paper](https://arxiv.org/abs/2410.18248)|[//]: #10/29
|[Faster Language Models with Better Multi-Token Prediction Using Tensor Decomposition](https://arxiv.org/abs/2410.17765) <br> Artem Basharin, Andrei Chertkov, Ivan Oseledets |<img width="1002" alt="image" src="figures/canonical_tensor_decomposition.png"> |[Paper](https://arxiv.org/abs/2410.17765)|[//]: #10/29
|[![Star](https://img.shields.io/github/stars/Infini-AI-Lab/MagicPIG.svg?style=social&label=Star)](https://github.com/Infini-AI-Lab/MagicPIG)<br>[MagicPIG: LSH Sampling for Efficient LLM Generation](https://arxiv.org/abs/2410.16179) <br> Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, Yuandong Tian, Matthijs Douze, Leon Bottou, Zhihao Jia, Beidi Chen |<img width="1002" alt="image" src="https://arxiv.org/html/2410.16179v2/x15.png"> |[Github](https://github.com/Infini-AI-Lab/MagicPIG) <br> [Paper](https://arxiv.org/abs/2410.16179)|[//]: #10/30
|[Dynamic Strategy Planning for Efficient Question Answering with Large Language Models](https://arxiv.org/abs/2410.23511) <br> Tanmay Parekh, Pradyot Prakash, Alexander Radovic, Akshay Shekher, Denis Savenkov |<img width="1002" alt="image" src="https://arxiv.org/html/2410.23511v1/x1.png"> |[Paper](https://arxiv.org/abs/2410.23511)|[//]: #11/17
|[SuffixDecoding: A Model-Free Approach to Speeding Up Large Language Model Inference](https://arxiv.org/abs/2411.04975) <br> Gabriele Oliaro, Zhihao Jia, Daniel Campos, Aurick Qiao |<img width="1002" alt="image" src="https://arxiv.org/html/2411.04975v1/x1.png"> |[Paper](https://arxiv.org/abs/2411.04975)|[//]: #11/18
|[The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculation](https://arxiv.org/abs/2411.03786) <br> Lawrence Stewart, Matthew Trager, Sujan Kumar Gonugondla, Stefano Soatto | |[Paper](https://arxiv.org/abs/2411.03786)|[//]: #11/18
|[Accelerated AI Inference via Dynamic Execution Methods](https://arxiv.org/abs/2411.00853) <br> Haim Barad, Jascha Achterberg, Tien Pei Chou, Jean Yu | |[Paper](https://arxiv.org/abs/2411.00853)|[//]: #11/18
|[![Star](https://img.shields.io/github/stars/David-Li0406/SMoA.svg?style=social&label=Star)](https://github.com/David-Li0406/SMoA)<br>[SMoA: Improving Multi-agent Large Language Models with Sparse Mixture-of-Agents](https://arxiv.org/abs/2411.03284) <br> Dawei Li, Zhen Tan, Peijia Qian, Yifan Li, Kumar Satvik Chaudhary, Lijie Hu, Jiayi Shen |<img width="1002" alt="image" src="figures/SMoA.png"> |[Github](https://github.com/David-Li0406/SMoA) <br> [Paper](https://arxiv.org/abs/2411.03284)|[//]: #11/18
|[![Publish](https://img.shields.io/badge/Conference-NeurIPS'24%20ENLSP-blue)]()<br>[FastDraft: How to Train Your Draft](https://arxiv.org/abs/2411.11055) <br> Ofir Zafrir, Igor Margulis, Dorin Shteyman, Guy Boudoukh | |[Paper](https://arxiv.org/abs/2411.11055)|[//]: #11/24
|[PLD+: Accelerating LLM inference by leveraging Language Model Artifacts](https://arxiv.org/abs/2412.01447) <br> Shwetha Somasundaram, Anirudh Phukan, Apoorv Saxena |<img width="1002" alt="image" src="https://arxiv.org/html/2412.01447v1/x1.png"> |[Paper](https://arxiv.org/abs/2412.01447)|[//]: #12/09
|[Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration](https://arxiv.org/abs/2412.00061) <br> Zhuofan Wen, Shangtong Gui, Yang Feng |<img width="302" alt="image" src="https://arxiv.org/html/2412.00061v1/x1.png"> |[Paper](https://arxiv.org/abs/2412.00061)|[//]: #12/09