
## Efficient MOE
| Title & Authors | Introduction | Links |
|:--|  :----: | :---:|
|[SiDA: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models](https://arxiv.org/abs/2310.18859) <br> Zhixu Du, Shiyu Li, Yuhao Wu, Xiangyu Jiang, Jingwei Sun, Qilin Zheng, Yongkai Wu, Ang Li, Hai "Helen" Li, Yiran Chen |<img width="1002" alt="image" src="figures/SiDA.png"> |[Paper](https://arxiv.org/abs/2310.18859)|
|[![Star](https://img.shields.io/github/stars/dvmazur/mixtral-offloading.svg?style=social&label=Star)](https://github.com/dvmazur/mixtral-offloading)<br>[Fast Inference of Mixture-of-Experts Language Models with Offloading](https://arxiv.org/abs/2312.17238) <br> Artyom Eliseev, Denis Mazur |<img width="1002" alt="image" src="figures/mixtral_offloading.png"> |[Github](https://github.com/dvmazur/mixtral-offloading) <br> [Paper](https://arxiv.org/abs/2312.17238)|
|[![Star](https://img.shields.io/github/stars/robertcsordas/moe_attention.svg?style=social&label=Star)](https://github.com/robertcsordas/moe_attention)<br>[SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention](https://arxiv.org/abs/2312.07987) <br> Róbert Csordás, Piotr Piękos, Kazuki Irie, Jürgen Schmidhuber |<img width="1002" alt="image" src="figures/switchhead.png"> |[Github](https://github.com/robertcsordas/moe_attention) <br> [Paper](https://arxiv.org/abs/2312.07987)|
|[![Star](https://img.shields.io/github/stars/YJHMITWEB/ExFlow.svg?style=social&label=Star)](https://github.com/YJHMITWEB/ExFlow)<br>[Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference](https://arxiv.org/abs/2401.08383) <br> Jinghan Yao, Quentin Anthony, Aamir Shafi, Hari Subramoni, Dhabaleswar K. (DK)Panda |<img width="1002" alt="image" src="figures/exflow.png"> |[Github](https://github.com/YJHMITWEB/ExFlow) <br> [Paper](https://arxiv.org/abs/2401.08383)|
|[![Star](https://img.shields.io/github/stars/TorchMoE/MoE-Infinity.svg?style=social&label=Star)](https://github.com/TorchMoE/MoE-Infinity)<br>[MoE-Infinity: Activation-Aware Expert Offloading for Efficient MoE Serving](https://arxiv.org/abs/2401.14361) <br> Leyang Xue, Yao Fu, Zhan Lu, Luo Mai, Mahesh Marina |<img width="1002" alt="image" src="figures/MOE-Infinity.png"> |[Github](https://github.com/TorchMoE/MoE-Infinity) <br> [Paper](https://arxiv.org/abs/2401.14361)|
|[![Star](https://img.shields.io/github/stars/efeslab/fiddler.svg?style=social&label=Star)](https://github.com/efeslab/fiddler)<br>[Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models](https://arxiv.org/abs/2402.07033) <br> Keisuke Kamahori, Yile Gu, Kan Zhu, Baris Kasikci |<img width="1002" alt="image" src="https://github.com/efeslab/fiddler/blob/main/asset/key-idea.png"> |[Github](https://github.com/efeslab/fiddler) <br> [Paper](https://arxiv.org/abs/2402.07033)|
|[![Star](https://img.shields.io/github/stars/Lucky-Lance/Expert_Sparsity.svg?style=social&label=Star)](https://github.com/Lucky-Lance/Expert_Sparsity)<br>[Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models](https://arxiv.org/abs/2402.14800) <br> Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, Hongsheng Li |<img width="1002" alt="image" src="https://arxiv.org/html/2402.14800v1/x2.png"> |[Github](https://github.com/Lucky-Lance/Expert_Sparsity) <br> [Paper](https://arxiv.org/abs/2402.14800)|
|[Enhancing Efficiency in Sparse Models with Sparser Selection](https://arxiv.org/abs/2403.18926) <br> Yuanhang Yang, Shiyi Qi, Wenchao Gu, Chaozheng Wang, Cuiyun Gao, Zenglin Xu |<img width="1002" alt="image" src="https://arxiv.org/html/2403.18926v1/x3.png"> |[Github](https://anonymous.4open.science/r/XMoE) <br> [Paper](https://arxiv.org/abs/2403.18926)|
|[![Star](https://img.shields.io/github/stars/hdong920/GRIFFIN.svg?style=social&label=Star)](https://github.com/hdong920/GRIFFIN)<br>[Prompt-prompted Mixture of Experts for Efficient LLM Generation](https://arxiv.org/abs/2404.01365) <br> Harry Dong, Beidi Chen, Yuejie Chi |<img width="1002" alt="image" src="https://arxiv.org/html/2404.01365v1/extracted/5509263/figures/algorithm.png"> |[Github](https://github.com/hdong920/GRIFFIN) <br> [Paper](https://arxiv.org/abs/2404.01365)|
|[Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts](https://arxiv.org/abs/2404.05019) <br> Weilin Cai, Juyong Jiang, Le Qin, Junwei Cui, Sunghun Kim, Jiayi Huang |<img width="1002" alt="image" src="https://arxiv.org/html/2404.05019v1/x1.png"> |[Paper](https://arxiv.org/abs/2404.05019)|
|[SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts](https://arxiv.org/abs/2404.05089) <br> Alexandre Muzio, Alex Sun, Churan He |<img width="1002" alt="image" src="https://arxiv.org/html/2404.05089v1/x1.png"> |[Paper](https://arxiv.org/abs/2404.05089)|
|[Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models](https://arxiv.org/abs/2404.05567) <br> Bowen Pan, Yikang Shen, Haokun Liu, Mayank Mishra, Gaoyuan Zhang, Aude Oliva, Colin Raffel, Rameswar Panda |<img width="1002" alt="image" src="https://arxiv.org/html/2404.05567v1/x2.png"> |[Paper](https://arxiv.org/abs/2404.05567)|
|[![Publish](https://img.shields.io/badge/Conference-MLsys'24-blue)]()<br>[Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlapping](https://arxiv.org/abs/2404.19429) <br> Chenyu Jiang, Ye Tian, Zhen Jia, Shuai Zheng, Chuan Wu, Yida Wang |<img width="1002" alt="image" src="https://arxiv.org/html/2404.19429v1/x4.png"> |[Paper](https://arxiv.org/abs/2404.19429)|
|[![Publish](https://img.shields.io/badge/Conference-ICML'24-blue)]()<br>[A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts](https://arxiv.org/pdf/2405.16646) <br> Mohammed Nowaz Rabbani Chowdhury, Meng Wang, Kaoutar El Maghraoui, Naigang Wang, Pin-Yu Chen, Christopher Carothers |<img width="1002" alt="image" src="https://arxiv.org/html/2405.16646v2/extracted/5626402/Fig/token_expert_combined_2.png"> |[Paper](https://arxiv.org/pdf/2405.16646)|
|[![Star](https://img.shields.io/github/stars/LINs-lab/DynMoE.svg?style=social&label=Star)](https://github.com/LINs-lab/DynMoE)<br>[Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models](https://arxiv.org/abs/2405.14297) <br> Yongxin Guo, Zhenglin Cheng, Xiaoying Tang, Tao Lin |<img width="1002" alt="image" src="figures/dynmoe.png"> |[Github](https://github.com/LINs-lab/DynMoE) <br> [Paper](https://arxiv.org/abs/2405.14297)|
|[![Publish](https://img.shields.io/badge/Conference-DAC'24-blue)]()<br>[MoNDE: Mixture of Near-Data Experts for Large-Scale Sparse Models](https://arxiv.org/abs/2405.18832) <br> Taehyun Kim, Kwanseok Choi, Youngmock Cho, Jaehoon Cho, Hyuk-Jae Lee, Jaewoong Sim |<img width="1002" alt="image" src="https://arxiv.org/html/2405.18832v1/x4.png"> |[Paper](https://arxiv.org/abs/2405.18832)|
|[![Star](https://img.shields.io/github/stars/DaizeDong/Unified-MoE-Compression.svg?style=social&label=Star)](https://github.com/DaizeDong/Unified-MoE-Compression)<br>[Demystifying the Compression of Mixture-of-Experts Through a Unified Framework](https://arxiv.org/abs/2406.02500) <br> Shwai He, Daize Dong, Liang Ding, Ang Li |<img width="1002" alt="image" src="https://arxiv.org/html/2406.02500v1/x1.png"> |[Github](https://github.com/DaizeDong/Unified-MoE-Compression) <br> [Paper](https://arxiv.org/abs/2406.02500)|
|[ME-Switch: A Memory-Efficient Expert Switching Framework for Large Language Models](https://arxiv.org/abs/2406.09041) <br> Jing Liu, Ruihao Gong, Mingyang Zhang, Yefei He, Jianfei Cai, Bohan Zhuang |<img width="1002" alt="image" src="https://arxiv.org/html/2406.09041v1/x1.png"> |[Paper](https://arxiv.org/abs/2406.09041)|[//]: #06/18
|[![Star](https://img.shields.io/github/stars/UNITES-Lab/moe-quantization.svg?style=social&label=Star)](https://github.com/UNITES-Lab/moe-quantization)<br>[Examining Post-Training Quantization for Mixture-of-Experts: A Benchmark](https://arxiv.org/abs/2406.08155) <br> Pingzhi Li, Xiaolong Jin, Yu Cheng, Tianlong Chen |<img width="1002" alt="image" src="https://arxiv.org/html/2406.08155v1/x1.png"> |[Github](https://github.com/UNITES-Lab/moe-quantization) <br> [Paper](https://arxiv.org/abs/2406.08155)|[//]: #06/18
|[![Star](https://img.shields.io/github/stars/imagination-research/EEP.svg?style=social&label=Star)](https://github.com/imagination-research/EEP)<br>[Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs](https://arxiv.org/abs/2407.00945) <br> Enshu Liu, Junyi Zhu, Zinan Lin, Xuefei Ning, Matthew B. Blaschko, Shengen Yan, Guohao Dai, Huazhong Yang, Yu Wang |<img width="1002" alt="image" src="https://arxiv.org/html/2407.00945v1/extracted/5697370/Figures/use_case.png"> |[Github](https://github.com/imagination-research/EEP) <br> [Paper](https://arxiv.org/abs/2407.00945)|[//]: #07/03
|[Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts](https://arxiv.org/abs/2407.09590) <br> Zeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu, Jianfeng Gao |<img width="1002" alt="image" src="https://arxiv.org/html/2407.09590v1/x3.png"> |[Paper](https://arxiv.org/abs/2407.09590)|[//]: #07/16
|[![Star](https://img.shields.io/github/stars/Aaronhuang-778/MC-MoE.svg?style=social&label=Star)](https://github.com/Aaronhuang-778/MC-MoE)<br>[MC-MoE: Mixture Compressor for Mixture-of-Experts LLMs Gains More](https://arxiv.org/abs/2410.06270) <br> Wei Huang, Yue Liao, Jianhui Liu, Ruifei He, Haoru Tan, Shiming Zhang, Hongsheng Li, Si Liu, Xiaojuan Qi |<img width="1002" alt="image" src="https://github.com/Aaronhuang-778/MC-MoE/raw/main/imgs/WX20241009-191322@2x.png"> |[Github](https://github.com/Aaronhuang-778/MC-MoE) <br> [Paper](https://arxiv.org/abs/2410.06270)|[//]: #10/14
|[EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference](https://arxiv.org/abs/2410.12247) <br> Yulei Qian, Fengcun Li, Xiangyang Ji, Xiaoyu Zhao, Jianchao Tan, Kefeng Zhang, Xunliang Cai | |[Paper](https://arxiv.org/abs/2410.12247)|[//]: #10/21
|[ExpertFlow: Optimized Expert Activation and Token Allocation for Efficient Mixture-of-Experts Inference](https://arxiv.org/abs/2410.17954) <br> Xin He, Shunkang Zhang, Yuxin Wang, Haiyan Yin, Zihao Zeng, Shaohuai Shi, Zhenheng Tang, Xiaowen Chu, Ivor Tsang, Ong Yew Soon |<img width="202" alt="image" src="https://arxiv.org/html/2410.17954v1/x1.png"> |[Paper](https://arxiv.org/abs/2410.17954)|[//]: #10/29
|[ProMoE: Fast MoE-based LLM Serving using Proactive Caching](https://arxiv.org/abs/2410.22134) <br> Xiaoniu Song, Zihang Zhong, Rong Chen |<img width="1002" alt="image" src="https://arxiv.org/html/2410.22134v1/x1.png"> |[Paper](https://arxiv.org/abs/2410.22134)|[//]: #11/17
|[HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference](https://arxiv.org/abs/2411.01433) <br> Peng Tang, Jiacheng Liu, Xiaofeng Hou, Yifei Pu, Jing Wang, Pheng-Ann Heng, Chao Li, Minyi Guo |<img width="1002" alt="image" src="https://arxiv.org/html/2411.01433v2/extracted/5980843/figures/overview5.png"> |[Paper](https://arxiv.org/abs/2411.01433)|[//]: #11/18
|[![Star](https://img.shields.io/github/stars/EnflameTechnology/DeepSpeed.svg?style=social&label=Star)](https://github.com/EnflameTechnology/DeepSpeed)<br>[MoNTA: Accelerating Mixture-of-Experts Training with Network-Traffc-Aware Parallel Optimization](https://arxiv.org/abs/2411.00662) <br> Jingming Guo, Yan Liu, Yu Meng, Zhiwei Tao, Banglan Liu, Gang Chen, Xiang Li |<img width="1002" alt="image" src="https://arxiv.org/html/2411.00662v1/x1.png"> |[Github](https://github.com/EnflameTechnology/DeepSpeed) <br> [Paper](https://arxiv.org/abs/2411.00662)|[//]: #11/18
|[![Star](https://img.shields.io/github/stars/xiaochengsky/MoEI-2.svg?style=social&label=Star)](https://github.com/xiaochengsky/MoEI-2)<br>[MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition](https://arxiv.org/abs/2411.01016) <br> Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Yuanlin Duan, Wenqi Jia, Miao Yin, Yu Cheng, Bo Yuan |<img width="1002" alt="image" src="https://arxiv.org/html/2411.01016v1/x1.png"> |[Github](https://github.com/xiaochengsky/MoEI-2) <br> [Paper](https://arxiv.org/abs/2411.01016)|[//]: #11/18
|[Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference](https://arxiv.org/abs/2412.00099) <br> Andrii Skliar, Ties van Rozendaal, Romain Lepert, Todor Boinovski, Mart van Baalen, Markus Nagel, Paul Whatmough, Babak Ehteshami Bejnordi |<img width="1002" alt="image" src="figures/cacheprior.png"> |[Paper](https://arxiv.org/abs/2412.00099)|
|[Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference](https://arxiv.org/abs/2412.00099) <br> Andrii Skliar, Ties van Rozendaal, Romain Lepert, Todor Boinovski, Mart van Baalen, Markus Nagel, Paul Whatmough, Babak Ehteshami Bejnordi |<img width="1002" alt="image" src="https://arxiv.org/html/2412.00099v1/x1.png"> |[Paper](https://arxiv.org/abs/2412.00099)|[//]: #12/09
|[![Star](https://img.shields.io/github/stars/duterscmy/CD-MoE.svg?style=social&label=Star)](https://github.com/duterscmy/CD-MoE)<br>[Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruning](https://arxiv.org/abs/2412.00069) <br> Mingyu Cao, Gen Li, Jie Ji, Jiaqi Zhang, Xiaolong Ma, Shiwei Liu, Lu Yin |<img width="1002" alt="image" src="https://arxiv.org/html/2412.00069v1/x2.png"> |[Github](https://github.com/duterscmy/CD-MoE) <br> [Paper](https://arxiv.org/abs/2412.00069)|[//]: #12/09