
## Network Pruning
| Title & Authors | Introduction | Links |
|:----|  :----: | :---:|
| [![Star](https://img.shields.io/github/stars/IST-DASLab/sparsegpt.svg?style=social&label=Star)](https://github.com/IST-DASLab/sparsegpt) [![Publish](https://img.shields.io/badge/Conference-ICML'23-blue)]() [![Type](https://img.shields.io/badge/Unstructured-C2A4A6)]() <br> [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://github.com/IST-DASLab/sparsegpt) <br> Elias Frantar, Dan Alistarh| <img width="522" alt="image" src="figures/sparsegpt.png"> |[Github](https://github.com/IST-DASLab/sparsegpt) [paper](https://arxiv.org/abs/2301.00774) |
| [![Star](https://img.shields.io/github/stars/horseee/LLM-Pruner.svg?style=social&label=Star)](https://github.com/horseee/LLM-Pruner) [![Publish](https://img.shields.io/badge/Conference-NeurIPS'23-blue)]() [![Type](https://img.shields.io/badge/Structural-C2A4A6)]() <br>[LLM-Pruner: On the Structural Pruning of Large Language Models](https://arxiv.org/abs/2305.11627) <br> Xinyin Ma, Gongfan Fang, Xinchao Wang | <img width="561" alt="image" src="figures/llm_pruner.png">| [Github](https://github.com/horseee/LLM-Pruner) [paper](https://arxiv.org/abs/2305.11627)|
|[![Star](https://img.shields.io/github/stars/VITA-Group/essential_sparsity.svg?style=social&label=Star)](https://github.com/VITA-Group/essential_sparsity) [![Publish](https://img.shields.io/badge/Conference-NeurIPS'23-blue)]() [![Type](https://img.shields.io/badge/Unstructured-C2A4A6)]() <br>[The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter](https://arxiv.org/abs/2306.03805) <br> Ajay Jaiswal, Shiwei Liu, Tianlong Chen, Zhangyang Wang |<img width="1002" alt="image" src="https://user-images.githubusercontent.com/6660499/243539825-ca3b1dbe-bc1c-45d9-a6ea-d1d0c991e997.png"> |[Github](https://github.com/VITA-Group/essential_sparsity) <br> [Paper](https://arxiv.org/abs/2306.03805)|
|[![Star](https://img.shields.io/github/stars/AlibabaResearch/flash-llm.svg?style=social&label=Star)](https://github.com/AlibabaResearch/flash-llm)[![Publish](https://img.shields.io/badge/Conference-VLDB'24-blue)]() [![Type](https://img.shields.io/badge/Unstructured-C2A4A6)]() <br>[Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity](https://arxiv.org/abs/2309.10285) <br> Haojun Xia, Zhen Zheng, Yuchao Li, Donglin Zhuang, Zhongzhu Zhou, Xiafei Qiu, Yong Li, Wei Lin, Shuaiwen Leon Song |<img width="602" alt="image" src="figures/FlashLLM.png"> |[Github](https://github.com/AlibabaResearch/flash-llm) <br> [Paper](https://arxiv.org/abs/2309.10285)|
|[![Star](https://img.shields.io/github/stars/locuslab/wanda.svg?style=social&label=Star)](https://github.com/locuslab/wanda) [![Publish](https://img.shields.io/badge/Conference-ICLR'24-blue)]() [![Type](https://img.shields.io/badge/Unstructured-C2A4A6)]()  <br>[A Simple and Effective Pruning Approach for Large Language Models](https://arxiv.org/abs/2306.11695) <br> Mingjie Sun, Zhuang Liu, Anna Bair, J. Zico Kolter |<img width="1002" alt="image" src="https://user-images.githubusercontent.com/20168304/245999360-f951de47-269d-491d-826a-8e6d85627849.png"> |[Github](https://github.com/locuslab/wanda) <br> [Paper](https://arxiv.org/abs/2306.11695)|
|[![Star](https://img.shields.io/github/stars/princeton-nlp/LLM-Shearing.svg?style=social&label=Star)](https://github.com/princeton-nlp/LLM-Shearing) [![Publish](https://img.shields.io/badge/Conference-ICLR'24-blue)]() [![Type](https://img.shields.io/badge/Structural-C2A4A6)]() <br>[Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning](https://arxiv.org/abs/2310.06694) <br> Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, Danqi Chen |<img width="1002" alt="image" src="figures/LLM-shearing.png"> |[Github](https://github.com/princeton-nlp/LLM-Shearing) <br> [Paper](https://arxiv.org/abs/2310.06694)|
|[![Star](https://img.shields.io/github/stars/biomedical-cybernetics/Relative-importance-and-activation-pruning.svg?style=social&label=Star)](https://github.com/biomedical-cybernetics/Relative-importance-and-activation-pruning)[![Publish](https://img.shields.io/badge/Conference-ICLR'24-blue)]()<br>[Plug-and-Play: An Efficient Post-training Pruning Method for Large Language Models](https://openreview.net/forum?id=Tr0lPx9woF) <br> Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, Carlo Vittorio Cannistraci |<img width="1002" alt="image" src="figures/RIA.png"> |[Github](https://github.com/biomedical-cybernetics/Relative-importance-and-activation-pruning) <br> [Paper](https://openreview.net/forum?id=Tr0lPx9woF)|
|[![Star](https://img.shields.io/github/stars/CASIA-IVA-Lab/FLAP.svg?style=social&label=Star)](https://github.com/CASIA-IVA-Lab/FLAP)[![Publish](https://img.shields.io/badge/Conference-AAAI'24-blue)]() [![Type](https://img.shields.io/badge/Structural-C2A4A6)]()<br>[Fluctuation-based Adaptive Structured Pruning for Large Language Models](https://arxiv.org/abs/2312.11983) <br> Yongqi An, Xu Zhao, Tao Yu, Ming Tang, Jinqiao Wang |<img width="1002" alt="image" src="https://github.com/CASIA-IVA-Lab/FLAP/raw/main/figures/overview.png"> |[Github](https://github.com/CASIA-IVA-Lab/FLAP) <br> [Paper](https://arxiv.org/abs/2312.11983)|
|[![Star](https://img.shields.io/github/stars/jongwooko/NASH-Pruning-Official.svg?style=social&label=Star)](https://github.com/jongwooko/NASH-Pruning-Official)[![Publish](https://img.shields.io/badge/Conference-EMNLP'23%20Findings-blue)]() [![Type](https://img.shields.io/badge/Structural-C2A4A6)]() <br>[NASH: A Simple Unified Framework of Structured Pruning for Accelerating Encoder-Decoder Language Models](https://arxiv.org/abs/2310.10054) <br> Jongwoo Ko, Seungjoon Park, Yujin Kim, Sumyeong Ahn, Du-Seong Chang, Euijai Ahn, Se-Young Yun |<img width="402" alt="image" src="figures/NASH.png"> |[Github](https://github.com/jongwooko/NASH-Pruning-Official) <br> [Paper](https://arxiv.org/abs/2310.10054)|
|[![Star](https://img.shields.io/github/stars/aim-uofa/LoRAPrune.svg?style=social&label=Star)](https://github.com/aim-uofa/LoRAPrune)[![Publish](https://img.shields.io/badge/Conference-ACL'24%20Findings-blue)]()<br>[LoRAPrune: Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning](https://arxiv.org/abs/2305.18403) <br> Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, Bohan Zhuang |<img width="1002" alt="image" src="figures/LoRAPrune.png"> |[Github](https://github.com/aim-uofa/LoRAPrune) <br> [Paper](https://arxiv.org/abs/2305.18403)|
|[![Type](https://img.shields.io/badge/Structural-C2A4A6)]() <br> [Pruning Large Language Models via Accuracy Predictor](https://arxiv.org/abs/2309.09507) <br> Yupeng Ji, Yibo Cao, Jiucai Liu |<img width="202" alt="image" src="figures/PruningAccuracyPredictor.png"> |[Paper](https://arxiv.org/abs/2309.09507)|
|[![Type](https://img.shields.io/badge/Benchmark-C2A4A6)]()<br> [Compressing LLMs: The Truth is Rarely Pure and Never Simple](https://arxiv.org/abs/2310.01382) <br> Ajay Jaiswal, Zhe Gan, Xianzhi Du, Bowen Zhang, Zhangyang Wang, Yinfei Yang |<img width="1002" alt="image" src="figures/LLM-KICK.png"> |[Paper](https://arxiv.org/abs/2310.01382)|
|[![Star](https://img.shields.io/github/stars/VITA-Group/Junk_DNA_Hypothesis.svg?style=social&label=Star)](https://github.com/VITA-Group/Junk_DNA_Hypothesis)[![Type](https://img.shields.io/badge/Unstructured-C2A4A6)]()<br>[Junk DNA Hypothesis: A Task-Centric Angle of LLM Pre-trained Weights through Sparsity](https://arxiv.org/abs/2310.02277) <br> Lu Yin, Shiwei Liu, Ajay Jaiswal, Souvik Kundu, Zhangyang Wang |<img width="1002" alt="image" src="figures/junk_DNA.png"> |[Github](https://github.com/VITA-Group/Junk_DNA_Hypothesis) <br> [Paper](https://arxiv.org/abs/2310.02277)|
|[![Star](https://img.shields.io/github/stars/luuyin/OWL.svg?style=social&label=Star)](https://github.com/luuyin/OWL)[![Type](https://img.shields.io/badge/Unstructured-C2A4A6)]() <br>[Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity](https://arxiv.org/abs/2310.05175) <br> Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Mykola Pechenizkiy, Yi Liang, Zhangyang Wang, Shiwei Liu |<img width="1002" alt="image" src="https://github.com/luuyin/OWL/blob/main/Images/Layer_wise_sparsity.png"> |[Github](https://github.com/luuyin/OWL) <br> [Paper](https://arxiv.org/abs/2310.05175)|
|[![Type](https://img.shields.io/badge/Structural-C2A4A6)]() <br>[Compresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language Models](https://arxiv.org/abs/2310.05015) <br> Song Guo, Jiahang Xu, Li Lyna Zhang, Mao Yang |<img width="1002" alt="image" src="figures/compresso.png"> |[Github](https://github.com/microsoft/Moonlit/tree/main/Compresso) <br> [Paper](https://arxiv.org/abs/2310.05015)|
|[![Star](https://img.shields.io/github/stars/IST-DASLab/SparseFinetuning.svg?style=social&label=Star)](https://github.com/IST-DASLab/SparseFinetuning) [![Type](https://img.shields.io/badge/Unstructured-C2A4A6)]() <br>[Sparse Finetuning for Inference Acceleration of Large Language Models](https://arxiv.org/abs/2310.06927) <br> Eldar Kurtic, Denis Kuznedelev, Elias Frantar, Michael Goin, Dan Alistarh |<img width="1002" alt="image" src="figures/SquareHead.png"> |[Github](https://github.com/IST-DASLab/SparseFinetuning) <br> [Paper](https://arxiv.org/abs/2310.06927)|
|[![Type](https://img.shields.io/badge/Activation-C2A4A6)]() <br> [ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models](https://arxiv.org/abs/2310.04564) <br> Iman Mirzadeh, Keivan Alizadeh, Sachin Mehta, Carlo C Del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, Mehrdad Farajtabar |<img width="1002" alt="image" src="figures/relufication.png"> |[Paper](https://arxiv.org/abs/2310.04564)|
|[![Type](https://img.shields.io/badge/Unstructured-C2A4A6)]() <br> [The Cost of Down-Scaling Language Models: Fact Recall Deteriorates before In-Context Learning](https://arxiv.org/abs/2310.04680) <br> Tian Jin, Nolan Clement, Xin Dong, Vaishnavh Nagarajan, Michael Carbin, Jonathan Ragan-Kelley, Gintare Karolina Dziugaite |<img width="1002" alt="image" src="figures/recall_and_icl.png"> |[Paper](https://arxiv.org/abs/2310.04680)|
|[![Star](https://img.shields.io/github/stars/talkking/MixGPT.svg?style=social&label=Star)](https://github.com/talkking/MixGPT)[![Publish](https://img.shields.io/badge/Conference-ICASSP2024-blue)]() [![Type](https://img.shields.io/badge/Unstructured-C2A4A6)]() <br>[One-Shot Sensitivity-Aware Mixed Sparsity Pruning for Large Language Models](https://arxiv.org/abs/2310.09499) <br> Hang Shao, Bei Liu, Bo Xiao, Ke Zeng, Guanglu Wan, Yanmin Qian |<img width="1002" alt="image" src="figures/sensitivity_sparse.png"> |[Github](https://github.com/talkking/MixGPT) <br> [Paper](https://arxiv.org/abs/2310.09499)|
|[![Star](https://img.shields.io/github/stars/microsoft/lorashear.svg?style=social&label=Star)](https://github.com/microsoft/lorashear) [![Type](https://img.shields.io/badge/Structural-C2A4A6)]() <br>[LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery](https://arxiv.org/abs/2310.18356) <br> Tianyi Chen, Tianyu Ding, Badal Yadav, Ilya Zharkov, Luming Liang |<img width="1002" alt="image" src="figures/LoRAShear.png"> |[Github](https://github.com/microsoft/lorashear) <br> [Paper](https://arxiv.org/abs/2310.18356)|
|[![Star](https://img.shields.io/github/stars/Aleph-Alpha/Divergent_Tokens.svg?style=social&label=Star)](https://github.com/Aleph-Alpha/Divergent_Tokens) [![Type](https://img.shields.io/badge/Metric-C2A4A6)]() <br>[Divergent Token Metrics: Measuring degradation to prune away LLM components -- and optimize quantization](https://arxiv.org/abs/2311.01544) <br> Björn Deiseroth, Max Meuer, Nikolas Gritsch, Constantin Eichenberg, Patrick Schramowski, Matthias Aßenmacher, Kristian Kersting |<img width="1002" alt="image" src="figures/FDT.png"> |[Github](https://github.com/Aleph-Alpha/Divergent_Tokens) <br> [Paper](https://arxiv.org/abs/2311.01544)|
|[![Star](https://img.shields.io/github/stars/VILA-Lab/GBLM-Pruner.svg?style=social&label=Star)](https://github.com/VILA-Lab/GBLM-Pruner) [![Type](https://img.shields.io/badge/Unstructured-C2A4A6)]()  <br>[Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models](https://arxiv.org/abs/2311.04902) <br> Rocktim Jyoti Das, Liqun Ma, Zhiqiang Shen |<img width="1002" alt="image" src="figures/GBLM-Pruner.png"> |[Github](https://github.com/VILA-Lab/GBLM-Pruner) <br> [Paper](https://arxiv.org/abs/2311.04902)|
|[![Star](https://img.shields.io/github/stars/zyxxmu/DSnoT.svg?style=social&label=Star)](https://github.com/zyxxmu/DSnoT)<br>[Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs](https://arxiv.org/abs/2310.08915) [![Type](https://img.shields.io/badge/Unstructured-C2A4A6)]() <br> Yuxin Zhang, Lirui Zhao, Mingbao Lin, Yunyun Sun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, Rongrong Ji |<img width="202" alt="image" src="https://github.com/zyxxmu/DSnoT/blob/main/imgs/framework.png"> |[Github](https://github.com/zyxxmu/DSnoT) <br> [Paper](https://arxiv.org/abs/2310.08915)|
|[![Type](https://img.shields.io/badge/Semi-structured-C2A4A6)]() [E-Sparse: Boosting the Large Language Model Inference through Entropy-based N:M Sparsity](https://arxiv.org/abs/2310.15929) <br> Yun Li, Lin Niu, Xipeng Zhang, Kai Liu, Jianchen Zhu, Zhanhui Kang |<img width="1002" alt="image" src="figures/e-sparse.png"> |[Paper](https://arxiv.org/abs/2310.15929)|
|[![Star](https://img.shields.io/github/stars/ZIB-IOL/PERP.svg?style=social&label=Star)](https://github.com/ZIB-IOL/PERP) [![Type](https://img.shields.io/badge/Semi-structured-C2A4A6)]() <br>[PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMs](https://arxiv.org/abs/2312.15230) <br> Max Zimmer, Megi Andoni, Christoph Spiegel, Sebastian Pokutta |<img width="1002" alt="image" src="figures/PERP.png"> |[Github](https://github.com/ZIB-IOL/PERP) <br> [Paper](https://arxiv.org/abs/2312.15230)|
|[![Star](https://img.shields.io/github/stars/fmfi-compbio/admm-pruning.svg?style=social&label=Star)](https://github.com/fmfi-compbio/admm-pruning)<br>[Fast and Optimal Weight Update for Pruned Large Language Models](https://arxiv.org/abs/2401.02938) [![Type](https://img.shields.io/badge/Unstructured-C2A4A6)]() <br> Vladimír Boža |<img width="202" alt="image" src="figures/admm.png"> |[Github](https://github.com/fmfi-compbio/admm-pruning) <br> [Paper](https://arxiv.org/abs/2401.02938)|
|[![Star](https://img.shields.io/github/stars/CrystalEye42/eval-safety.svg?style=social&label=Star)](https://github.com/CrystalEye42/eval-safety) [![Type](https://img.shields.io/badge/Unstructured-C2A4A6)]() <br>[Pruning for Protection: Increasing Jailbreak Resistance in Aligned LLMs Without Fine-Tuning](https://arxiv.org/abs/2401.10862) <br> Adib Hasan, Ileana Rugina, Alex Wang |<img width="1002" alt="image" src="figures/eval_safety.png"> |[Github](https://github.com/CrystalEye42/eval-safety) <br> [Paper](https://arxiv.org/abs/2401.10862)|
|[![Star](https://img.shields.io/github/stars/microsoft/TransformerCompression.svg?style=social&label=Star)](https://github.com/microsoft/TransformerCompression) [![Type](https://img.shields.io/badge/Structural-C2A4A6)]()<br>[SliceGPT: Compress Large Language Models by Deleting Rows and Columns](https://arxiv.org/abs/2401.15024) <br> Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, James Hensman |<img width="1002" alt="image" src="figures/SliceGPT.png"> |[Github](https://github.com/microsoft/TransformerCompression) <br> [Paper](https://arxiv.org/abs/2401.15024)|
|[![Type](https://img.shields.io/badge/Structural-C2A4A6)]() <br>[APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference](https://arxiv.org/abs/2401.12200) <br> Bowen Zhao, Hannaneh Hajishirzi, Qingqing Cao |<img width="1002" alt="image" src="figures/APT.png"> |[Paper](https://arxiv.org/abs/2401.12200)|
|[ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMs](https://arxiv.org/abs/2402.03804) <br> Zhengyan Zhang, Yixin Song, Guanghui Yu, Xu Han, Yankai Lin, Chaojun Xiao, Chenyang Song, Zhiyuan Liu, Zeyu Mi, Maosong Sun |<img width="1002" alt="image" src="figures/relu2wins.png"> |[Paper](https://arxiv.org/abs/2402.03804)|
|[![Star](https://img.shields.io/github/stars/ldery/Bonsai.svg?style=social&label=Star)](https://github.com/ldery/Bonsai)<br>[Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes](https://arxiv.org/abs/2402.05406) <br> Lucio Dery, Steven Kolawole, Jean-Francois Kagey, Virginia Smith, Graham Neubig, Ameet Talwalkar |<img width="1002" alt="image" src="figures/bonsai.png"> |[Github](https://github.com/ldery/Bonsai) <br> [Paper](https://arxiv.org/abs/2402.05406)|
|[![Star](https://img.shields.io/github/stars/boyiwei/alignment-attribution-code.svg?style=social&label=Star)](https://github.com/boyiwei/alignment-attribution-code)<br>[Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications](https://arxiv.org/abs/2402.05162) <br> Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia et al|<img width="1002" alt="image" src="https://boyiwei.com/alignment-attribution/static/images/main.png"> |[Github](https://github.com/boyiwei/alignment-attribution-code) <br> [Paper](https://arxiv.org/abs/2402.05162) <br> [Project](https://boyiwei.com/alignment-attribution/)|
|[NutePrune: Efficient Progressive Pruning with Numerous Teachers for Large Language Models](https://arxiv.org/abs/2402.09773) <br> Shengrui Li, Xueting Han, Jing Bai |<img width="202" alt="image" src="https://arxiv.org/html/2402.09773v1/x1.png"> |[Paper](https://arxiv.org/abs/2402.09773)|
|[Learn To be Efficient: Build Structured Sparsity in Large Language Models](https://arxiv.org/abs/2402.06126) <br> Haizhong Zheng, Xiaoyan Bai, Beidi Chen, Fan Lai, Atul Prakash |<img width="1002" alt="image" src="figures/LTE.png"> |[Paper](https://arxiv.org/abs/2402.06126)|
|[![Star](https://img.shields.io/github/stars/Nota-NetsPresso/shortened-llm.svg?style=social&label=Star)](https://github.com/Nota-NetsPresso/shortened-llm) [![Publish](https://img.shields.io/badge/Workshop-ICLRW'24-blue)]() [![Type](https://img.shields.io/badge/Structural-C2A4A6)]() <br> [Shortened LLaMA: A Simple Depth Pruning for Large Language Models](https://arxiv.org/abs/2402.02834) <br> Bo-Kyeong Kim, Geonmin Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, Hyoung-Kyu Song |<img width="1002" alt="image" src="figures/ShortenedLLaMA.png"> |[Github](https://github.com/Nota-NetsPresso/shortened-llm)<br>[Paper](https://arxiv.org/abs/2402.02834)|
|[![Star](https://img.shields.io/github/stars/leapingjagg-dev/SLEB.svg?style=social&label=Star)](https://github.com/leapingjagg-dev/SLEB)<br>[SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks](https://arxiv.org/abs/2402.09025) <br> Jiwon Song, Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, Jae-Joon Kim |<img width="1002" alt="image" src="figures/SLEB.png"> |[Github](https://github.com/leapingjagg-dev/SLEB) <br> [Paper](https://arxiv.org/abs/2402.09025)|
|[HiRE: High Recall Approximate Top-k Estimation for Efficient LLM Inference](https://arxiv.org/abs/2402.09360) <br> Yashas Samaga B L, Varun Yerram, Chong You, Srinadh Bhojanapalli, Sanjiv Kumar, Prateek Jain, Praneeth Netrapalli |<img width="202" alt="image" src="https://arxiv.org/html/2402.09360v1/extracted/5409158/figures/herd.png"> |[Paper](https://arxiv.org/abs/2402.09360)|
|[LaCo: Large Language Model Pruning via Layer Collapse](https://arxiv.org/abs/2402.11187) <br> Yifei Yang, Zouying Cao, Hai Zhao |<img width="1002" alt="image" src="figures/LaCo.png"> |[Paper](https://arxiv.org/abs/2402.11187)|
|[![Star](https://img.shields.io/github/stars/Raincleared-Song/sparse_gpu_operator.svg?style=social&label=Star)](https://github.com/Raincleared-Song/sparse_gpu_operator)<br>[ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Models](https://arxiv.org/abs/2402.13516) <br> Chenyang Song, Xu Han, Zhengyan Zhang, Shengding Hu, Xiyu Shi, Kuai Li et al |<img width="1002" alt="image" src="https://arxiv.org/html/2402.13516v1/x1.png"> |[Github](https://github.com/Raincleared-Song/sparse_gpu_operator) <br> [Paper](https://arxiv.org/abs/2402.13516) <br> [[Model-7B]](https://huggingface.co/SparseLLM/prosparse-llama-2-7b) [[Model-13B]](https://huggingface.co/SparseLLM/prosparse-llama-2-13b)|
|[![Star](https://img.shields.io/github/stars/sunggo/EBFT.svg?style=social&label=Star)](https://github.com/sunggo/EBFT)<br>[EBFT: Effective and Block-Wise Fine-Tuning for Sparse LLMs](https://arxiv.org/abs/2402.12419) <br> Song Guo, Fan Wu, Lei Zhang, Xiawu Zheng, Shengchuan Zhang, Fei Chao, Yiyu Shi, Rongrong Ji |<img width="1002" alt="image" src="figures/EBFT.png"> |[Github](https://github.com/sunggo/EBFT) <br> [Paper](https://arxiv.org/abs/2402.12419)|
|[![Star](https://img.shields.io/github/stars/OpenGVLab/LLMPrune-BESA.svg?style=social&label=Star)](https://github.com/OpenGVLab/LLMPrune-BESA)<br>[BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation](https://arxiv.org/pdf/2402.16880.pdf) <br> Peng Xu, Wenqi Shao, Mengzhao Chen, Shitao Tang, Kaipeng Zhang, Peng Gao, Fengwei An, Yu Qiao, Ping Luo |<img width="1002" alt="image" src="https://arxiv.org/html/2402.16880v1/x1.png"> |[Github](https://github.com/OpenGVLab/LLMPrune-BESA) <br> [Paper](https://arxiv.org/pdf/2402.16880.pdf)|
|[ShortGPT: Layers in Large Language Models are More Redundant Than You Expect](https://arxiv.org/abs/2403.03853) <br> Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, Weipeng Chen |<img width="1002" alt="image" src="https://arxiv.org/html/2403.03853v1/x1.png"> |[Paper](https://arxiv.org/abs/2403.03853)|
|[Efficient Pruning of Large Language Model with Adaptive Estimation Fusion](https://arxiv.org/abs/2403.10799) <br> Jun Liu, Chao Wu, Changdi Yang, Hao Tang, Haoye Dong, Zhenglun Kong, Geng Yuan, Wei Niu, Dong Huang, Yanzhi Wang |<img width="1002" alt="image" src="https://arxiv.org/html/2403.10799v1/x1.png"> |[Paper](https://arxiv.org/abs/2403.10799)|
|[![Star](https://img.shields.io/github/stars/decoding-comp-trust/comp-trust.svg?style=social&label=Star)](https://github.com/decoding-comp-trust/comp-trust) [![Type](https://img.shields.io/badge/w/Quantization-39B0A9)]() <br>[Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression](https://arxiv.org/abs/2403.15447) <br> Junyuan Hong, Jinhao Duan, Chenhui Zhang, Zhangheng Li, Chulin Xie et al|<img width="1002" alt="image" src="https://arxiv.org/html/2403.15447v1/extracted/5477136/fig/teaser.png"> |[Github](https://github.com/decoding-comp-trust/comp-trust) <br> [Paper](https://arxiv.org/abs/2403.15447) <br> [Project](https://decoding-comp-trust.github.io) |
|[Compressing Large Language Models by Streamlining the Unimportant Layer](https://arxiv.org/abs/2403.19135) <br> Xiaodong Chen, Yuxuan Hu, Jing Zhang |<img width="1002" alt="image" src="https://arxiv.org/html/2403.19135v1/x1.png"> |[Paper](https://arxiv.org/abs/2403.19135)|
|[![Star](https://img.shields.io/github/stars/X-LANCE/MBS.svg?style=social&label=Star)](https://github.com/X-LANCE/MBS)<br>[Multilingual Brain Surgeon: Large Language Models Can be Compressed Leaving No Language Behind](https://arxiv.org/abs/2404.04748) <br> Hongchuan Zeng, Hongshen Xu, Lu Chen, Kai Yu |<img width="1002" alt="image" src="https://github.com/HongchuanZeng/MBS/raw/main/mbs.png"> |[Github](https://github.com/X-LANCE/MBS) <br> [Paper](https://arxiv.org/abs/2404.04748)|
|[![Star](https://img.shields.io/github/stars/Adaxry/Unified_Layer_Skipping.svg?style=social&label=Star)](https://github.com/Adaxry/Unified_Layer_Skipping)<br>[Accelerating Inference in Large Language Models with a Unified Layer Skipping Strategy](https://arxiv.org/abs/2404.06954) <br> Yijin Liu, Fandong Meng, Jie Zhou |<img width="202" alt="image" src="https://github.com/Adaxry/Unified_Layer_Skipping/raw/main/figures/overview.png"> |[Github](https://github.com/Adaxry/Unified_Layer_Skipping) <br> [Paper](https://arxiv.org/abs/2404.06954)|
|[![Star](https://img.shields.io/github/stars/lihuang258/LoRAP.svg?style=social&label=Star)](https://github.com/lihuang258/LoRAP)[![Publish](https://img.shields.io/badge/Conference-ICML'24-blue)]() [![Type](https://img.shields.io/badge/Unstructured-C2A4A6)]()<br>[LoRAP: Transformer Sub-Layers Deserve Differentiated Structured Compression for Large Language Models](https://proceedings.mlr.press/v235/li24bi.html) <br> Guangyan Li, Yongqiang Tang, Wensheng Zhang |<img width="1002" alt="image" src="https://arxiv.org/html/2404.09695v1/x1.png"> |[Github](https://github.com/lihuang258/LoRAP) <br> [Paper](https://proceedings.mlr.press/v235/li24bi.html)
|[CATS: Contextually-Aware Thresholding for Sparsity in Large Language Models](https://arxiv.org/abs/2404.08763) <br> Je-Yong Lee, Donghyun Lee, Genghan Zhang, Mo Tiwari, Azalia Mirhoseini |<img width="1002" alt="image" src="https://arxiv.org/html/2404.08763v1/x5.png"> |[Paper](https://arxiv.org/abs/2404.08763)|
|[Layer Skip: Enabling Early Exit Inference and Self-Speculative Decoding](https://arxiv.org/abs/2404.16710) <br> Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer et al|<img width="1002" alt="image" src="figures/LayerSkip.png"> |[Paper](https://arxiv.org/abs/2404.16710)|
|[Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment](https://arxiv.org/abs/2405.03594) <br> Abhinav Agarwalla, Abhay Gupta, Alexandre Marques, Shubhra Pandit, Michael Goin, Eldar Kurtic, Kevin Leong, Tuan Nguyen, Mahmoud Salem, Dan Alistarh, Sean Lie, Mark Kurtz |<img width="1002" alt="image" src="figures/high_sparsity_pretraining.png"> |[Paper](https://arxiv.org/abs/2405.03594)|
|[Dependency-Aware Semi-Structured Sparsity of GLU Variants in Large Language Models](https://arxiv.org/abs/2405.01943) <br> Zhiyu Guo, Hidetaka Kamigaito, Taro Wanatnabe |<img width="1002" alt="image" src="figures/DaSS.png"> |[Paper](https://arxiv.org/abs/2405.01943)|
|[Mixture-of-Depths: Dynamically allocating compute in transformer-based language models](https://arxiv.org/abs/2404.02258) <br> David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, Adam Santoro |<img width="1002" alt="image" src="https://arxiv.org/html/2404.02258v1/extracted/2404.02258v1/mod.png"> |[Paper](https://arxiv.org/abs/2404.02258)|
|[![Star](https://img.shields.io/github/stars/IntelLabs/Hardware-Aware-Automated-Machine-Learning.svg?style=social&label=Star)](https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/LoNAS)[![Publish](https://img.shields.io/badge/Conference-COLING'24-blue)]() [![Type](https://img.shields.io/badge/Structural-C2A4A6)]()<br>[LoNAS: Elastic Low-Rank Adapters for Efficient Large Language Models](https://aclanthology.org/2024.lrec-main.940) <br> Juan Pablo Munoz, Jinjie Yuan, Yi Zheng, Nilesh Jain |<img width="1002" alt="image" src="figures/LoNAS.png"> |[Github](https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/LoNAS) <br> [Paper](https://aclanthology.org/2024.lrec-main.940)|
|[![Star](https://img.shields.io/github/stars/IntelLabs/Hardware-Aware-Automated-Machine-Learning.svg?style=social&label=Star)](https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/Shears)[![Publish](https://img.shields.io/badge/Conference-NAACL'24%20Industry%20Track-blue)]() [![Type](https://img.shields.io/badge/Unstructured-C2A4A6)]()<br>[Shears: Unstructured Sparsity with Neural Low-rank Adapter Search](https://arxiv.org/abs/2404.10934) <br> Juan Pablo Munoz, Jinjie Yuan, Nilesh Jain |<img width="1002" alt="image" src="figures/Shears.png"> |[Github](https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/Shears) <br> [Paper](https://arxiv.org/abs/2404.10934)|
|[![Star](https://img.shields.io/github/stars/psunlpgroup/D-Pruner.svg?style=social&label=Star)](https://github.com/psunlpgroup/D-Pruner)[![Publish](https://img.shields.io/badge/Conference-NAACL'24%20Findings-blue)]()<br>[Pruning as a Domain-specific LLM Extractor](https://arxiv.org/abs/2405.06275) <br> Nan Zhang, Yanchi Liu, Xujiang Zhao, Wei Cheng, Runxue Bao, Rui Zhang, Prasenjit Mitra, Haifeng Chen |<img width="1002" alt="image" src="https://github.com/psunlpgroup/D-Pruner/raw/main/assets/prune_types_example.png"> |[Github](https://github.com/psunlpgroup/D-Pruner) <br> [Paper](https://arxiv.org/abs/2405.06275)|
|[![Star](https://img.shields.io/github/stars/mshamrai/language-specific-pruning.svg?style=social&label=Star)](https://github.com/mshamrai/language-specific-pruning)[![Publish](https://img.shields.io/badge/Conference-UNLP'24-blue)]()<br>[Language-Specific Pruning for Efficient Reduction of Large Language Models](https://aclanthology.org/2024.unlp-1.16/) <br> Maksym Shamrai | |[Github](https://github.com/mshamrai/language-specific-pruning) <br> [Paper](https://aclanthology.org/2024.unlp-1.16/)|
|[![Star](https://img.shields.io/github/stars/OpenNLG/OpenBA-v2.svg?style=social&label=Star)](https://github.com/OpenNLG/OpenBA-v2)<br>[OpenBA-V2: Reaching 77.3% High Compression Ratio with Fast Multi-Stage Pruning](https://arxiv.org/abs/2405.05957) <br> Dan Qiao, Yi Su, Pinzheng Wang, Jing Ye, Wenjing Xie et al |<img width="1002" alt="image" src="figures/OpenBA.png"> |[Github](https://github.com/OpenNLG/OpenBA-v2) <br> [Paper](https://arxiv.org/abs/2405.05957)|
|[FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Models](https://arxiv.org/abs/2405.18218) <br> Yang Zhang, Yawei Li, Xinpeng Wang, Qianli Shen, Barbara Plank, Bernd Bischl, Mina Rezaei, Kenji Kawaguchi |<img width="1002" alt="image" src="https://arxiv.org/html/2405.18218v1/x1.png"> |[Paper](https://arxiv.org/abs/2405.18218)|
|[![Star](https://img.shields.io/github/stars/Mohammad-Mozaffari/slope.svg?style=social&label=Star)](https://github.com/Mohammad-Mozaffari/slope)<br>[SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs](https://arxiv.org/abs/2405.16325) <br> Mohammad Mozaffari, Amir Yazdanbakhsh, Zhao Zhang, Maryam Mehri Dehnavi |<img width="1002" alt="image" src="https://arxiv.org/html/2405.16325v1/x1.png"> |[Github](https://github.com/Mohammad-Mozaffari/slope) <br> [Paper](https://arxiv.org/abs/2405.16325)|
|[![Star](https://img.shields.io/github/stars/Lucky-Lance/SPP.svg?style=social&label=Star)](https://github.com/Lucky-Lance/SPP)<br>[SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models](https://arxiv.org/abs/2405.16057) <br> Xudong Lu, Aojun Zhou, Yuhui Xu, Renrui Zhang, Peng Gao, Hongsheng Li |<img width="1002" alt="image" src="https://github.com/Lucky-Lance/SPP/raw/main/asserts/SPP.png"> |[Github](https://github.com/Lucky-Lance/SPP) <br> [Paper](https://arxiv.org/abs/2405.16057)|
|[Large Language Model Pruning](https://arxiv.org/abs/2406.00030) <br> Hanjuan Huang, Hao-Jia Song, Hsing-Kuo Pao |<img width="1002" alt="image" src="https://arxiv.org/html/2406.00030v1/x1.png"> |[Paper](https://arxiv.org/abs/2406.00030)|[//]: #06/05
|[![Type](https://img.shields.io/badge/w/Quantization-39B0A9)]() [Effective Interplay between Sparsity and Quantization: From Theory to Practice](https://arxiv.org/abs/2405.20935) <br> Simla Burcu Harma, Ayan Chakraborty, Elizaveta Kostenok, Danila Mishin, Dongho Ha, Babak Falsafi, Martin Jaggi, Ming Liu, Yunho Oh, Suvinay Subramanian, Amir Yazdanbakhsh ||[Paper](https://arxiv.org/abs/2405.20935)|[//]: #06/05
|[VTrans: Accelerating Transformer Compression with Variational Information Bottleneck based Pruning](https://arxiv.org/abs/2406.05276) <br> Oshin Dutta, Ritvik Gupta, Sumeet Agarwal |<img width="1002" alt="image" src="figures/VTrans.png"> |[Paper](https://arxiv.org/abs/2406.05276)|[//]: #06/11
|[Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters](https://arxiv.org/abs/2406.05955) <br> Yixin Song, Haotong Xie, Zhengyan Zhang, Bo Wen, Li Ma, Zeyu Mi, Haibo Chen |<img width="1002" alt="image" src="https://arxiv.org/html/2406.05955v1/x2.png"> |[Paper](https://arxiv.org/abs/2406.05955) <br> [Model](https://huggingface.co/PowerInfer/TurboSparse-Mixtral) |[//]: #06/11
|[![Star](https://img.shields.io/github/stars/pprp/Pruner-Zero.svg?style=social&label=Star)](https://github.com/pprp/Pruner-Zero)[![Publish](https://img.shields.io/badge/Conference-ICML'24-blue)]()<br>[Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Models](https://arxiv.org/abs/2406.02924) <br> Peijie Dong, Lujun Li, Zhenheng Tang, Xiang Liu, Xinglin Pan, Qiang Wang, Xiaowen Chu |<img width="1002" alt="image" src="https://raw.githubusercontent.com/pprp/Pruner-Zero/main/.github/images/pruner-zero-main-figure.png"> |[Github](https://github.com/pprp/Pruner-Zero) <br> [Paper](https://arxiv.org/abs/2406.02924)|[//]: #06/11
|[![Star](https://img.shields.io/github/stars/ShiningSord/MoreauPruner.svg?style=social&label=Star)](https://github.com/ShiningSord/MoreauPruner)<br>[MoreauPruner: Robust Pruning of Large Language Models against Weight Perturbations](https://arxiv.org/abs/2406.07017) <br> Zixiao Wang, Jingwei Zhang, Wenqian Zhao, Farzan Farnia, Bei Yu |<img width="1002" alt="image" src="https://arxiv.org/html/2406.07017v1/x1.png"> |[Github](https://github.com/ShiningSord/MoreauPruner) <br> [Paper](https://arxiv.org/abs/2406.07017)|[//]: #06/12
|[ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models](https://arxiv.org/abs/2406.07831) <br> Xiang Meng, Kayhan Behdin, Haoyue Wang, Rahul Mazumder |<img width="1002" alt="image" src="figures/ALPS.png"> |[Paper](https://arxiv.org/abs/2406.07831)|[//]: #06/18
|[Optimization-based Structural Pruning for Large Language Models without Back-Propagation](https://arxiv.org/abs/2406.10576) <br> Yuan Gao, Zujing Liu, Weizhong Zhang, Bo Du, Gui-Song Xia |<img width="1002" alt="image" src="https://arxiv.org/html/2406.10576v1/extracted/5669159/imgs/overview5.png"> |[Paper](https://arxiv.org/abs/2406.10576)|[//]: #06/23
|[![Star](https://img.shields.io/github/stars/shadow_llm/.svg?style=social&label=Star)](https://github.com/shadow_llm/)<br>[ShadowLLM: Predictor-based Contextual Sparsity for Large Language Models](https://arxiv.org/abs/2406.16635) <br> Yash Akhauri, Ahmed F AbouElhamayed, Jordan Dotzel, Zhiru Zhang, Alexander M Rush, Safeen Huda, Mohamed S Abdelfattah |<img width="1002" alt="image" src="https://arxiv.org/html/2406.16635v1/x4.png"> |[Github](https://github.com/abdelfattah-lab/shadow_llm/) <br> [Paper](https://arxiv.org/abs/2406.16635)|[//]: #06/26
|[Rethinking Pruning Large Language Models: Benefits and Pitfalls of Reconstruction Error Minimization](https://arxiv.org/abs/2406.15524) <br> Sungbin Shin, Wonpyo Park, Jaeho Lee, Namhoon Lee |<img width="1002" alt="image" src="https://arxiv.org/html/2406.15524v1/x3.png"> |[Paper](https://arxiv.org/abs/2406.15524)|[//]: #06/26
|[![Publish](https://img.shields.io/badge/Conference-COLT'24-blue)]()<br>[Learning Neural Networks with Sparse Activations](https://arxiv.org/abs/2406.17989) <br> Pranjal Awasthi, Nishanth Dikkala, Pritish Kamath, Raghu Meka | |[Paper](https://arxiv.org/abs/2406.17989)|[//]: #06/28
|[FoldGPT: Simple and Effective Large Language Model Compression Scheme](https://arxiv.org/abs/2407.00928) <br> Songwei Liu, Chao Zeng, Lianqiang Li, Chenqian Yan, Lean Fu, Xing Mei, Fangmin Chen |<img width="1002" alt="image" src="https://arxiv.org/html/2407.00928v1/extracted/5701554/flodGPT.png"> |[Paper](https://arxiv.org/abs/2407.00928)|[//]: #07/03
|[![Publish](https://img.shields.io/badge/Conference-NAACL'24%20Findings-blue)]()<br>[Structured Pruning for Large Language Models Using Coupled Components Elimination and Minor Fine-tuning](https://aclanthology.org/2024.findings-naacl.1/) <br> Honghe Zhang, XiaolongShi XiaolongShi, Jingwei Sun, Guangzhong Sun |<img width="1002" alt="image" src="figures/CCEMF.png"> |[Paper](https://aclanthology.org/2024.findings-naacl.1/)|[//]: #07/05
|[![Star](https://img.shields.io/github/stars/MrGGLS/BlockPruner.svg?style=social&label=Star)](https://github.com/MrGGLS/BlockPruner)<br>[BlockPruner: Fine-grained Pruning for Large Language Models](https://arxiv.org/abs/2406.10594) <br> Longguang Zhong, Fanqi Wan, Ruijun Chen, Xiaojun Quan, Liangzhi Li |<img width="1002" alt="image" src="https://arxiv.org/html/2406.10594v2/x3.png"> |[Github](https://github.com/MrGGLS/BlockPruner) <br> [Paper](https://arxiv.org/abs/2406.10594)|[//]: #07/05
|[![Publish](https://img.shields.io/badge/Conference-ICML'24-blue)]()<br>[Flextron: Many-in-One Flexible Large Language Model](https://arxiv.org/abs/2406.10260) <br> Ruisi Cai, Saurav Muralidharan, Greg Heinrich, Hongxu Yin, Zhangyang Wang, Jan Kautz, Pavlo Molchanov |<img width="1002" alt="image" src="https://arxiv.org/html/2406.10260v1/x1.png"> |[Paper](https://arxiv.org/abs/2406.10260)|[//]: #07/05
|[![Star](https://img.shields.io/github/stars/sbwww/TransAct-pruning.svg?style=social&label=Star)](https://github.com/sbwww/TransAct-pruning)[![Publish](https://img.shields.io/badge/Conference-ACL24'Findings-blue)]()<br>[Pruning Large Language Models to Intra-module Low-rank Architecture with Transitional Activations](https://arxiv.org/abs/2407.05690) <br> Bowen Shen, Zheng Lin, Daren Zha, Wei Liu, Jian Luan, Bin Wang, Weiping Wang |<img width="1002" alt="image" src="https://arxiv.org/html/2407.05690v1/x2.png"> |[Github](https://github.com/sbwww/TransAct-pruning) <br> [Paper](https://arxiv.org/abs/2407.05690)|[//]: #07/10
|[![Star](https://img.shields.io/github/stars/zhichaoxu-shufe/Beyond-Perplexity-Compression-Safety-Eval.svg?style=social&label=Star)](https://github.com/zhichaoxu-shufe/Beyond-Perplexity-Compression-Safety-Eval) [![Type](https://img.shields.io/badge/w/Quantization-39B0A9)]() <br>[Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compression](https://arxiv.org/abs/2407.04965) <br> Zhichao Xu, Ashim Gupta, Tao Li, Oliver Bentham, Vivek Srikumar | |[Github](https://github.com/zhichaoxu-shufe/Beyond-Perplexity-Compression-Safety-Eval) <br> [Paper](https://arxiv.org/abs/2407.04965)|[//]: #07/10
|[Q-Sparse: All Large Language Models can be Fully Sparsely-Activated](https://arxiv.org/abs/2407.10969) <br> Hongyu Wang, Shuming Ma, Ruiping Wang, Furu Wei |<img width="1002" alt="image" src="https://arxiv.org/html/2407.10969v1/x3.png"> |[Paper](https://arxiv.org/abs/2407.10969)|[//]: #07/16
|[Reconstruct the Pruned Model without Any Retraining](https://arxiv.org/abs/2407.13331) <br> Pingjie Wang, Ziqing Fan, Shengchao Hu, Zhe Chen, Yanfeng Wang, Yu Wang |<img width="1002" alt="image" src="https://arxiv.org/html/2407.13331v1/x3.png"> |[Paper](https://arxiv.org/abs/2407.13331)|[//]: #07/21
|[MINI-LLM: Memory-Efficient Structured Pruning for Large Language Models](https://arxiv.org/abs/2407.11681mini) <br> Hongrong Cheng, Miao Zhang, Javen Qinfeng Shi |<img width="1002" alt="image" src="figures/minillm.png"> |[Paper](https://arxiv.org/abs/2407.11681mini)|[//]: #07/21
|[![Star](https://img.shields.io/github/stars/NVlabs/Minitron.svg?style=social&label=Star)](https://github.com/NVlabs/Minitron)<br>[Compact Language Models via Pruning and Knowledge Distillation](https://arxiv.org/abs/2407.14679) <br> Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, Pavlo Molchanov |<img width="1002" alt="image" src="https://arxiv.org/html/2407.14679v1/x2.png"> |[Github](https://github.com/NVlabs/Minitron) <br> [Paper](https://arxiv.org/abs/2407.14679)|[//]: #07/29
|[Greedy Output Approximation: Towards Efficient Structured Pruning for LLMs Without Retraining](https://arxiv.org/abs/2407.19126) <br> Jianwei Li, Yijun Dong, Qi Lei |<img width="1002" alt="image" src="https://arxiv.org/html/2407.19126v1/x2.png"> |[Paper](https://arxiv.org/abs/2407.19126)|[//]: #08/08
|[Pruning Large Language Models with Semi-Structural Adaptive Sparse Training](https://arxiv.org/abs/2407.20584) <br> Weiyu Huang, Guohao Jian, Yuezhou Hu, Jun Zhu, Jianfei Chen |<img width="1002" alt="image" src="https://arxiv.org/html/2407.20584v1/extracted/5756562/4.png"> |[Paper](https://arxiv.org/abs/2407.20584)|[//]: #08/08
|[A Convex-optimization-based Layer-wise Post-training Pruner for Large Language Models](https://arxiv.org/abs/2408.03728) <br> Pengxiang Zhao, Hanyu Hu, Ping Li, Yi Zheng, Zhefeng Wang, Xiaoming Yuan |<img width="1002" alt="image" src="https://arxiv.org/html/2408.03728v1/x1.png"> |[Paper](https://arxiv.org/abs/2408.03728)|[//]: #08/08
|[Enhancing One-shot Pruned Pre-trained Language Models through Sparse-Dense-Sparse Mechanism](https://arxiv.org/abs/2408.10473) <br> Guanchen Li, Xiandong Zhao, Lian Liu, Zeping Li, Dong Li, Lu Tian, Jie He, Ashish Sirasao, Emad Barsoum |<img width="1002" alt="image" src="https://arxiv.org/html/2408.10473v1/x1.png"> |[Paper](https://arxiv.org/abs/2408.10473)|[//]: #08/27
|[![Star](https://img.shields.io/github/stars/YupengSu/LLM-Barber.svg?style=social&label=Star)](https://github.com/YupengSu/LLM-Barber)<br>[LLM-Barber: Block-Aware Rebuilder for Sparsity Mask in One-Shot for Large Language Models](https://arxiv.org/abs/2408.10631) <br> Yupeng Su, Ziyi Guan, Xiaoqun Liu, Tianlai Jin, Dongkuan Wu, Graziano Chesi, Ngai Wong, Hao Yu |<img width="1002" alt="image" src="https://github.com/YupengSu/LLM-Barber/raw/main/img/figure1a.png"> |[Github](https://github.com/YupengSu/LLM-Barber) <br> [Paper](https://arxiv.org/abs/2408.10631)|[//]: #08/27
|[LLM Pruning and Distillation in Practice: The Minitron Approach](https://arxiv.org/abs/2408.11796) <br> Sharath Turuvekere Sreenivas, Saurav Muralidharan, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, Pavlo Molchanov |<img width="1002" alt="image" src="https://arxiv.org/html/2408.11796v2/x1.png"> |[Paper](https://arxiv.org/abs/2408.11796)|[//]: #08/27
|[Language-specific Calibration for Pruning Multilingual Language Models](https://arxiv.org/abs/2408.14398) <br> Simon Kurz, Zhixue Zhao, Jian-Jia Chen, Lucie Flek ||[Paper](https://arxiv.org/abs/2408.14398)|[//]: #08/27
|[![Star](https://img.shields.io/github/stars/kriskrisliu/PAT_Pruning-Aware-Tuning.svg?style=social&label=Star)](https://github.com/kriskrisliu/PAT_Pruning-Aware-Tuning)<br>[PAT: Pruning-Aware Tuning for Large Language Models](https://arxiv.org/abs/2408.14721) <br> Yijiang Liu, Huanrui Yang, Youxin Chen, Rongyu Zhang, Miao Wang, Yuan Du, Li Du |<img width="1002" alt="image" src="figures/PAT.png"> |[Github](https://github.com/kriskrisliu/PAT_Pruning-Aware-Tuning) <br> [Paper](https://arxiv.org/abs/2408.14721)|[//]: #09/02
|[STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning](https://arxiv.org/abs/2409.06211) <br> Jaeseong Lee, seung-won hwang, Aurick Qiao, Daniel F Campos, Zhewei Yao, Yuxiong He |<img width="1002" alt="image" src="https://arxiv.org/html/2409.06211v1/x1.png"> |[Paper](https://arxiv.org/abs/2409.06211)|[//]: #09/13
|[Evaluating the Impact of Compression Techniques on Task-Specific Performance of Large Language Models](https://arxiv.org/abs/2409.11233) <br> Bishwash Khanal, Jeffery M. Capone |<img width="1002" alt="image" src="https://arxiv.org/html/2409.11233v1/extracted/5860861/images/GPT4template.jpg"> |[Paper](https://arxiv.org/abs/2409.11233)|[//]: #09/21
|[KVPruner: Structural Pruning for Faster and Memory-Efficient Large Language Models](https://arxiv.org/abs/2409.11057) <br> Bo Lv, Quan Zhou, Xuanang Ding, Yan Wang, Zeming Ma |<img width="302" alt="image" src="https://arxiv.org/html/2409.11057v1/x2.png"> |[Paper](https://arxiv.org/abs/2409.11057)|[//]: #09/21
|[OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition](https://arxiv.org/abs/2409.13652) <br> Stephen Zhang, Vardan Papyan | |[Paper](https://arxiv.org/abs/2409.13652)|[//]: #09/27
|[![Star](https://img.shields.io/github/stars/wyxscir/CFSP.svg?style=social&label=Star)](https://github.com/wyxscir/CFSP)<br>[CFSP: An Efficient Structured Pruning Framework for LLMs with Coarse-to-Fine Activation Information](https://arxiv.org/abs/2409.13199) <br> Yuxin Wang, Minghua Ma, Zekun Wang, Jingchang Chen, Huiming Fan, Liping Shan, Qing Yang, Dongliang Xu, Ming Liu, Bing Qin |<img width="1002" alt="image" src="https://arxiv.org/html/2409.13199v1/x1.png"> |[Github](https://github.com/wyxscir/CFSP) <br> [Paper](https://arxiv.org/abs/2409.13199)|[//]: #09/27
|[![Publish](https://img.shields.io/badge/Conference-NeurIPS'24-blue)]()<br>[Search for Efficient Large Language Models](https://arxiv.org/abs/2409.17372) <br> Xuan Shen, Pu Zhao, Yifan Gong, Zhenglun Kong, Zheng Zhan, Yushu Wu, Ming Lin, Chao Wu, Xue Lin, Yanzhi Wang |<img width="1002" alt="image" src="https://arxiv.org/html/2409.17372v1/x2.png"> |[Paper](https://arxiv.org/abs/2409.17372)|[//]: #09/27
|[![Star](https://img.shields.io/github/stars/NVlabs/MaskLLM.svg?style=social&label=Star)](https://github.com/NVlabs/MaskLLM) [![Publish](https://img.shields.io/badge/Conference-NeurIPS'24-blue)]() <br>[MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models](https://arxiv.org/abs/2409.17481) <br> Gongfan Fang, Hongxu Yin, Saurav Muralidharan, Greg Heinrich, Jeff Pool, Jan Kautz, Pavlo Molchanov, Xinchao Wang |<img width="302" alt="image" src="https://github.com/NVlabs/MaskLLM/blob/main/assets/animation-LQ.gif"> |[Github](https://github.com/NVlabs/MaskLLM) <br> [Paper](https://arxiv.org/abs/2409.17481)|[//]: #09/27
|[![Star](https://img.shields.io/github/stars/IntelLabs/Hardware-Aware-Automated-Machine-Learning.svg?style=social&label=Star)](https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/SQFT)[![Publish](https://img.shields.io/badge/Conference-EMNLP'24%20Findings-blue)]() [![Type](https://img.shields.io/badge/Unstructured-C2A4A6)]() [![Type](https://img.shields.io/badge/w/Quantization-39B0A9)]() <br>[SQFT: Low-cost Model Adaptation in Low-precision Sparse Foundation Models](https://arxiv.org/abs/2410.03750) <br> Juan Pablo Munoz, Jinjie Yuan, Nilesh Jain |<img width="1002" alt="image" src="figures/SQFT.png"> |[Github](https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/SQFT) <br> [Paper](https://arxiv.org/abs/2410.03750)|[//]: #10/01
|[Mitigating Copy Bias in In-Context Learning through Neuron Pruning](https://arxiv.org/abs/2410.01288) <br> Ameen Ali, Lior Wolf, Ivan Titov |<img width="1002" alt="image" src="figures/copy_icl.png"> |[Paper](https://arxiv.org/abs/2410.01288)|[//]: #10/04
|[![Star](https://img.shields.io/github/stars/abx393/llm-pruning-calibration-data.svg?style=social&label=Star)](https://github.com/abx393/llm-pruning-calibration-data)[![Publish](https://img.shields.io/badge/Conference-EMNLP'24-blue)]()<br>[Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruning](https://arxiv.org/abs/2410.07461) <br> Abhinav Bandari, Lu Yin, Cheng-Yu Hsieh, Ajay Kumar Jaiswal, Tianlong Chen, Li Shen, Ranjay Krishna, Shiwei Liu |<img width="1002" alt="image" src="https://arxiv.org/html/2410.07461v1/x1.png"> |[Github](https://github.com/abx393/llm-pruning-calibration-data) <br> [Paper](https://arxiv.org/abs/2410.07461)|[//]: #10/13
|[LLM-Rank: A Graph Theoretical Approach to Pruning Large Language Models](https://arxiv.org/abs/2410.13299) <br> David Hoffmann, Kailash Budhathoki, Matthaeus Kleindessner |<img width="1002" alt="image" src="https://arxiv.org/html/2410.13299v1/extracted/5931028/img/llm_to_mlp.png"> |[Paper](https://arxiv.org/abs/2410.13299)|[//]: #10/21
|[![Publish](https://img.shields.io/badge/Conference-NeurIPS'24-blue)]()<br>[DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models](https://arxiv.org/abs/2410.11988) <br> Shangqian Gao, Chi-Heng Lin, Ting Hua, Tang Zheng, Yilin Shen, Hongxia Jin, Yen-Chang Hsu |<img width="1002" alt="image" src="https://arxiv.org/html/2410.11988v1/x1.png"> |[Paper](https://arxiv.org/abs/2410.11988)|[//]: #10/21
|[Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix](https://arxiv.org/abs/2410.11261) <br> Yingyu Liang, Jiangxuan Long, Zhenmei Shi, Zhao Song, Yufa Zhou |<img width="1002" alt="image" src="https://arxiv.org/html/2410.11261v1/x1.png"> |[Paper](https://arxiv.org/abs/2410.11261)|[//]: #10/21
|[![Star](https://img.shields.io/github/stars/haiquanlu/AlphaPruning.svg?style=social&label=Star)](https://github.com/haiquanlu/AlphaPruning)[![Publish](https://img.shields.io/badge/Conference-NeurIPS'24-blue)]()<br>[AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models](https://arxiv.org/abs/2410.10912) <br> Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang, Michael W. Mahoney, Yaoqing Yang |<img width="1002" alt="image" src="https://arxiv.org/html/2410.10912v1/x1.png"> |[Github](https://github.com/haiquanlu/AlphaPruning) <br> [Paper](https://arxiv.org/abs/2410.10912)|[//]: #10/21
|[![Publish](https://img.shields.io/badge/Conference-NeurIPS'24%20Workshop-blue)]()<br>[Self-Data Distillation for Recovering Quality in Pruned Large Language Models](https://arxiv.org/abs/2410.09982) <br> Vithursan Thangarasa, Ganesh Venkatesh, Nish Sinnadurai, Sean Lie |<img width="1002" alt="image" src="https://arxiv.org/html/2410.09982v2/x1.png"> |[Paper](https://arxiv.org/abs/2410.09982)|[//]: #10/21
|[Self-calibration for Language Model Quantization and Pruning](https://arxiv.org/abs/2410.17170) <br> Miles Williams, George Chrysostomou, Nikolaos Aletras |<img width="1002" alt="image" src="https://arxiv.org/html/2410.17170v1/x1.png"> |[Paper](https://arxiv.org/abs/2410.17170)|[//]: #10/29
|[Beware of Calibration Data for Pruning Large Language Models](https://arxiv.org/abs/2410.17711) <br> Yixin Ji, Yang Xiang, Juntao Li, Qingrong Xia, Ping Li, Xinyu Duan, Zhefeng Wang, Min Zhang | |[Paper](https://arxiv.org/abs/2410.17711)|[//]: #10/29
|[![Star](https://img.shields.io/github/stars/piuzha/APT.svg?style=social&label=Star)](https://github.com/piuzha/APT)<br>[Pruning Foundation Models for High Accuracy without Retraining](https://arxiv.org/abs/2410.15567) <br> Pu Zhao, Fei Sun, Xuan Shen, Pinrui Yu, Zhenglun Kong, Yanzhi Wang, Xue Lin | |[Github](https://github.com/piuzha/APT) <br> [Paper](https://arxiv.org/abs/2410.15567)|[//]: #10/30
|[FedSpaLLM: Federated Pruning of Large Language Models](https://arxiv.org/abs/2410.14852) <br> Guangji Bai, Yijiang Li, Zilinghan Li, Liang Zhao, Kibaek Kim |<img width="1002" alt="image" src="https://arxiv.org/html/2410.14852v1/x1.png"> |[Paper](https://arxiv.org/abs/2410.14852)|[//]: #10/30
|[![Star](https://img.shields.io/github/stars/IST-DASLab/EvoPress.svg?style=social&label=Star)](https://github.com/IST-DASLab/EvoPress)<br>[EvoPress: Towards Optimal Dynamic Model Compression via Evolutionary Search](https://arxiv.org/abs/2410.14649) <br> Oliver Sieberling, Denis Kuznedelev, Eldar Kurtic, Dan Alistarh |<img width="1002" alt="image" src="figures/evopress.png"> |[Github](https://github.com/IST-DASLab/EvoPress) <br> [Paper](https://arxiv.org/abs/2410.14649)|[//]: #10/30
|[Beyond 2:4: exploring V:N:M sparsity for efficient transformer inference on GPUs](https://arxiv.org/abs/2410.16135) <br> Kang Zhao, Tao Yuan, Han Bao, Zhenfeng Su, Chang Gao, Zhaofeng Sun, Zichen Liang, Liping Jing, Jianfei Chen |<img width="1002" alt="image" src="https://arxiv.org/html/2410.16135v1/x1.png"> |[Paper](https://arxiv.org/abs/2410.16135)|[//]: #10/30
|[![Star](https://img.shields.io/github/stars/AboveParadise/LLMCBench.svg?style=social&label=Star)](https://github.com/AboveParadise/LLMCBench)<br>[LLMCBench: Benchmarking Large Language Model Compression for Efficient Deployment](https://arxiv.org/abs/2410.21352) <br> Ge Yang, Changyi He, Jinyang Guo, Jianyu Wu, Yifu Ding, Aishan Liu, Haotong Qin, Pengliang Ji, Xianglong Liu |<img width="1002" alt="image" src="https://github.com/AboveParadise/LLMCBench/raw/main/figs/f1.png"> |[Github](https://github.com/AboveParadise/LLMCBench) <br> [Paper](https://arxiv.org/abs/2410.21352)|[//]: #11/17
|[Tailored-LLaMA: Optimizing Few-Shot Learning in Pruned LLaMA Models with Task-Specific Prompts](https://arxiv.org/abs/2410.19185) <br> Danyal Aftab, Steven Davy |<img width="1002" alt="image" src="https://arxiv.org/html/2410.19185v1/x1.png"> |[Paper](https://arxiv.org/abs/2410.19185)|[//]: #11/18
|[![Star](https://img.shields.io/github/stars/thunlp/SparsingLaw.svg?style=social&label=Star)](https://github.com/thunlp/SparsingLaw)<br>[Sparsing Law: Towards Large Language Models with Greater Activation Sparsity](https://arxiv.org/abs/2411.02335) <br> Yuqi Luo, Chenyang Song, Xu Han, Yingfa Chen, Chaojun Xiao, Zhiyuan Liu, Maosong Sun |<img width="1002" alt="image" src="https://github.com/thunlp/SparsingLaw/raw/master/figs/sample.jpg"> |[Github](https://github.com/thunlp/SparsingLaw) <br> [Paper](https://arxiv.org/abs/2411.02335)|[//]: #11/18
|[AVSS: Layer Importance Evaluation in Large Language Models via Activation Variance-Sparsity Analysis](https://arxiv.org/abs/2411.02117) <br> Zichen Song, Yuxin Wu, Sitan Huang, Zhongfeng Kang |<img width="1002" alt="image" src="https://arxiv.org/html/2411.02117v1/x1.png"> |[Paper](https://arxiv.org/abs/2411.02117)|[//]: #11/18
|[![Star](https://img.shields.io/github/stars/hexuandeng/DRPruning.svg?style=social&label=Star)](https://github.com/hexuandeng/DRPruning)<br>[DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization](https://arxiv.org/abs/2411.14055) <br> Hexuan Deng, Wenxiang Jiao, Xuebo Liu, Min Zhang, Zhaopeng Tu |<img width="1002" alt="image" src="https://github.com/hexuandeng/DRPruning/raw/main/pic/main.png"> |[Github](https://github.com/hexuandeng/DRPruning) <br> [Paper](https://arxiv.org/abs/2411.14055)|[//]: #11/24
|[![Star](https://img.shields.io/github/stars/GATECH-EIC/AmoebaLLM.svg?style=social&label=Star)](https://github.com/GATECH-EIC/AmoebaLLM)[![Publish](https://img.shields.io/badge/Conference-NeurIPS'24-blue)]()<br>[AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deployment](https://arxiv.org/abs/2411.10606) <br> Yonggan Fu, Zhongzhi Yu, Junwei Li, Jiayi Qian, Yongan Zhang, Xiangchi Yuan, Dachuan Shi, Roman Yakunin, Yingyan Celine Lin |<img width="1002" alt="image" src="https://arxiv.org/html/2411.10606v1/x2.png"> |[Github](https://github.com/GATECH-EIC/AmoebaLLM) <br> [Paper](https://arxiv.org/abs/2411.10606)|[//]: #11/24
|[Scaling Law for Post-training after Model Pruning](https://arxiv.org/abs/2411.10272) <br> Xiaodong Chen, Yuxuan Hu, Jing Zhang, Xiaokang Zhang, Cuiping Li, Hong Chen | |[Paper](https://arxiv.org/abs/2411.10272)|[//]: #11/24
|[Layer Importance and Hallucination Analysis in Large Language Models via Enhanced Activation Variance-Sparsity](https://arxiv.org/abs/2411.10069) <br> Zichen Song, Sitan Huang, Yuxin Wu, Zhongfeng Kang |<img width="1002" alt="image" src="https://arxiv.org/html/2411.10069v1/x1.png"> |[Paper](https://arxiv.org/abs/2411.10069)|[//]: #11/24
|[![Star](https://img.shields.io/github/stars/yaolu-zjut/Navigation-LLM-layer-pruning.svg?style=social&label=Star)](https://github.com/yaolu-zjut/Navigation-LLM-layer-pruning)<br>[Reassessing Layer Pruning in LLMs: New Insights and Methods](https://arxiv.org/abs/2411.15558) <br> Yao Lu, Hao Cheng, Yujie Fang, Zeyu Wang, Jiaheng Wei, Dongwei Xu, Qi Xuan, Xiaoniu Yang, Zhaowei Zhu |<img width="1002" alt="image" src="https://github.com/yaolu-zjut/Navigation-LLM-layer-pruning/raw/main/framework.JPG"> |[Github](https://github.com/yaolu-zjut/Navigation-LLM-layer-pruning) <br> [Paper](https://arxiv.org/abs/2411.15558)|[//]: #12/03
|[Puzzle: Distillation-Based NAS for Inference-Optimized LLMs](https://arxiv.org/abs/2411.19146) <br> Akhiad Bercovich, Tomer Ronen, Talor Abramovich, Nir Ailon, Nave Assaf, Mohammad Dabbah et al |<img width="1002" alt="image" src="https://arxiv.org/html/2411.19146v2/x1.png"> |[Paper](https://arxiv.org/abs/2411.19146)|[//]: #12/09
|[Efficient LLM Inference using Dynamic Input Pruning and Cache-Aware Masking](https://arxiv.org/abs/2412.01380) <br> Marco Federici, Davide Belli, Mart van Baalen, Amir Jalalirad, Andrii Skliar, Bence Major, Markus Nagel, Paul Whatmough |<img width="1002" alt="image" src="https://arxiv.org/html/2412.01380v1/x1.png"> |[Paper](https://arxiv.org/abs/2412.01380)|[//]: #12/09
|[SlimGPT: Layer-wise Structured Pruning for Large Language Models](https://arxiv.org/abs/2412.18110) <br> Gui Ling, Ziyang Wang, Yuliang Yan, Qingwen Liu |<img width="302" alt="image" src="https://arxiv.org/html/2412.18110v1/x1.png"> |[Paper](https://arxiv.org/abs/2412.18110)|[//]: #12/30
|[Less is More: Towards Green Code Large Language Models via Unified Structural Pruning](https://arxiv.org/abs/2412.15921) <br> Guang Yang, Yu Zhou, Xiangyu Zhang, Wei Cheng, Ke Liu, Xiang Chen, Terry Yue Zhuo, Taolue Chen |<img width="1002" alt="image" src="figures/Flab-Pruner.png"> |[Paper](https://arxiv.org/abs/2412.15921)|[//]: #12/30
|[HashAttention: Semantic Sparsity for Faster Inference](https://arxiv.org/abs/2412.14468) <br> Aditya Desai, Shuo Yang, Alejandro Cuadron, Ana Klimovic, Matei Zaharia, Joseph E. Gonzalez, Ion Stoica |<img width="1002" alt="image" src="https://arxiv.org/html/2412.14468v1/extracted/6081011/images/sparseatt.png"> |[Paper](https://arxiv.org/abs/2412.14468)|[//]: #12/30
|[Adaptive Pruning for Large Language Models with Structural Importance Awareness](https://arxiv.org/abs/2412.15127) <br> Haotian Zheng, Jinke Ren, Yushan Sun, Ruichen Zhang, Wenbo Zhang, Zhen Li, Dusit Niyato, Shuguang Cui, Yatong Han |<img width="1002" alt="image" src="https://arxiv.org/html/2412.15127v1/x2.png"> |[Paper](https://arxiv.org/abs/2412.15127)|[//]: #12/30