
## Low-Rank Decomposition
| Title & Authors | Introduction | Links |
|:--|  :----: | :---:|
|[![Star](https://img.shields.io/github/stars/yxli2123/LoSparse.svg?style=social&label=Star)](https://github.com/yxli2123/LoSparse) [![Publish](https://img.shields.io/badge/Conference-ICML'23-blue)]() <br>[LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation](https://arxiv.org/abs/2306.11222) <br> Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, Tuo Zhao |<img width="302" alt="image" src="figures/LoSparse.png"> |[Github](https://github.com/yxli2123/LoSparse) <br> [Paper](https://arxiv.org/abs/2306.11222)|
|[![Star](https://img.shields.io/github/stars/pilancilab/matrix-compressor.svg?style=social&label=Star)](https://github.com/pilancilab/matrix-compressor)[![Publish](https://img.shields.io/badge/Conference-NeurIPS'23-blue)]()<br>[Matrix Compression via Randomized Low Rank and Low Precision Factorization](https://arxiv.org/abs/2310.11028) <br> Rajarshi Saha, Varun Srivastava, Mert Pilanci |<img width="1002" alt="image" src="figures/LPLR.png"> |[Github](https://github.com/pilancilab/matrix-compressor) <br> [Paper](https://arxiv.org/abs/2310.11028)|
|[TensorGPT: Efficient Compression of the Embedding Layer in LLMs based on the Tensor-Train Decomposition](https://arxiv.org/abs/2307.00526) <br> Mingxue Xu, Yao Lei Xu, Danilo P. Mandic |<img width="1002" alt="image" src="figures/TT-SVD.png"> |[Paper](https://arxiv.org/abs/2307.00526)|
|[LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compression](https://arxiv.org/abs/2309.14021) <br> Ayush Kaushal, Tejas Vaidhya, Irina Rish |<img width="302" alt="image" src="figures/LoRD.png"> |[Paper](https://arxiv.org/abs/2309.14021)<br>[Project](https://huggingface.co/nolanoAI)|
|[![Star](https://img.shields.io/github/stars/algorithms/llm-rom.svg?style=social&label=Star)](https://github.com/algorithms/llm-rom)<br>[Rethinking Compression: Reduced Order Modelling of Latent Features in Large Language Models](https://arxiv.org/abs/2312.07046) <br> Arnav Chavan, Nahush Lele, Deepak Gupta |<img width="1002" alt="image" src="figures/LLM-ROM.png"> |[Github](https://github.com/transmuteAI/trailmet/tree/main/trailmet/algorithms/llm-rom) <br> [Paper](https://arxiv.org/abs/2312.07046)|
|[Data-free Weight Compress and Denoise for Large Language Models](https://arxiv.org/abs/2402.16319) <br> Runyu Peng, Yunhua Zhou, Qipeng Guo, Yang Gao, Hang Yan, Xipeng Qiu, Dahua Lin |<img width="1002" alt="image" src="https://arxiv.org/html/2402.16319v1/extracted/5401579/icml2024/denoise.png"> |[Paper](https://arxiv.org/abs/2402.16319)|
|[![Star](https://img.shields.io/github/stars/AIoT-MLSys-Lab/SVD-LLM.svg?style=social&label=Star)](https://github.com/AIoT-MLSys-Lab/SVD-LLM)<br>[SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression](https://arxiv.org/abs/2403.07378) <br> Xin Wang, Yu Zheng, Zhongwei Wan, Mi Zhang |<img width="1002" alt="image" src="https://github.com/AIoT-MLSys-Lab/SVD-LLM/blob/main/figures/framework.jpg"> |[Github](https://github.com/AIoT-MLSys-Lab/SVD-LLM) <br> [Paper](https://arxiv.org/abs/2403.07378)|
|[![Star](https://img.shields.io/github/stars/Dereck0602/Bolaco.svg?style=social&label=Star)](https://github.com/Dereck0602/Bolaco)[![Publish](https://img.shields.io/badge/Conference-ACL'24%20Findings-blue)]()<br>[Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimization](https://arxiv.org/abs/2405.10616) <br> Yixin Ji, Yang Xiang, Juntao Li, Wei Chen, Zhongyi Liu, Kehai Chen, Min Zhang |<img width="1002" alt="image" src="figures/bolaco.png"> |[Github](https://github.com/Dereck0602/Bolaco) <br> [Paper](https://arxiv.org/abs/2405.10616)|
|[![Star](https://img.shields.io/github/stars/nyunAI/SFSD-LLM.svg?style=social&label=Star)](https://github.com/nyunAI/SFSD-LLM)[![Publish](https://img.shields.io/badge/Conference-ACL'24-blue)]()<br>[Surgical Feature-Space Decomposition of LLMs: Why, When and How?](https://arxiv.org/abs/2405.13039) <br> Arnav Chavan, Nahush Lele, Deepak Gupta |<img width="1002" alt="image" src="figures/SFSD-LLM.png"> |[Github](https://github.com/nyunAI/SFSD-LLM) <br> [Paper](https://arxiv.org/abs/2405.13039)|
|[MCNC: Manifold Constrained Network Compression](https://arxiv.org/abs/2406.19301) <br> Chayne Thrash, Ali Abbasi, Parsa Nooralinejad, Soroush Abbasi Koohpayegani, Reed Andreas, Hamed Pirsiavash, Soheil Kolouri |<img width="1002" alt="image" src="https://arxiv.org/html/2406.19301v1/x1.png"> |[Paper](https://arxiv.org/abs/2406.19301)|[//]: #06/28
|[MoDeGPT: Modular Decomposition for Large Language Model Compression](https://arxiv.org/abs/2408.09632) <br> Chi-Heng Lin, Shangqian Gao, James Seale Smith, Abhishek Patel, Shikhar Tuli, Yilin Shen, Hongxia Jin, Yen-Chang Hsu |<img width="1002" alt="image" src="https://arxiv.org/html/2408.09632v1/x2.png"> |[Paper](https://arxiv.org/abs/2408.09632)|[//]: #08/20
|[![Publish](https://img.shields.io/badge/Conference-NeurIPS'24-blue)]()<br>[ESPACE: Dimensionality Reduction of Activations for Model Compression](https://arxiv.org/abs/2410.05437) <br> Charbel Sakr, Brucek Khailany |<img width="1002" alt="image" src="figures/ESPACE.png"> |[Paper](https://arxiv.org/abs/2410.05437)|[//]: #10/14
|[CompAct: Compressed Activations for Memory-Efficient LLM Training](https://arxiv.org/abs/2410.15352) <br> Yara Shamshoum, Nitzan Hodos, Yuval Sieradzki, Assaf Schuster |<img width="202" alt="image" src="https://arxiv.org/html/2410.15352v1/x1.png"> |[Paper](https://arxiv.org/abs/2410.15352)|[//]: #10/30
|[![Star](https://img.shields.io/github/stars/selfsupervised-ai/Natural-GaLore.svg?style=social&label=Star)](https://github.com/selfsupervised-ai/Natural-GaLore)<br>[Natural GaLore: Accelerating GaLore for memory-efficient LLM Training and Fine-tuning](https://arxiv.org/abs/2410.16029) <br> Arijit Das | |[Github](https://github.com/selfsupervised-ai/Natural-GaLore) <br> [Paper](https://arxiv.org/abs/2410.16029)|[//]: #10/30