

## Tuning
* [CPET: Effective Parameter-Efficient Tuning for Compressed Large Language Models](https://arxiv.org/abs/2307.07705). Weilin Zhao, Yuxiang Huang, Xu Han, Zhiyuan Liu, Zhengyan Zhang, Maosong Sun. [[Paper]](https://arxiv.org/abs/2307.07705)
* [![Star](https://img.shields.io/github/stars/liziniu/ReMax.svg?style=social&label=Star)](https://github.com/liziniu/ReMax) [ReMax: A Simple, Effective, and Efficient Method for Aligning Large Language Models](https://arxiv.org/abs/2310.10505). Ziniu Li, Tian Xu, Yushun Zhang, Yang Yu, Ruoyu Sun, Zhi-Quan Luo. [[Paper]](https://arxiv.org/abs/2310.10505)[[Github]](https://github.com/liziniu/ReMax)
* [TRANSOM: An Efficient Fault-Tolerant System for Training LLMs](https://arxiv.org/abs/2310.10046). Baodong Wu, Lei Xia, Qingping Li, Kangyu Li, Xu Chen, Yongqiang Guo, Tieyao Xiang, Yuheng Chen, Shigang Li. [[Paper]](https://arxiv.org/abs/2310.10046)
* [DEFT: Data Efficient Fine-Tuning for Large Language Models via Unsupervised Core-Set Selection](https://arxiv.org/abs/2310.16776). Devleena Das, Vivek Khetan. [[Paper]](https://arxiv.org/abs/2310.16776)
* [![Star](https://img.shields.io/github/stars/yangjianxin1/LongQLoRA.svg?style=social&label=Star)](https://github.com/yangjianxin1/LongQLoRA) [LongQLoRA: Efficient and Effective Method to Extend Context Length of Large Language Models](https://arxiv.org/abs/2311.04879). Jianxin Yang. [[Paper]](https://arxiv.org/abs/2311.04879)[[Github]](https://github.com/yangjianxin1/LongQLoRA)
* [![Star](https://img.shields.io/github/stars/ist-daslab/sparsefinetuning.svg?style=social&label=Star)](https://github.com/ist-daslab/sparsefinetuning) [Sparse Fine-tuning for Inference Acceleration of Large Language Models](https://arxiv.org/abs/2310.06927v2). Eldar Kurtic, Denis Kuznedelev, Elias Frantar, Michael Goin, Dan Alistarh. [[Paper]](https://arxiv.org/abs/2310.06927v2)[[Github]](https://github.com/ist-daslab/sparsefinetuning)[[Github]](https://github.com/neuralmagic/deepsparse/tree/main/research/mpt)
* [![Star](https://img.shields.io/github/stars/prateeky2806/compeft.svg?style=social&label=Star)](https://github.com/prateeky2806/compeft) [ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization](https://arxiv.org/abs/2311.13171). Prateek Yadav, Leshem Choshen, Colin Raffel, Mohit Bansal. [[Paper]](https://arxiv.org/abs/2311.13171)[[Github]](https://github.com/prateeky2806/compeft)
* [Towards Better Parameter-Efficient Fine-Tuning for Large Language Models: A Position Paper](https://arxiv.org/abs/2311.13126). Chengyu Wang, Junbing Yan, Wei Zhang, Jun Huang. [[Paper]](https://arxiv.org/abs/2311.13126)
* [![Star](https://img.shields.io/github/stars/ytgui/SPT-proto.svg?style=social&label=Star)](https://github.com/ytgui/SPT-proto) [SPT: Fine-Tuning Transformer-based Language Models Efficiently with Sparsification](https://arxiv.org/abs/2312.10365). Yuntao Gui, Xiao Yan, Peiqi Yin, Han Yang, James Cheng. [[Paper]](https://arxiv.org/abs/2312.10365)[[Github]](https://github.com/ytgui/SPT-proto)
* [![Star](https://img.shields.io/github/stars/nikhil-ghosh-berkeley/loraplus.svg?style=social&label=Star)](https://github.com/nikhil-ghosh-berkeley/loraplus) [LoRA+: Efficient Low Rank Adaptation of Large Models](https://arxiv.org/abs/2402.12354). Soufiane Hayou, Nikhil Ghosh, Bin Yu. [[Paper]](https://arxiv.org/abs/2402.12354)[[Github]](https://github.com/nikhil-ghosh-berkeley/loraplus)
* [Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning](https://arxiv.org/abs/2402.15751). Yong Liu, Zirui Zhu, Chaoyu Gong, Minhao Cheng, Cho-Jui Hsieh, Yang You. [[Paper]](https://arxiv.org/abs/2402.15751)
* [![Star](https://img.shields.io/github/stars/WooSunghyeon/dropbp.svg?style=social&label=Star)](https://github.com/WooSunghyeon/dropbp) [DropBP: Accelerating Fine-Tuning of Large Language Models by Dropping Backward Propagation](https://arxiv.org/abs/2402.17812). Sunghyeon Woo, Baeseong Park, Byeongwook Kim, Minjung Jo, Sejung Kwon, Dongsuk Jeon, Dongsoo Lee. [[Paper]](https://arxiv.org/abs/2402.17812)[[Github]](https://github.com/WooSunghyeon/dropbp)
* [LoRA-SP: Streamlined Partial Parameter Adaptation for Resource-Efficient Fine-Tuning of Large Language Models](https://arxiv.org/abs/2403.08822). Yichao Wu, Yafei Xiang, Shuning Huo, Yulu Gong, Penghao Liang. [[Paper]](https://arxiv.org/abs/2403.08822)
* [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://arxiv.org/abs/2403.14608). Zeyu Han, Chao Gao, Jinyang Liu, Jeff (Jun)Zhang, Sai Qian Zhang. [[Paper]](https://arxiv.org/abs/2403.14608)
* [![Publish](https://img.shields.io/badge/Conference-SemEval'24-blue)]() [![Star](https://img.shields.io/github/stars/ngregoriade/Semeval2024-Shroom.svg?style=social&label=Star)](https://github.com/ngregoriade/Semeval2024-Shroom) [AILS-NTUA at SemEval-2024 Task 6: Efficient model tuning for hallucination detection and analysis](https://arxiv.org/pdf/2404.01210.pdf). Natalia Griogoriadou, Maria Lymperaiou, Giorgos Filandrianos, Giorgos Stamou. [[Paper]](https://arxiv.org/pdf/2404.01210.pdf)[[Github]](https://github.com/ngregoriade/Semeval2024-Shroom)
* [![Star](https://img.shields.io/github/stars/Ledzy/BAdam.svg?style=social&label=Star)](https://github.com/Ledzy/BAdam) [BAdam: A Memory Efficient Full Parameter Training Method for Large Language Models](https://arxiv.org/abs/2404.02827). Qijun Luo, Hengxu Yu, Xiao Li. [[Paper]](https://arxiv.org/abs/2404.02827)[[Github]](https://github.com/Ledzy/BAdam)
* [Intuition-aware Mixture-of-Rank-1-Experts for Parameter Efficient Finetuning](https://arxiv.org/abs/2404.08985). Yijiang Liu, Rongyu Zhang, Huanrui Yang, Kurt Keutzer, Yuan Du, Li Du, Shanghang Zhang. [[Paper]](https://arxiv.org/abs/2404.08985)
* [![Publish](https://img.shields.io/badge/Conference-ICML'24-blue)]() [![Star](https://img.shields.io/github/stars/JingXuTHU/Random-Masking-Finds-Winning-Tickets-for-Parameter-Efficient-Fine-tuning.svg?style=social&label=Star)](https://github.com/JingXuTHU/Random-Masking-Finds-Winning-Tickets-for-Parameter-Efficient-Fine-tuning) [Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuning](https://arxiv.org/abs/2405.02596). Jing Xu, Jingzhao Zhang. [[Paper]](https://arxiv.org/abs/2405.02596)[[Github]](https://github.com/JingXuTHU/Random-Masking-Finds-Winning-Tickets-for-Parameter-Efficient-Fine-tuning)
* [Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsity](https://arxiv.org/abs/2406.02913). Wentao Guo, Jikai Long, Yimeng Zeng, Zirui Liu, Xinyu Yang, Yide Ran, Jacob R. Gardner, Osbert Bastani, Christopher De Sa, Xiaodong Yu, Beidi Chen, Zhaozhuo Xu. [[Paper]](https://arxiv.org/abs/2406.02913)
* [![Publish](https://img.shields.io/badge/Conference-ACL'24%20Findings-blue)]() [![Star](https://img.shields.io/github/stars/gccnlp/Light-PEFT.svg?style=social&label=Star)](https://github.com/gccnlp/Light-PEFT) [Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruning](https://arxiv.org/abs/2406.03792). Naibin Gu, Peng Fu, Xiyu Liu, Bowen Shen, Zheng Lin, Weiping Wang. [[Paper]](https://arxiv.org/abs/2406.03792)[[Github]](https://github.com/gccnlp/Light-PEFT)
* [BlockLLM: Memory-Efficient Adaptation of LLMs by Selecting and Optimizing the Right Coordinate Blocks](https://arxiv.org/abs/2406.17296). Amrutha Varshini Ramesh, Vignesh Ganapathiraman, Issam H. Laradji, Mark Schmidt. [[Paper]](https://arxiv.org/abs/2406.17296)[[Github]](https://github.com/RAmruthaVignesh/blockllm)
* [Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead](https://arxiv.org/abs/2407.00066). Rickard Brüel-Gabrielsson, Jiacheng Zhu, Onkar Bhardwaj, Leshem Choshen, Kristjan Greenewald, Mikhail Yurochkin, Justin Solomon. [[Paper]](https://arxiv.org/abs/2407.00066)
* [![Publish](https://img.shields.io/badge/Conference-ICLR'24-blue)]() [![Star](https://img.shields.io/github/stars/LINs-lab/CapaBoost.svg?style=social&label=Star)](https://github.com/LINs-lab/CapaBoost) [Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuning](https://arxiv.org/abs/2407.01320). Haobo Song, Hao Zhao, Soumajit Majumder, Tao Lin. [[Paper]](https://arxiv.org/abs/2407.01320)[[Github]](https://github.com/LINs-lab/CapaBoost)
* [![Publish](https://img.shields.io/badge/Conference-ACL'24%20PrivateNLP-blue)]() [PocketLLM: Enabling On-Device Fine-Tuning for Personalized LLMs](https://arxiv.org/abs/2407.01031). Dan Peng, Zhihui Fu, Jun Wang. [[Paper]](https://arxiv.org/abs/2407.01031)
* [Code Less, Align More: Efficient LLM Fine-tuning for Code Generation with Data Pruning](https://arxiv.org/abs/2407.05040). Yun-Da Tsai, Mingjie Liu, Haoxing Ren. [[Paper]](https://arxiv.org/abs/2407.05040)
* [Tensor Train Low-rank Approximation (TT-LoRA): Democratizing AI with Accelerated LLMs](https://arxiv.org/abs/2408.01008). Afia Anjum, Maksim E. Eren, Ismael Boureima, Boian Alexandrov, Manish Bhattarai. [[Paper]](https://arxiv.org/abs/2408.01008)
* [Enabling Resource-Efficient On-Device Fine-Tuning of LLMs Using Only Inference Engines](https://arxiv.org/abs/2409.15520). Lei Gao, Amir Ziashahabi, Yue Niu, Salman Avestimehr, Murali Annavaram. [[Paper]](https://arxiv.org/abs/2409.15520)
* [![Star](https://img.shields.io/github/stars/JL-er/Bone.svg?style=social&label=Star)](https://github.com/JL-er/Bone) [Bone: Block Affine Transformation as Parameter Efficient Fine-tuning Methods for Large Language Models](https://arxiv.org/abs/2409.15371). Jiale Kang. [[Paper]](https://arxiv.org/abs/2409.15371)[[Github]](https://github.com/JL-er/Bone)
* [![Star](https://img.shields.io/github/stars/sayankotor/sparse_grads.svg?style=social&label=Star)](https://github.com/sayankotor/sparse_grads) [SparseGrad: A Selective Method for Efficient Fine-tuning of MLP Layers](https://arxiv.org/abs/2410.07383). Viktoriia Chekalina, Anna Rudenko, Gleb Mezentsev, Alexander Mikhalev, Alexander Panchenko, Ivan Oseledets. [[Paper]](https://arxiv.org/abs/2410.07383)[[Github]](https://github.com/sayankotor/sparse_grads)
* [SpaLLM: Unified Compressive Adaptation of Large Language Models with Sketching](https://arxiv.org/abs/2410.06364). Tianyi Zhang, Junda Su, Oscar Wu, Zhaozhuo Xu, Anshumali Shrivastava. [[Paper]](https://arxiv.org/abs/2410.06364)
* [![Publish](https://img.shields.io/badge/Conference-EMNLP'24-blue)]() [![Star](https://img.shields.io/github/stars/Kaiseem/IST.svg?style=social&label=Star)](https://github.com/Kaiseem/IST) [Layer-wise Importance Matters: Less Memory for Better Performance in Parameter-efficient Fine-tuning of Large Language Models](https://arxiv.org/abs/2410.11772). Kai Yao, Penlei Gao, Lichun Li, Yuan Zhao, Xiaofeng Wang, Wei Wang, Jianke Zhu. [[Paper]](https://arxiv.org/abs/2410.11772)[[Github]](https://github.com/Kaiseem/IST)
* [![Publish](https://img.shields.io/badge/Conference-Nature%20Scientific%20Reports-blue)]() [Parameter-Efficient Fine-Tuning of Large Language Models using Semantic Knowledge Tuning](https://arxiv.org/abs/2410.08598). Nusrat Jahan Prottasha, Asif Mahmud, Md. Shohanur Islam Sobuj, Prakash Bhat, Md Kowsher, Niloofar Yousefi, Ozlem Ozmen Garibay. [[Paper]](https://arxiv.org/abs/2410.08598)
* [![Publish](https://img.shields.io/badge/Conference-EMNLP'24%20Findings-blue)]() [![Star](https://img.shields.io/github/stars/xvyaward/qeft.svg?style=social&label=Star)](https://github.com/xvyaward/qeft) [QEFT: Quantization for Efficient Fine-Tuning of LLMs](https://arxiv.org/abs/2410.08661). Changhun Lee, Jun-gyu Jin, Younghyun Cho, Eunhyeok Park. [[Paper]](https://arxiv.org/abs/2410.08661)[[Github]](https://github.com/xvyaward/qeft)
* [![Publish](https://img.shields.io/badge/Conference-EMNLP'24%20Findings-blue)]() [![Star](https://img.shields.io/github/stars/Aofei-Chang/BIPEFT.svg?style=social&label=Star)](https://github.com/Aofei-Chang/BIPEFT) [BIPEFT: Budget-Guided Iterative Search for Parameter Efficient Fine-Tuning of Large Pretrained Language Models](https://arxiv.org/abs/2410.09079). Aofei Chang, Jiaqi Wang, Han Liu, Parminder Bhatia, Cao Xiao, Ting Wang, Fenglong Ma. [[Paper]](https://arxiv.org/abs/2410.09079)[[Github]](https://github.com/Aofei-Chang/BIPEFT)
* [![Star](https://img.shields.io/github/stars/Kowsher/RoCoFT.svg?style=social&label=Star)](https://github.com/Kowsher/RoCoFT) [RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updates](https://arxiv.org/abs/2410.10075). Md Kowsher, Tara Esmaeilbeig, Chun-Nam Yu, Mojtaba Soltanalian, Niloofar Yousefi. [[Paper]](https://arxiv.org/abs/2410.10075)[[Github]](https://github.com/Kowsher/RoCoFT)
* [![Publish](https://img.shields.io/badge/Conference-EMNLP'24%20Findings-blue)]() [MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning](https://arxiv.org/abs/2410.18035). Jingfan Zhang, Yi Zhao, Dan Chen, Xing Tian, Huanran Zheng, Wei Zhu. [[Paper]](https://arxiv.org/abs/2410.18035)
* [Less is More: Extreme Gradient Boost Rank-1 Adaption for Efficient Finetuning of LLMs](https://arxiv.org/abs/2410.19694). Yifei Zhang, Hao Zhu, Aiwei Liu, Han Yu, Piotr Koniusz, Irwin King. [[Paper]](https://arxiv.org/abs/2410.19694)
* [![Star](https://img.shields.io/github/stars/LCS2-IIITD/MonteCLoRA.svg?style=social&label=Star)](https://github.com/LCS2-IIITD/MonteCLoRA) [Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptation](https://arxiv.org/abs/2411.04358). Ayan Sengupta, Vaibhav Seth, Arinjay Pathak, Natraj Raman, Sriram Gopalakrishnan, Tanmoy Chakraborty. [[Paper]](https://arxiv.org/abs/2411.04358)[[Github]](https://github.com/LCS2-IIITD/MonteCLoRA)
* [AutoMixQ: Self-Adjusting Quantization for High Performance Memory-Efficient Fine-Tuning](https://arxiv.org/abs/2411.13814). Changhai Zhou, Shiyang Zhang, Yuhua Zhou, Zekai Liu, Shichao Weng. [[Paper]](https://arxiv.org/abs/2411.13814)
* [HELENE: Hessian Layer-wise Clipping and Gradient Annealing for Accelerating Fine-tuning LLM with Zeroth-order Optimization](https://arxiv.org/abs/2411.10696). Huaqin Zhao, Jiaxi Li, Yi Pan, Shizhe Liang, Xiaofeng Yang, Wei Liu, Xiang Li, Fei Dou, Tianming Liu, Jin Lu. [[Paper]](https://arxiv.org/abs/2411.10696)
