# Efficient-PLM

A curated list for **Efficient Pre-trained Language Models**:
  - [Knowledge Distillation](#knowledge-distillation)
  - [Network Pruning](#network-pruning)
  - [Quantization](#quantization)
  - [Inference Acceleration](#inference-acceleration)
  - [Structure Design](#structure-design)
  - [Hardware](#hardware)
  - [Evaluation](#evaluation)
  - [Others](#others)

This part is still under construction to include papers published from 2018-2023.



## Knowledge Distillation
* [![Publish](https://img.shields.io/badge/Conference-NAACL'24-blue)]() [![Star](https://img.shields.io/github/stars/wutaiqiang/WID-NAACL2024.svg?style=social&label=Star)](https://github.com/wutaiqiang/WID-NAACL2024) [Weight-Inherited Distillation for Task-Agnostic BERT Compression](https://arxiv.org/abs/2305.09098). Taiqiang Wu, Cheng Hou, Shanshan Lao, Jiayi Li, Ngai Wong, Zhe Zhao, Yujiu Yang. [[Paper]](https://arxiv.org/abs/2305.09098)[[Github]](https://github.com/wutaiqiang/WID-NAACL2024)
* [![Publish](https://img.shields.io/badge/Conference-ACL'23-blue)]() [![Star](https://img.shields.io/github/stars/nitaytech/KD4Gen.svg?style=social&label=Star)](https://github.com/nitaytech/KD4Gen) [A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Training](https://arxiv.org/abs/2305.02031). Nitay Calderon, Subhabrata Mukherjee, Roi Reichart, Amir Kantor. [[Paper]](https://arxiv.org/abs/2305.02031)[[Github]](https://github.com/nitaytech/KD4Gen)
* [![Publish](https://img.shields.io/badge/Conference-ACL'23-blue)]() [![Star](https://img.shields.io/github/stars/twinkle0331/LGTM.svg?style=social&label=Star)](https://github.com/twinkle0331/LGTM) [Tailoring Instructions to Student's Learning Levels Boosts Knowledge Distillation](https://arxiv.org/abs/2305.09651). Yuxin Ren, Zihan Zhong, Xingjian Shi, Yi Zhu, Chun Yuan, Mu Li. [[Paper]](https://arxiv.org/abs/2305.09651)[[Github]](https://github.com/twinkle0331/LGTM)
* [![Publish](https://img.shields.io/badge/Conference-ACL'23-blue)]() [![Star](https://img.shields.io/github/stars/MANGA-UOFA/fdistill.svg?style=social&label=Star)](https://github.com/MANGA-UOFA/fdistill) [f-Divergence Minimization for Sequence-Level Knowledge Distillation](https://aclanthology.org/2023.acl-long.605/). Yuqiao Wen, Zichao Li, Wenyu Du, Lili Mou. [[Paper]](https://aclanthology.org/2023.acl-long.605/)[[Github]](https://github.com/MANGA-UOFA/fdistill)
* [![Publish](https://img.shields.io/badge/Conference-ACL'23-blue)]() [![Star](https://img.shields.io/github/stars/brucewsy/AD-KD.svg?style=social&label=Star)](https://github.com/brucewsy/AD-KD) [AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression](https://arxiv.org/abs/2305.10010). Siyue Wu, Hongzhan Chen, Xiaojun Quan, Qifan Wang, Rui Wang. [[Paper]](https://arxiv.org/abs/2305.10010)[[Github]](https://github.com/brucewsy/AD-KD)
* [![Publish](https://img.shields.io/badge/Conference-ACL'23-blue)]() [![Star](https://img.shields.io/github/stars/GeneZC/MiniMoE.svg?style=social&label=Star)](https://github.com/GeneZC/MiniMoE) [Lifting the Curse of Capacity Gap in Distilling Language Models](https://arxiv.org/abs/2305.12129). Chen Zhang, Yang Yang, Jiahao Liu, Jingang Wang, Yunsen Xian, Benyou Wang, Dawei Song. [[Paper]](https://arxiv.org/abs/2305.12129)[[Github]](https://github.com/GeneZC/MiniMoE)
* [![Publish](https://img.shields.io/badge/Conference-ACL'23-blue)]() [![Star](https://img.shields.io/github/stars/thunlp-mt/dbkd-plm.svg?style=social&label=Star)](https://github.com/thunlp-mt/dbkd-plm) [Bridging the Gap between Decision and Logits in Decision-based Knowledge Distillation for Pre-trained Language Models](https://arxiv.org/abs/2306.08909). Qinhong Zhou, Zonghan Yang, Peng Li, Yang Liu. [[Paper]](https://arxiv.org/abs/2306.08909)[[Github]](https://github.com/thunlp-mt/dbkd-plm)
* [![Publish](https://img.shields.io/badge/Conference-ACL'23-blue)]() [![Star](https://img.shields.io/github/stars/mainlp/How-to-distill-your-BERT.svg?style=social&label=Star)](https://github.com/mainlp/How-to-distill-your-BERT) [How to Distill your BERT: An Empirical Study on the Impact of Weight Initialisation and Distillation Objectives](https://arxiv.org/abs/2305.15032). Xinpeng Wang, Leonie Weissweiler, Hinrich Schütze, Barbara Plank. [[Paper]](https://arxiv.org/abs/2305.15032)[[Github]](https://github.com/mainlp/How-to-distill-your-BERT)
* [![Publish](https://img.shields.io/badge/Conference-ACL'23-blue)]() [ReAugKD: Retrieval-augmented knowledge distillation for pre-trained language models](https://www.amazon.science/publications/reaugkd-retrieval-augmented-knowledge-distillation-for-pre-trained-language-models). Jianyi Zhang, Aashiq Muhamed, Aditya Anantharaman, Guoyin Wang, Changyou Chen, Kai Zhong, Qingjun Cui, Yi Xu, Belinda Zeng, Trishul Chilimbi, Yiran Chen. [[Paper]](https://www.amazon.science/publications/reaugkd-retrieval-augmented-knowledge-distillation-for-pre-trained-language-models)
* [![Publish](https://img.shields.io/badge/Conference-ICML'23-blue)]() [![Star](https://img.shields.io/github/stars/cliang1453/task-aware-distillation.svg?style=social&label=Star)](https://github.com/cliang1453/task-aware-distillation) [Less is More: Task-aware Layer-wise Distillation for Language Model Compression](https://openreview.net/forum?id=B6J3ygdPJh). Chen Liang, Simiao Zuo, Qingru Zhang, Pengcheng He, Weizhu Chen, Tuo Zhao. [[Paper]](https://openreview.net/forum?id=B6J3ygdPJh)[[Github]](https://github.com/cliang1453/task-aware-distillation)
* [![Publish](https://img.shields.io/badge/Conference-ACL'23%20Findings-blue)]() [Adaptive Contrastive Knowledge Distillation for BERT Compression](https://aclanthology.org/2023.findings-acl.569/). Jinyang Guo, Jiaheng Liu, Zining Wang, Yuqing Ma, Ruihao Gong, Ke Xu, Xianglong Liu. [[Paper]](https://aclanthology.org/2023.findings-acl.569/)
* [![Publish](https://img.shields.io/badge/Conference-ACL'23%20Findings-blue)]() [Distilling Reasoning Capabilities into Smaller Language Models](https://arxiv.org/abs/2212.00193). Kumar Shridhar, Alessandro Stolfo, Mrinmaya Sachan. [[Paper]](https://arxiv.org/abs/2212.00193)
* [![Publish](https://img.shields.io/badge/Conference-ACL'23%20Findings-blue)]() [![Star](https://img.shields.io/github/stars/aitsc/glmkd.svg?style=social&label=Star)](https://github.com/aitsc/glmkd) [Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Method](https://arxiv.org/abs/2306.06625). Shicheng Tan, Weng Lam Tam, Yuanchun Wang, Wenwen Gong, Shu Zhao, Peng Zhang, Jie Tang. [[Paper]](https://arxiv.org/abs/2306.06625)[[Github]](https://github.com/aitsc/glmkd)
* [![Publish](https://img.shields.io/badge/Conference-ACL'23%20Findings-blue)]() [![Star](https://img.shields.io/github/stars/tiiiger/templm.svg?style=social&label=Star)](https://github.com/tiiiger/templm) [TempLM: Distilling Language Models into Template-Based Generators](https://arxiv.org/abs/2205.11055). Tianyi Zhang, Mina Lee, Lisa Li, Ende Shen, Tatsunori B. Hashimoto. [[Paper]](https://arxiv.org/abs/2205.11055)[[Github]](https://github.com/tiiiger/templm)
* [![Publish](https://img.shields.io/badge/Conference-ACL'23%20Findings-blue)]() [Cost-effective Distillation of Large Language Models](https://aclanthology.org/2023.findings-acl.463/). Sayantan Dasgupta, Trevor Cohn, Timothy Baldwin. [[Paper]](https://aclanthology.org/2023.findings-acl.463/)
* [![Publish](https://img.shields.io/badge/Conference-ACL'23%20Findings-blue)]() [A Study on Knowledge Distillation from Weak Teacher for Scaling Up Pre-trained Language Models](https://arxiv.org/abs/2305.18239). Hayeon Lee, Rui Hou, Jongpil Kim, Davis Liang, Sung Ju Hwang, Alexander Min. [[Paper]](https://arxiv.org/abs/2305.18239)
* [![Publish](https://img.shields.io/badge/Conference-EMNLP'23%20Industry%20Track-blue)]() [A Comparative Analysis of Task-Agnostic Distillation Methods for Compressing Transformer Language Models](https://arxiv.org/abs/2310.08797). Takuma Udagawa, Aashka Trivedi, Michele Merler, Bishwaranjan Bhattacharjee. [[Paper]](https://arxiv.org/abs/2310.08797)
* [![Publish](https://img.shields.io/badge/Conference-NeurIPS'23%20ENLSP-blue)]() [What is Lost in Knowledge Distillation?](https://arxiv.org/abs/2311.04142). Manas Mohanty, Tanya Roosta, Peyman Passban. [[Paper]](https://arxiv.org/abs/2311.04142)
* [![Publish](https://img.shields.io/badge/Conference-EMNLP'23%20Findings-blue)]() [Co-training and Co-distillation for Quality Improvement and Compression of Language Models](https://arxiv.org/abs/2311.02849). Hayeon Lee, Rui Hou, Jongpil Kim, Davis Liang, Hongbo Zhang, Sung Ju Hwang, Alexander Min. [[Paper]](https://arxiv.org/abs/2311.02849)
* [![Publish](https://img.shields.io/badge/Conference-NAACL'24%20workshop-blue)]() [![Star](https://img.shields.io/github/stars/MinhDucBui/revisiting_distillation.svg?style=social&label=Star)](https://github.com/MinhDucBui/revisiting_distillation) [Knowledge Distillation vs. Pretraining from Scratch under a Fixed (Computation) Budget](https://arxiv.org/abs/2404.19319). Minh Duc Bui, Fabian David Schmidt, Goran Glavaš, Katharina von der Wense. [[Paper]](https://arxiv.org/abs/2404.19319)[[Github]](https://github.com/MinhDucBui/revisiting_distillation)
* [Self-Knowledge Distillation for Learning Ambiguity](https://arxiv.org/abs/2406.09719). Hancheol Park, Soyeong Jeong, Sukmin Cho, Jong C. Park. [[Paper]](https://arxiv.org/abs/2406.09719)

## Network Pruning
* [![Publish](https://img.shields.io/badge/Conference-ACL'23-blue)]() [![Star](https://img.shields.io/github/stars/kongds/SMP.svg?style=social&label=Star)](https://github.com/allenai/kongds/SMP) [Pruning Pre-trained Language Models Without Fine-Tuning](https://aclanthology.org/2023.acl-long.35/). Ting Jiang, Deqing Wang, Fuzhen Zhuang, Ruobing Xie, Feng Xia. [[Paper]](https://aclanthology.org/2023.acl-long.35/) [[Github]](https://github.com/kongds/SMP) 
* [![Publish](https://img.shields.io/badge/Conference-ACL'23-blue)]() [![Star](https://img.shields.io/github/stars/airaria/GRAIN.svg?style=social&label=Star)](https://github.com/airaria/GRAIN) [Gradient-based Intra-attention Pruning on Pre-trained Language Models](https://arxiv.org/abs/2212.07634). Ziqing Yang, Yiming Cui, Xin Yao, Shijin Wang. [[Paper]](https://arxiv.org/abs/2212.07634)[[Github]](https://github.com/airaria/GRAIN)
* [![Publish](https://img.shields.io/badge/Conference-ACL'23-blue)]() [![Star](https://img.shields.io/github/stars/naver/nllb-pruning.svg?style=social&label=Star)](https://github.com/naver/nllb-pruning) [Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Model](https://arxiv.org/abs/2212.09811). Yeskendir Koishekenov, Alexandre Berard, Vassilina Nikoulina. [[Paper]](https://arxiv.org/abs/2212.09811)[[Github]](https://github.com/naver/nllb-pruning)
* [![Publish](https://img.shields.io/badge/Conference-ACL'23%20Findings-blue)]() [Structured Pruning for Efficient Generative Pre-trained Language Models](https://aclanthology.org/2023.findings-acl.692/). 
Chaofan Tao, Lu Hou, Haoli Bai, Jiansheng Wei, Xin Jiang, Qun Liu, Ping Luo, Ngai Wong. [[Paper]](https://aclanthology.org/2023.findings-acl.692/) 
* [![Publish](https://img.shields.io/badge/Conference-ACL'23%20Findings-blue)]() [![Star](https://img.shields.io/github/stars/DRSY/PINS.svg?style=social&label=Star)](https://github.com/DRSY/PINS) [Pruning Pre-trained Language Models with Principled Importance and Self-regularization](https://aclanthology.org/2023.findings-acl.573/). Siyu Ren, Kenny Zhu. [[Paper]](https://aclanthology.org/2023.findings-acl.573/)[[Github]](https://github.com/DRSY/PINS)
* [Knowledge-preserving Pruning for Pre-trained Language Models without Retraining](https://arxiv.org/abs/2308.03449). Seungcheol Park, Hojun Choi, U Kang. [[Paper]](https://arxiv.org/abs/2308.03449)
* [Towards Robust Pruning: An Adaptive Knowledge-Retention Pruning Strategy for Language Models](https://arxiv.org/abs/2310.13191). Jianwei Li, Qi Lei, Wei Cheng, Dongkuan Xu. [[Paper]](https://arxiv.org/abs/2310.13191)
* [Transfer Learning for Structured Pruning under Limited Task Data](https://arxiv.org/abs/2311.06382). Lucio Dery, David Grangier, Awni Hannun. [[Paper]](https://arxiv.org/abs/2311.06382)
* [![Publish](https://img.shields.io/badge/Conference-NeurIPS'23%20MLNCP-blue)]() [Activity Sparsity Complements Weight Sparsity for Efficient RNN Inference](https://arxiv.org/abs/2311.07625). Rishav Mukherji, Mark Schöne, Khaleelulla Khan Nazeer, Christian Mayr, Anand Subramoney. [[Paper]](https://arxiv.org/abs/2311.07625)
* [DSFormer: Effective Compression of Text-Transformers by Dense-Sparse Weight Factorization](https://arxiv.org/abs/2312.13211). Rahul Chand, Yashoteja Prabhu, Pratyush Kumar. [[Paper]](https://arxiv.org/abs/2312.13211)
* [![Publish](https://img.shields.io/badge/Conference-EACL'24-blue)]() [PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation](https://arxiv.org/abs/2401.11316). Nadav Benedek, Lior Wolf. [[Paper]](https://arxiv.org/abs/2401.11316)
* [![Publish](https://img.shields.io/badge/Conference-ICLR'24-blue)]() [The Need for Speed: Pruning Transformers with One Recipe](https://arxiv.org/abs/2403.17921). Samir Khaki, Konstantinos N. Plataniotis. [[Paper]](https://arxiv.org/abs/2403.17921)
* [NeuroPrune: A Neuro-inspired Topological Sparse Training Algorithm for Large Language Models](https://arxiv.org/abs/2404.01306). Amit Dhurandhar, Tejaswini Pedapati, Ronny Luss, Soham Dan, Aurelie Lozano, Payel Das, Georgios Kollias. [[Paper]](https://arxiv.org/abs/2404.01306)
* [Structural Pruning of Pre-trained Language Models via Neural Architecture Search](https://arxiv.org/abs/2405.02267). Aaron Klein, Jacek Golebiowski, Xingchen Ma, Valerio Perrone, Cedric Archambeau. [[Paper]](https://arxiv.org/abs/2405.02267)
* [![Publish](https://img.shields.io/badge/Conference-ICML'24-blue)]() [Switchable Decision: Dynamic Neural Generation Networks](https://arxiv.org/abs/2405.04513). Shujian Zhang, Korawat Tanwisuth, Chengyue Gong, Pengcheng He, Mingyuan Zhou. [[Paper]](https://arxiv.org/abs/2405.04513)
* [![Publish](https://img.shields.io/badge/Conference-CoLLAs'24-blue)]() [Automatic Pruning of Fine-tuning Datasets for Transformer-based Language Models](https://arxiv.org/abs/2407.08887). Mohammadreza Tayaranian, Seyyed Hasan Mozafari, Brett H. Meyer, James J. Clark, Warren J. Gross. [[Paper]](https://arxiv.org/abs/2407.08887)
* [Hybrid Dynamic Pruning: A Pathway to Efficient Transformer Inference](https://arxiv.org/abs/2407.12893). Ghadeer Jaradat, Mohammed Tolba, Ghada Alsuhli, Hani Saleh, Mahmoud Al-Qutayri, Thanos Stouraitis, Baker Mohammad. [[Paper]](https://arxiv.org/abs/2407.12893)

## Quantization
* [![Star](https://img.shields.io/github/stars/wimh966/outlier_suppression.svg?style=social&label=Star)](https://github.com/wimh966/outlier_suppression) [Outlier Suppression: Pushing the Limit of Low-bit Transformer](https://arxiv.org/abs/2209.13325). Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, Xianglong Liu. [[Paper]](https://arxiv.org/abs/2209.13325)[[Github]](https://github.com/wimh966/outlier_suppression)
* [![Publish](https://img.shields.io/badge/Conference-ACL'23-blue)]() [Self-Distilled Quantization: Achieving High Compression Rates in Transformer-Based Language Models](https://aclanthology.org/2023.acl-short.114/). James O’Neill, Sourav Dutta. [[Paper]](https://aclanthology.org/2023.acl-short.114/)
* [![Publish](https://img.shields.io/badge/Conference-ICML'23-blue)]() [Understanding Int4 Quantization for Language Models: Latency Speedup, Composability, and Failure Cases](https://openreview.net/forum?id=q1WGm3hItW). Xiaoxia Wu, Cheng Li, Reza Yazdani Aminabadi, Zhewei Yao, Yuxiong He. [[Paper]](https://openreview.net/forum?id=q1WGm3hItW)
* [![Publish](https://img.shields.io/badge/Conference-ACL'23%20Findings-blue)]() [PreQuant: A Task-agnostic Quantization Approach for Pre-trained Language Models](https://arxiv.org/abs/2306.00014). Zhuocheng Gong, Jiahao Liu, Qifan Wang, Yang Yang, Jingang Wang, Wei Wu, Yunsen Xian, Dongyan Zhao, Rui Yan. [[Paper]](https://arxiv.org/abs/2306.00014)
* [![Publish](https://img.shields.io/badge/Conference-ACL'23%20Findings-blue)]() [Boost Transformer-based Language Models with GPU-Friendly Sparsity and Quantization](https://aclanthology.org/2023.findings-acl.15.pdf). Chong Yu, Tao Chen, Zhongxue Gan. [[Paper]](https://aclanthology.org/2023.findings-acl.15.pdf)
* [![Publish](https://img.shields.io/badge/Conference-EMNLP'23-blue)]() [Zero-Shot Sharpness-Aware Quantization for Pre-trained Language Models](https://arxiv.org/abs/2310.13315). Miaoxi Zhu, Qihuang Zhong, Li Shen, Liang Ding, Juhua Liu, Bo Du, Dacheng Tao. [[Paper]](https://arxiv.org/abs/2310.13315)

## Inference Acceleration
* [![Publish](https://img.shields.io/badge/Conference-ICML'20-blue)]() [![Star](https://img.shields.io/github/stars/IBM/PoWER-BERT.svg?style=social&label=Star)](https://github.com/IBM/PoWER-BERT) [PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination](https://arxiv.org/abs/2001.08950). Saurabh Goyal, Anamitra R. Choudhury, Saurabh M. Raje, Venkatesan T. Chakaravarthy, Yogish Sabharwal, Ashish Verma. [[Paper]](https://arxiv.org/abs/2001.08950)[[Github]](https://github.com/IBM/PoWER-BERT)
* [![Publish](https://img.shields.io/badge/Conference-ACL'22%20Findings-blue)]() [![Star](https://img.shields.io/github/stars/txsun1997/HashEE.svg?style=social&label=Star)](https://github.com/txsun1997/HashEE) [A Simple Hash-Based Early Exiting Approach For Language Understanding and Generation](https://arxiv.org/abs/2203.01670). Tianxiang Sun, Xiangyang Liu, Wei Zhu, Zhichao Geng, Lingling Wu, Yilong He, Yuan Ni, Guotong Xie, Xuanjing Huang, Xipeng Qiu. [[Paper]](https://arxiv.org/abs/2203.01670)[[Github]](https://github.com/txsun1997/HashEE)
* [![Publish](https://img.shields.io/badge/Conference-KDD'22-blue)]() [![Star](https://img.shields.io/github/stars/kssteven418/LTP.svg?style=social&label=Star)](https://github.com/kssteven418/LTP) [Learned Token Pruning for Transformers](https://arxiv.org/abs/2107.00910). Sehoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, Kurt Keutzer. [[Paper]](https://arxiv.org/abs/2107.00910)[[Github]](https://github.com/kssteven418/LTP)
* [![Publish](https://img.shields.io/badge/Conference-NeurIPS'22%20Oral-blue)]() [Confident Adaptive Language Modeling](https://arxiv.org/abs/2207.07061). Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q. Tran, Yi Tay, Donald Metzler. [[Paper]](https://arxiv.org/abs/2207.07061)[[Github]](https://github.com/google-research/t5x/tree/main/t5x/contrib/calm)
* [![Publish](https://img.shields.io/badge/Conference-ICLR'23-blue)]() [Sparse Token Transformers with Attention Back Tracking](https://openreview.net/pdf?id=VV0hSE8AxCw). Heejun Lee, Minki Kang, Youngwan Lee, Sung Ju Hwang. [[Paper]](https://openreview.net/pdf?id=VV0hSE8AxCw)
* [![Publish](https://img.shields.io/badge/Conference-ACL'23-blue)]() [![Star](https://img.shields.io/github/stars/dropreg/DEER.svg?style=social&label=Star)](https://github.com/dropreg/DEER) [Dynamic and Efficient Inference for Text Generation via BERT Family](https://aclanthology.org/2023.acl-long.162/). Xiaobo Liang, Juntao Li, Lijun Wu, Ziqiang Cao, Min Zhang. [[Paper]](https://aclanthology.org/2023.acl-long.162/)[[Github]](https://github.com/dropreg/DEER)
* [![Publish](https://img.shields.io/badge/Conference-KDD'23-blue)]() [![Star](https://img.shields.io/github/stars/microsoft/Moonlit.svg?style=social&label=Star)](https://github.com/microsoft/Moonlit) [Constraint-aware and Ranking-distilled Token Pruning for Efficient Transformer Inference](https://arxiv.org/abs/2306.14393). Junyan Li, Li Lyna Zhang, Jiahang Xu, Yujing Wang, Shaoguang Yan, Yunqing Xia, Yuqing Yang, Ting Cao, Hao Sun, Weiwei Deng, Qi Zhang, Mao Yang. [[Paper]](https://arxiv.org/abs/2306.14393)[[Github]](https://github.com/microsoft/Moonlit)
* [![Publish](https://img.shields.io/badge/IEEE-TC'22-blue)]() [Dynamic Sparse Attention for Scalable Transformer Acceleration](https://ieeexplore.ieee.org/document/9896137). Liu Liu*, Zheng Qu*, Zhaodong Chen, Fengbin Tu, Yufei Ding, Yuan Xie.. [[Paper]](https://ieeexplore.ieee.org/document/9896137)

## Structure Design
* [![Star](https://img.shields.io/github/stars/pbelcak/UltraFastBERT.svg?style=social&label=Star)](https://github.com/pbelcak/UltraFastBERT) [Exponentially Faster Language Modelling](https://arxiv.org/abs/2311.10770). Peter Belcak, Roger Wattenhofer. [[Paper]](https://arxiv.org/abs/2311.10770)[[Github]](https://github.com/pbelcak/UltraFastBERT)

## Hardware
* [![Publish](https://img.shields.io/badge/Conference-ICMLA'23-blue)]() [Quantized Transformer Language Model Implementations on Edge Devices](https://arxiv.org/abs/2310.03971). Mohammad Wali Ur Rahman, Murad Mehrab Abrar, Hunter Gibbons Copening, Salim Hariri, Sicong Shao, Pratik Satam, Soheil Salehi. [[Paper]](https://arxiv.org/abs/2310.03971)

## Evaluation
* [![Publish](https://img.shields.io/badge/Conference-EMNLP'23-blue)]() [![Star](https://img.shields.io/github/stars/wearepal/compression-subgroup.svg?style=social&label=Star)](https://github.com/wearepal/compression-subgroup) [Are Compressed Language Models Less Subgroup Robust?](https://arxiv.org/abs/2403.17811). Leonidas Gee, Andrea Zugarini, Novi Quadrianto. [[Paper]](https://arxiv.org/abs/2403.17811)[[Github]](https://github.com/wearepal/compression-subgroup)

## Others
* [![Publish](https://img.shields.io/badge/Conference-ICML'23-blue)]() [![Star](https://img.shields.io/github/stars/weitianxin/mlp_fusion.svg?style=social&label=Star)](https://github.com/weitianxin/mlp_fusion) [NTK-approximating MLP Fusion for Efficient Language Model Fine-tuning](https://proceedings.mlr.press/v202/wei23b). Tianxin Wei, Zeming Guo, Yifan Chen, Jingrui He. [[Paper]](https://proceedings.mlr.press/v202/wei23b)[[Github]](https://github.com/weitianxin/mlp_fusion)
* [![Publish](https://img.shields.io/badge/Conference-NeurIPS'23-blue)]() [MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining](https://arxiv.org/abs/2312.17482). Jacob Portes, Alex Trott, Sam Havens, Daniel King, Abhinav Venigalla, Moin Nadeem, Nikhil Sardana, Daya Khudia, Jonathan Frankle. [[Paper]](https://arxiv.org/abs/2312.17482)[[Github]](https://github.com/mosaicml/examples/tree/main/examples/benchmarks/bert)[[Project]](https://mosaicbert.github.io/)
* [![Publish](https://img.shields.io/badge/Conference-ACL'23-blue)]() [PuMer: Pruning and Merging Tokens for Efficient Vision Language Models](https://arxiv.org/abs/2305.17530). Qingqing Cao, Bhargavi Paranjape, Hannaneh Hajishirzi. [[Paper]](https://arxiv.org/abs/2305.17530)
* [![Publish](https://img.shields.io/badge/Conference-CIKM%20short-blue)]() [Lightweight Adaptation of Neural Language Models via Subspace Embedding](https://arxiv.org/abs/2308.08688). Amit Kumar Jaiswal, Haiming Liu. [[Paper]](https://arxiv.org/abs/2308.08688)
* [Frustratingly Simple Memory Efficiency for Pre-trained Language Models via Dynamic Embedding Pruning](https://arxiv.org/abs/2309.08708). Miles Williams, Nikolaos Aletras. [[Paper]](https://arxiv.org/abs/2309.08708)
* [![Publish](https://img.shields.io/badge/Conference-EMNLP'23%20Findings-blue)]() [![Star](https://img.shields.io/github/stars/robertcsordas/moe.svg?style=social&label=Star)](https://github.com/robertcsordas/moe) [Approximating Two-Layer Feedforward Networks for Efficient Transformers](https://arxiv.org/abs/2310.10837). Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber. [[Paper]](https://arxiv.org/abs/2310.10837)[[Github]](https://github.com/robertcsordas/moe)
* [![Star](https://img.shields.io/github/stars/ltgoslo/elc-bert.svg?style=social&label=Star)](https://github.com/ltgoslo/elc-bert) [Not all layers are equally as important: Every Layer Counts BERT](https://arxiv.org/abs/2311.02265). Lucas Georges Gabriel Charpentier, David Samuel. [[Paper]](https://arxiv.org/abs/2311.02265)[[Github]](https://github.com/ltgoslo/elc-bert)