# 4.3 其他低秩分解技术

## 4.3.1 delta-tuning

### 4.3.1.1 delta-tuning 介绍

### 4.3.1.2 delta-tuning 的数学解释

### 4.3.1.3 delta-tuning 代码实战

### 4.3.1.4 模型评估



## 4.3.2 DoRA: Weight-Decomposed Low-Rank Adaptation

### 4.3.2.1 DoRA 介绍

### 4.3.2.2 DoRA 的数学解释

### 4.3.2.3 DoRA 代码实战

### 4.3.2.4 模型评估



## 4.3.3 Matrix-Transformation Based Low-Rank Adaptation (MTLoRA)

### 4.3.3.1 MTLoRA 介绍

### 4.3.3.2 MTLoRA 的数学解释

### 4.3.3.3 MTLoRA 代码实战

### 4.3.3.4 模型评估



## 4.3.4 AFLoRA: Adaptive Freezing of Low Rank Adaptation

### 4.3.4.1 AFLoRA 介绍

### 4.3.4.2 AFLoRA 的数学解释

### 4.3.4.3 AFLoRA 代码实战

### 4.3.4.4 模型评估



## 4.3.5 the Mixture of LoRA Experts (MoLE) approach

### 4.3.5.1 MoLE 介绍

### 4.3.5.2 MoLE 的数学解释

### 4.3.5.3 MoLE 代码实战

### 4.3.5.4 模型评估



## 参考文献

Lialin, Vladislav, Vijeta Deshpande, and Anna Rumshisky. "Scaling down to scale up: A guide to parameter-efficient fine-tuning." *arXiv preprint arXiv:2303.15647* (2023).

Ding, Ning, et al. "Parameter-efficient fine-tuning of large-scale pre-trained language models." *Nature Machine Intelligence* 5.3 (2023): 220-235.

Liu, Shih-Yang, et al. "Dora: Weight-decomposed low-rank adaptation." *arXiv preprint arXiv:2402.09353* (2024).

Liang, Yao, Yuwei Wang, and Yi Zeng. "Matrix-transformation based low-rank adaptation (mtlora): A brain-inspired method for parameter-efficient fine-tuning." *arXiv preprint arXiv:2403.07440* (2024).

Liu, Zeyu, et al. "AFLoRA: Adaptive Freezing of Low Rank Adaptation in Parameter Efficient Fine-Tuning of Large Models." *arXiv preprint arXiv:2403.13269* (2024).

Wu, Xun, Shaohan Huang, and Furu Wei. "Mixture of lora experts." *arXiv preprint arXiv:2404.13628* (2024).