# Project for Efficient LLM

## Tools
* [![Star](https://img.shields.io/github/stars/vllm-project/vllm.svg?style=social&label=Star)](https://github.com/vllm-project/vllm) **vllm**: A high-throughput and memory-efficient inference and serving engine for LLMs. [[link]](https://github.com/vllm-project/vllm)[[paper]](https://arxiv.org/abs/2309.06180)
* [![Star](https://img.shields.io/github/stars/TimDettmers/bitsandbytes.svg?style=social&label=Star)](https://github.com/TimDettmers/bitsandbytes)  **bitsandbytes**: 8-bit CUDA functions for PyTorch. [[link]](https://github.com/TimDettmers/bitsandbytes)
* [![Star](https://img.shields.io/github/stars/qwopqwop200/GPTQ-for-LLaMa.svg?style=social&label=Star)](https://github.com/qwopqwop200/GPTQ-for-LLaMa) **GPTQ-for-LLaMa**: 4 bits quantization of LLaMA using GPTQ. [[link]](https://github.com/qwopqwop200/GPTQ-for-LLaMa)
* [![Star](https://img.shields.io/github/stars/mit-han-lab/TinyChatEngine.svg?style=social&label=Star)](https://github.com/mit-han-lab/TinyChatEngine) **TinyChatEngine**: TinyChatEngine: On-Device LLM Inference Library. [[link]](https://github.com/mit-han-lab/TinyChatEngine)
* [![Star](https://img.shields.io/github/stars/microsoft/LMOps.svg?style=social&label=Star)](https://github.com/microsoft/LMOps) **LMOps**: General technology for enabling AI capabilities w/ LLMs and MLLMs. [[link]](https://github.com/microsoft/LMOps)
* [![Star](https://img.shields.io/github/stars/Lightning-AI/lit-gpt.svg?style=social&label=Star)](https://github.com/Lightning-AI/lit-gpt) **lit-gpt**: Hackable implementation of state-of-the-art open-source LLMs based on nanoGPT. Supports flash attention, 4-bit and 8-bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.. [[link]](https://github.com/Lightning-AI/lit-gpt)
* [![Star](https://img.shields.io/github/stars/ztxz16/fastllm.svg?style=social&label=Star)](https://github.com/ztxz16/fastllm) **fastllm**: 纯c++的全平台llm加速库，支持python调用，chatglm-6B级模型单卡可达10000+token / s，支持glm, llama, moss基座，手机端流畅运行. [[link]](https://github.com/ztxz16/fastllm)
* [![Star](https://img.shields.io/github/stars/kuleshov-group/llmtools.svg?style=social&label=Star)](https://github.com/kuleshov-group/llmtools) **llmtools**: 4-Bit Finetuning of Large Language Models on One Consumer GPU. [[link]](https://github.com/kuleshov-group/llmtools)
* [![Star](https://img.shields.io/github/stars/yoshitomo-matsubara/torchdistill.svg?style=social&label=Star)](https://github.com/yoshitomo-matsubara/torchdistill/) **torchdistill**: A coding-free framework built on PyTorch for reproducible deep learning studies. 🏆20 knowledge distillation methods presented at CVPR, ICLR, ECCV, NeurIPS, ICCV, etc are implemented so far. 🎁 Trained models, training logs and configurations are available for ensuring the reproducibiliy and benchmark.. [[link]](https://github.com/yoshitomo-matsubara/torchdistill/)[[paper]](https://arxiv.org/pdf/2310.17644.pdf)
* [![Star](https://img.shields.io/github/stars/nomic-ai/gpt4all.svg?style=social&label=Star)](https://github.com/nomic-ai/gpt4all) **gpt4all**: open-source LLM chatbots that you can run anywhere. [[link]](https://github.com/nomic-ai/gpt4all)[[paper]](https://arxiv.org/abs/2311.04931)
* [![Star](https://img.shields.io/github/stars/GreenBitAI/low_bit_llama.svg?style=social&label=Star)](https://github.com/GreenBitAI/low_bit_llama) **low_bit_llama**: Advanced Ultra-Low Bitrate Compression Techniques for the LLaMA Family of LLMs. [[link]](https://github.com/GreenBitAI/low_bit_llama)
* [![Star](https://img.shields.io/github/stars/turboderp/exllama.svg?style=social&label=Star)](https://github.com/turboderp/exllama) **exllama**: A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.. [[link]](https://github.com/turboderp/exllama)

## Open-source Lightweight LLM
* [![Star](https://img.shields.io/github/stars/jzhang38/TinyLlama.svg?style=social&label=Star)](https://github.com/jzhang38/TinyLlama) **TinyLlama**: The TinyLlama project is an open endeavor to pretrain a 1.1B Llama model on 3 trillion tokens.. [[link]](https://github.com/jzhang38/TinyLlama)
