Awesome
<div align="center">
<h1 align="center">The default model repository of <a href="https://github.com/bentoml/openllm">openllm</a></h1>
</div>
This repo (on main branch) is already included by openllm by default.
If you want more up-to-date untested models, please add our nightly branch.
openllm repo add nightly https://github.com/bentoml/openllm-models@nightly
Supported Models
Table of Contents
Llama-3.1 <a id="llama3.1"></a>
| Model | Version | Huggingface Link |
|---|
| llama3.1 | 405b-instruct-awq-4bit-df2a | HF Link |
| llama3.1 | 70b-instruct-awq-4bit-2988 | HF Link |
| llama3.1 | 70b-instruct-fp16-ace8 | HF Link |
| llama3.1 | 8b-instruct-awq-4bit-fe8c | HF Link |
| llama3.1 | 8b-instruct-fp16-2f36 | HF Link |
Llama-3 <a id="llama3"></a>
| Model | Version | Huggingface Link |
|---|
| llama3 | 70b-instruct-awq-4bit-aebb | HF Link |
| llama3 | 70b-instruct-fp16-1315 | HF Link |
| llama3 | 8b-instruct-awq-4bit-3f34 | HF Link |
| llama3 | 8b-instruct-fp16-8f83 | HF Link |
Phi-3 <a id="phi3"></a>
| Model | Version | Huggingface Link |
|---|
| phi3 | 3.8b-instruct-fp16-166c | HF Link |
| phi3 | 3.8b-instruct-ggml-q4-76aa | HF Link |
Mistral <a id="mistral"></a>
| Model | Version | Huggingface Link |
|---|
| mistral | 24b-instruct-nemo-ec54 | HF Link |
| mistral | 7b-instruct-awq-4bit-01cd | HF Link |
| mistral | 7b-instruct-fp16-e1cd | HF Link |
Gemma-2 <a id="gemma2"></a>
| Model | Version | Huggingface Link |
|---|
| gemma2 | 27b-instruct-fp16-6b83 | HF Link |
| gemma2 | 9b-instruct-fp16-6e86 | HF Link |
Qwen-2 <a id="qwen2"></a>
| Model | Version | Huggingface Link |
|---|
| qwen2 | 0.5b-instruct-fp16-33df | HF Link |
| qwen2 | 1.5b-instruct-fp16-7cda | HF Link |
| qwen2 | 57b-a14b-instruct-fp16-365f | HF Link |
| qwen2 | 72b-instruct-awq-4bit-33fa | HF Link |
| qwen2 | 72b-instruct-fp16-8cb4 | HF Link |
| qwen2 | 7b-instruct-awq-4bit-14aa | HF Link |
| qwen2 | 7b-instruct-fp16-bbf2 | HF Link |
Gemma <a id="gemma"></a>
| Model | Version | Huggingface Link |
|---|
| gemma | 2b-instruct-fp16-6ee7 | HF Link |
| gemma | 7b-instruct-awq-4bit-df0b | HF Link |
| gemma | 7b-instruct-fp16-2297 | HF Link |
Llama-2 <a id="llama2"></a>
| Model | Version | Huggingface Link |
|---|
| llama2 | 13b-chat-fp16-ef61 | HF Link |
| llama2 | 70b-chat-fp16-16a0 | HF Link |
| llama2 | 7b-chat-awq-4bit-4f93 | HF Link |
| llama2 | 7b-chat-fp16-21b9 | HF Link |
Mixtral <a id="mixtral"></a>
| Model | Version | Huggingface Link |
|---|
| mixtral | 8x7b-instruct-v0.1-awq-4bit-06fd | HF Link |
| mixtral | 8x7b-instruct-v0.1-fp16-e289 | HF Link |
Mistral-Large <a id="mistral-large"></a>
| Model | Version | Huggingface Link |
|---|
| mistral-large | 123b-instruct-awq-4bit-1d37 | HF Link |
| mistral-large | 123b-instruct-fp16-5c96 | HF Link |
Codestral <a id="codestral"></a>
| Model | Version | Huggingface Link |
|---|
| codestral | 22b-v0.1-fp16-b677 | HF Link |