Awesome
mnn-llm
示例工程
- cli: 使用命令行编译,android编译参考android_build.sh
- web: 使用命令行编译,运行时需要指定web资源
- android: 使用Android Studio打开编译;
- ios: 使用Xcode打开编译;🚀🚀🚀该示例代码100%由ChatGPT生成🚀🚀🚀
- python: 对mnn-llm的python封装
mnnllm
; - other: 新增文本embedding,向量查询,文本解析,记忆库与知识库能力🔥;
模型导出与下载
llm模型导出onnx
和mnn
模型请使用llm-export
构建
CI构建状态:
本地编译
# clone
git clone --recurse-submodules https://github.com/wangzhaode/mnn-llm.git
cd mnn-llm
# linux
./script/build.sh
# macos
./script/build.sh
# windows msvc
./script/build.ps1
# python wheel
./script/py_build.sh
# android
./script/android_build.sh
# android apk
./script/android_app_build.sh
# ios
./script/ios_build.sh
一些编译宏:
BUILD_FOR_ANDROID
: 编译到Android设备;LLM_SUPPORT_VISION
: 是否支持视觉处理能力;DUMP_PROFILE_INFO
: 每次对话后dump出性能数据到命令行中;
默认使用CPU
,如果使用其他后端或能力,可以在编译MNN时添加MNN
编译宏
- cuda:
-DMNN_CUDA=ON
- opencl:
-DMNN_OPENCL=ON
- metal:
-DMNN_METAL=ON
4. 执行
# linux/macos
./cli_demo ./Qwen2-1.5B-Instruct-MNN/config.json # cli demo
./web_demo ./Qwen2-1.5B-Instruct-MNN/config.json ../web # web ui demo
# windows
.\Debug\cli_demo.exe ./Qwen2-1.5B-Instruct-MNN/config.json
.\Debug\web_demo.exe ./Qwen2-1.5B-Instruct-MNN/config.json ../web
# android
adb push android_build/MNN/OFF/arm64-v8a/libMNN.so /data/local/tmp
adb push android_build/MNN/express/OFF/arm64-v8a/libMNN_Express.so /data/local/tmp
adb push android_build/libllm.so android_build/cli_demo /data/local/tmp
adb push Qwen2-1.5B-Instruct-MNN /data/local/tmp
adb shell "cd /data/local/tmp && export LD_LIBRARY_PATH=. && ./cli_demo ./Qwen2-1.5B-Instruct-MNN/config.json"
Reference
<details> <summary>reference</summary>- cpp-httplib
- chatgpt-web
- ChatViewDemo
- nlohmann/json
- Qwen-1.8B-Chat
- Qwen-7B-Chat
- Qwen-VL-Chat
- Qwen1.5-0.5B-Chat
- Qwen1.5-1.8B-Chat
- Qwen1.5-4B-Chat
- Qwen1.5-7B-Chat
- Qwen2-0.5B-Instruct
- Qwen2-1.5B-Instruct
- Qwen2-7B-Instruct
- Qwen2-VL-2B-Instruct
- Qwen2-VL-7B-Instruct
- Qwen2.5-0.5B-Instruct
- Qwen2.5-1.5B-Instruct
- Qwen2.5-3B-Instruct
- Qwen2.5-7B-Instruct
- Qwen2.5-Coder-1.5B-Instruct
- Qwen2.5-Coder-7B-Instruct
- Qwen2.5-Math-1.5B-Instruct
- Qwen2.5-Math-7B-Instruct
- chatglm-6b
- chatglm2-6b
- codegeex2-6b
- chatglm3-6b
- glm4-9b-chat
- Llama-2-7b-chat-ms
- Llama-3-8B-Instruct
- Llama-3.2-1B-Instruct
- Llama-3.2-3B-Instruct
- Baichuan2-7B-Chat
- internlm-chat-7b
- Yi-6B-Chat
- deepseek-llm-7b-chat
- TinyLlama-1.1B-Chat-v0.6
- phi-2
- bge-large-zh
- gte_sentence-embedding_multilingual-base