Awesome

rwkv_chatbot

This is an inference engine for RWKV 4.

I have ported the main inference engine to pypi. https://github.com/harrisonvanderbyl/rwkvstic

Please pip install rwkvstic

run python3 runOptimised.py for having a look at running or python3 ./multi.py

to run a discord bot or for a chat-gpt like react-based frontend, and a simplistic chatbot backend server

To load a model, just download it and have it in the root folder of this project.

When you run the program, you will be prompted on what file to use,

You will also be prompted for the backend to use, such as jax,pytorch, or tensorflow, and a few more in there

You can also use export-torchscript option to export a .pt file that can be loaded in as a prebuilt torchscript file.

You may also be promped specify the float mode to use: fp64, fp32, fp16, or bf16 (some may not work on your device, and inference speed varies)

When using cuda, you can optionally specify another dtype to use during vector operations.