Home

Awesome

This is a native port of the new RWKV tokenizer.

Usage

  1. Install Rust.
  2. Run python tokenizer_fast.py.

License

Licensed under either of

at your option.