Awesome

tokenizer

C++ implementation of tokenizers, including tiktoken.