Home

Awesome

🦙 llama-tokenizer-js 🦙

JavaScript tokenizer for LLaMA 1 and LLaMA 2 (I made a separate repo for LLaMA 3 here)

The tokenizer works client-side in the browser (and also in Node) (and now with TypeScript support)

Intended use case is calculating token count accurately on the client-side.

<a href="https://belladoreai.github.io/llama-tokenizer-js/example-demo/build/">Click here for demo</a>

Features

Import

Recommended way: Install as an npm package and import as ES6 module

npm install llama-tokenizer-js
import llamaTokenizer from 'llama-tokenizer-js'

console.log(llamaTokenizer.encode("Hello world!").length)

Alternative: Load as ES6 module with <script> tags in your HTML

<script type="module" src="https://belladoreai.github.io/llama-tokenizer-js/llama-tokenizer.js"></script>

Alternative: for CommonJS projects this should work:

async function main() {
    const llamaTokenizer=await import('llama-tokenizer-js')
    console.log(llamaTokenizer.default.encode("Hello world!"))
}

main();

Usage

Once you have the module imported, you can encode or decode with it. Training is not supported.

When used in browser, llama-tokenizer-js pollutes global namespace with llamaTokenizer.

Encode:

llamaTokenizer.encode("Hello world!")
> [1, 15043, 3186, 29991]

Decode:

llamaTokenizer.decode([1, 15043, 3186, 29991])
> 'Hello world!'

Note that special "beginning of sentence" token and preceding space are added by default when encoded (and correspondingly expected when decoding). These affect token count. There may be some use cases where you don't want to add these. You can pass additional boolean parameters in these use cases. For example, if you want to decode an individual token:

llamaTokenizer.decode([3186], false, false)
> 'Hello'

Tests

You can run tests with:

llamaTokenizer.runTests()

The test suite is small, but it covers different edge cases very well.

Note that tests can be run both in browser and in Node (this is necessary because some parts of the code work differently in different environments).

Comparison to alternatives

llama-tokenizer-js is the first JavaScript tokenizer for LLaMA which works client-side in the browser. You might be wondering, what other solutions are people using to count tokens in web applications?

Compatibility

The tokenizer used by LLaMA is a SentencePiece Byte-Pair Encoding tokenizer.

Note that this is a tokenizer for LLaMA models, and it's different than the tokenizers used by OpenAI models. If you need a tokenizer for OpenAI models, I recommend gpt-tokenizer.

What is this tokenizer compatible with? All LLaMA models which have been trained on top of checkpoints (model weights) released by Facebook in March 2023 ("LLaMA") and July of 2023 ("LLaMA2").

Examples of compatible models:

Incompatible LLaMA models are those which have been trained from scratch, not on top of the checkpoints released by Facebook. For example, OpenLLaMA models are incompatible.

When you see a new LLaMA model released, this tokenizer is mostly likely compatible with it without any modifications. If you are unsure, try it and see if the token ids are the same (compared to running the model with, for example, oobabooga webui). You can find great test input/output samples by searching for runTests inside llama-tokenizer.js.

If you want to modify this library to support a new LLaMA tokenizer (new as in trained from scratch, not using the same tokenizer as most LLaMA models do), you should be able to do so by swapping the vocabulary and merge data (the 2 long variables near the end of llama-tokenizer.js file). This repo has a Python script for your convenience.

You can pass custom vocab and merge data to the tokenizer by instantiating it like this:

import { LlamaTokenizer } from 'llama-tokenizer-js'
const tokenizer = new LlamaTokenizer(custom_vocab, custom_merge_data);

Repo maintenance

Release steps:

  1. node test-llama-tokenizer.js
  2. open test.html
  3. do you need to update this README?
  4. bump version number in root package.json
  5. push tokenizer changes to github
  6. npm publish --dry-run
  7. npm publish
  8. bump version number in example-demo/package.json
  9. cd example-demo && npm install && npm run build && live-server
  10. push example demo changes to github
  11. create release in github

Who did this

The example-demo (tokenizer playground) is a fork of gpt-tokenizer playground.

Llama-tokenizer-js is developed by belladore.ai with contributions from xenova, blaze2004, imoneoi and ConProgramming.