Home

Awesome

llama2.jl

<p align="center"> <img src="assets/jl_cute_lama.png" width="300" height="300" alt="Cute Llama"> </p>

Tired of low-level languages? Ever wanted to infer a baby Llama 2 model in pure Julia? Great news – you can now do so at in under 300 lines of Julia.

This is a fork of Andrej's llama2.c which has been ported to (for now) a slightly hacky version of Julia. This README is heavily inspired by the Rust port llama.rs.

Don't want to read? Got ya back!

git clone https://github.com/juvi21/llama2.jl && cd llama2.jl && wget https://huggingface.co/karpathy/tinyllamas/resolve/main/stories15M.bin && julia jl_helpers/install_pkg.jl && julia run.jl stories15M.bin tokenizer.bin

How to run?

  1. Grab Andrej's baby Llama2 (see the original instructions) pretrained on the TinyStories dataset:

    wget https://huggingface.co/karpathy/tinyllamas/resolve/main/stories15M.bin
wget https://huggingface.co/karpathy/tinyllamas/resolve/main/stories42M.bin
wget https://huggingface.co/karpathy/tinyllamas/resolve/main/stories110M.bin

  2. Ensure you have the tokenizer binary - tokenizer.bin (if not, see tokenizer.py).

  3. Run run.jl:

    Single-threaded:

    julia run.jl <model> <tokenizer> --temp [temperature]

    Multi-Threaded: In Progress
    CUDA: In Progress

Performance

On my current workstation, the performance is quite fast. However, I have been away visiting my parents for a few days, so I only had the opportunity to test it on one of my very first and less powerful station. More testing is coming soon! NOTE: I compiled llama2.c with the provided command in Andrej's README which is only the basic one to get started and not very optimized.

gcc -O3 -o run run.c -lm
systemmodelllama2.cllmaa2.c -0fastllama2.jl
Ubuntu 22.04 AMD Ryzen 2600stories15M.bin85.418752 tok/s189.591078 tok/s257.445516 tok/s
Ubuntu 22.04 AMD Ryzen 2600stories42M.bin30.761836 tok/s78.485688 tok/s92.567484 tok/s
Ubuntu 22.04 AMD Ryzen 2600stories110.bin11.585283 tok/s30.375223 tok/s38.543434 tok/s

Contributions

Join the dark side and code in Julia. Contributions are highly encouraged!

Contribution Ideas:

Art

@Midjourney