Awesome

Julia LLM Leaderboard

Comparison of Julia language generation capabilities of various Large Language Models

• Julia LLM Leaderboard
  • Introduction
  • Test Cases
  • Automated Evaluation Methodology
  • Results (Preview)
    • Paid APIs
    • Local Models
    • Overall Time to Run vs Score
    • Prompting Templates
  • Running Evaluation / Adding More Results
  • Debugging
  • Contributing Your Test Case
    • Anatomy of definition.toml
  • Contributing GenAI Conversations with/about Julia
  • Feedback and Improvements

Introduction

Welcome to the Julia Code Generation Benchmark Repository!

This project is designed for the Julia community to compare the code generation capabilities of various AI models. Unlike academic benchmarks, our focus is practicality and simplicity: "Generate code, run it, and see if it works(-ish)."

This repository aims to understand how different AI models and prompting strategies perform in generating syntactically correct Julia code to guide users in choosing the best model for their needs.

Itchy fingers? Jump to examples/ or just run your own benchmark with run_benchmark() (eg, examples/code_gen_benchmark.jl).

Test Cases

Test cases are defined in a definition.toml file, providing a standard structure for each test. If you want to contribute a test case, please follow the instructions in the Contributing Your Test Case section.

Automated Evaluation Methodology

Each model's and prompt's performance is evaluated based on several criteria:

1. Parsing: Does the generated code parse correctly in Julia?
2. Execution: Can the code execute without errors?
3. Unit Tests: Do the included unit tests pass?
4. Example Runs: Does the code run in a provided example scenario?

At the moment, all criteria are weighed equally and each test case can earn a maximum of 100 points. If a code passes all criteria, it gets 100/100 points. If it fails one criterion (eg, all unit tests), it gets 75/100 points. If it fails two criteria (eg, it runs but all examples and unit tests are broken), it gets 50 points, and so on.

Results (Preview)

To provide a glimpse of the repository's functionality, we have included example results for the first 14 test cases. Open the documentation for the full results and a deep dive on each test case.

[!WARNING]
These scores might change as we evolve the supporting functionality and add more models.

Remember that the benchmark is quite challenging for any model - a single extra space or parentheses and the score might become 0 (="unable to parse")!

Paid APIs

Claude 3.5 Sonnet is the highest-performing model. For the best value-for-money, look to Mistral Codestral, Claude 3 Haiku and, recently released, GPT 4o mini (60% cheaper than GPT3.5!!!).

Note: From mid-February 2024, "gpt-3.5-turbo" will point to the latest release, "gpt-3.5-turbo-0125" (deprecating the June release).

Same information, but as a bar chart:

In addition, we can consider the performance (score) versus the cost (measured in US cents):

Local Models

Locally-hosted models are generally not as good as the best paid APIs, but they are getting close! Note that the "mistral-small" is already available to be run locally and there will be many future finetunes!

[!NOTE] Big thank you to 01.ai and Jun Tian in particular for providing the compute for several parts of this benchmark!

The best trade-off performance vs size is the latest Meta llama3 8bn. Otherwise, the leading model is Mixtral-8x22bn.

Same information, but as a bar chart:

And with a separate bar for each prompt template:

[!NOTE] Qwen-1.5 models have been removed from the overviews as the underlying model on Ollama repository (and HF) is not correct and has very low performance.

[!NOTE] I have noticed that some evals in Ollama/llama.cpp now score slightly higher now than in Dec-23, so it's on a roadmap to re-run the above evals.

Overall Time to Run vs Score

Clearly, the paid APIs win (the latest release: GPT-3.5-Turbo-1106), but that's not the whole story.

Prompting Templates

We hope to be able to provide some guidance around prompting strategies, eg, when is it better to use a "JuliaExpert*" prompt template vs an "In Julia, answer XYZ" prompt.

Learnings so far:

• Never use the "AsIs" prompt (ie, raw task definition). ALWAYS add some context around the language, situation, etc.
• Even a simple "In Julia, answer XYZ" prompt can be quite effective. Note that the bigger prompts ("CoT" stands for Chain of Thought) might be confusing the smaller models, hence why this prompt is so effective on average.

Note: The XML-based templates are tested only for Claude 3 models (Haiku and Sonnet), that's why we removed them from the comparison.

Make your own analysis with examples/summarize_results.jl!

Running Evaluation / Adding More Results

1. Existing Evaluations: Check scripts/code_gen_benchmark.jl for the example of previous evaluations.
2. Run Your Evaluation: Choose your model and prompt, and run the test.
3. Save Results: Store both the conversation and the evaluation.
4. Open a PR: Include the part of the code snippet you changed in the PR comments. We generally require 1-2 independent verifications of your result.

Want to run some experiments and save the results? Check out examples/experiment_hyperparameter_scan.jl!

Debugging

Want to review some of the past benchmark runs? Check out examples/summarize_results.jl for overall statistics and examples/debugging_results.jl for reviewing the individual conversations/model responses.

Contributing Your Test Case

To contribute a test case:

1. Naming Convention: Create nested folders following the format code_generation/category/test_case_name/definition.toml.
2. Saving Results: Store the full conversation and the evaluation results in a path nested by a model name like code_generation/category/test case/model/evaluation__PROMPT__STRATEGY__TIMESTAMP.json and code_generation/category/test case/model/conversation__PROMPT__STRATEGY__TIMESTAMP.json

Anatomy of definition.toml

Required fields in definition.toml include:

• name: Corresponding to the file path.
• contributor: The creator of the test case (and their collaborators).
• criteria: The evaluation criteria (eg, parsing, execution, unit_tests, examples).
• prompt: The problem statement or task.
• version: The version of the test case. Starts at "1.0".
• examples: Example scenarios for testing, provided as a vector of executable statements using the function name (eg, my_function(1, 2)).
• unit_tests: Tests to validate the code, provided as a vector of @test X = Z statements.
• imports: Packages that are made available to the model (to avoid failures due to a failed dependency).
• reference_solution: A reference solution to the problem, provided as a string of Julia code (no code fences).

There are several optional fields:

• examples_setup: Code to run before each example eval, provided as a string of Julia code (no code fences). Used to setup any variables or functions needed for the examples.
• examples_teardown: Code to run after each example eval, provided as a string of Julia code (no code fences). Used to clean up any variables or functions needed for the examples.
• unit_tests_setup: Code to run before each unit test eval, provided as a string of Julia code (no code fences). Used to setup any variables or functions needed for the unit tests.
• unit_tests_teardown: Code to run after each unit test eval, provided as a string of Julia code (no code fences). Used to clean up any variables or functions needed for the unit tests.

The above fields can improve re-use of code across the examples/unit tests.

See an example in examples/create_definition.jl. You can validate your test case definitions with validate_definition().

Contributing GenAI Conversations with/about Julia

Please PR and add any relevant and MOSTLY CORRECT conversations with/in/about Julia in folder julia_conversations/.

The goal is to have a collection of conversations that are useful for finetuning Julia knowledge in smaller models.

Feedback and Improvements

We highly value community input. If you have suggestions or ideas for improvement, please open an issue. All contributions are welcome!