Home

Awesome

Interactive Composition Explorer 🧊

ICE is a Python library and trace visualizer for language model programs.

Screenshot

<p align="center"> <img alt="ice-screenshot" src="https://user-images.githubusercontent.com/382515/192681645-6ed87072-2dc6-4982-92d1-8de209bc3ef6.png" /> Execution trace visualized in ICE </p>

Features

ICE is pre-1.0

:warning: The ICE API may change at any point. The ICE interface is being actively developed and we may change the API at any point, including removing functionality, renaming methods, splitting ICE into multiple projects, and other similarly disruptive changes. Use at your own risk.

Requirements

ICE requires Python 3.9, 3.10, or 3.11. If you don't have a supported version of Python installed, we recommend using pyenv to install a supported Python version and manage multiple Python versions.

If you use Windows, you'll need to run ICE inside of WSL.

Getting started

  1. As part of general good Python practice, consider first creating and activating a virtual environment to avoid installing ICE 'globally'. For example:

    python -m venv venv
    source venv/bin/activate
    
  2. Install ICE:

    pip install ought-ice
    
  3. Run the Hello World recipe in the Primer to see the trace rendered.

  4. Optionally, set secrets (like your OpenAI API key) in ~/.ought-ice/.env. See .env.example for the format. If these are not set, you'll be prompted for them when you run recipes that need them.

Developing ICE

  1. If you want to make changes to ICE itself, clone the repository, then install it in editable mode:

    python -m venv venv
    source venv/bin/activate
    pip install --upgrade pip
    pip install -e '.[dev]' --config-settings editable_mode=compat
    pre-commit install
    npm --prefix ui ci
    npm --prefix ui run dev
    
  2. If you're working on the backend, you might find it helpful to remove the cache of language model calls:

    rm -r ~/.ought-ice/cache
    
  3. pre-commit complains if your code doesn't pass certain checks. It runs when you commit, and will possibly reject your commit and make you have to fix the problem(s) before you can commit again. (So you should probably use the same commit message you used the first time.)

Note that you don't technically need to run pre-commit install, but not doing so may cause your commits to fail CI. (Which can be noisy, including by generating commits that will e.g. fix formatting.)

Storybook

We use Storybook for UI tests. You can run them locally:

npm --prefix ui run storybook

Note that build-storybook is only for CI and shouldn't be run locally.

Terminology

Additional resources

  1. Join the ICE Slack channel to collaborate with other people composing language model tasks. You can also use it to ask questions about using ICE.

  2. Watch the recording of Ought's Lab Meeting to understand the high-level goals for ICE, how it interacts with Ought's other work, and how it contributes to alignment research.

  3. Read the ICE announcement post for another introduction.

Contributions

ICE is an open-source project by Ought. We're an applied ML lab building the AI research assistant Elicit.

We welcome community contributions:

For larger contributions, make an issue for discussion before submitting a PR.

And for even larger contributions, join us - we're hiring!

How to cite

If you use ICE, please cite:

Iterated Decomposition: Improving Science Q&A by Supervising Reasoning Processes. Justin Reppert, Ben Rachbach, Charlie George, Luke Stebbing Jungwon Byun, Maggie Appleton, Andreas StuhlmĂĽller (2023). Ought Technical Report. arXiv:2301.01751 [cs.CL]

Bibtex:

@article{reppert2023iterated,
  author = {Justin Reppert and Ben Rachbach and Charlie George and Luke Stebbing and Jungwon Byun and Maggie Appleton and Andreas Stuhlm\"{u}ller},
  archivePrefix = {arXiv},
  eprint = {2301.01751},
  primaryClass = {cs.CL},
  title = {Iterated Decomposition: Improving Science Q&A by Supervising Reasoning Processes},
  year = 2023,
  keywords = {language models, decomposition, workflow, debugging},
  url = {https://arxiv.org/abs/2301.01751}
}