Home

Awesome

AgentTuning: Enabling Generalized Agent Abilities For LLMs

<p align="center"> 🤗 <a href="https://huggingface.co/THUDM/agentlm-70b" target="_blank">Model (AgentLM-70B)</a> • 🤗 <a href="https://huggingface.co/datasets/THUDM/AgentInstruct" target="_blank">Dataset (AgentInstruct)</a> • 📃 <a href="https://arxiv.org/abs/2310.12823" target="_blank">Paper</a> • 🌐 <a href="https://thudm.github.io/AgentTuning/" target="_blank">Project Page</a> <br> </p> <center><img src="assets/main-figure.svg" alt="main-figure" style="zoom:50%;" /></center>

中文版(Chinese)

AgentTuning represents the very first attempt to instruction-tune LLMs using interaction trajectories across multiple agent tasks. Evaluation results indicate that AgentTuning enables the agent capabilities of LLMs with robust generalization on unseen agent tasks while remaining strong in general language abilities. We have open-sourced the AgentInstruct dataset and AgentLM.

Main Result

<center><img src="assets/head-figure.svg" alt="head-figure" width="1500" /></center> <center><b>Figure 1</b>&nbsp;&nbsp;Overall score in our held-in and held-out tasks</center>

AgentInstruct

AgentInstruct is a meticulously curated dataset featuring 1,866 high-quality interactions designed to enhance AI agents across 6 diverse real-world tasks.

AgentInstruct dataset is available on 🤗Huggingface Repo.

AgentLM

AgentLM models are produced by mixed training on AgentInstruct dataset and ShareGPT dataset from Llama2-chat series.

The models follow the conversation format of Llama-2-chat, with the system prompt fixed as You are a helpful, respectful and honest assistant.

The 7B, 13B, and 70B models are available on Huggingface model hub.

ModelHuggingface Repo
AgentLM-7B🤗Huggingface Repo
AgentLM-13B🤗Huggingface Repo
AgentLM-70B🤗Huggingface Repo

Run AgentLM

We use Text-Generation-Inference to accelerate the evaluation process.

You can start a AgentLM-70b instance with:

cd docker
docker compose -f agentlm-70b.yml up

Upon successful execution, a client will be available on port 30070. Here is an example of launching a request:

curl 127.0.0.1:30070/generate \
    -X POST \
    -H 'Content-Type: application/json' \
    -d '{"inputs": "[INST] <<SYS>>\nYou are a helpful, respectful and honest assistant.\n<</SYS>>\n\nHello! [/INST]", "parameters":{"temperature": 1.0}}'

# {"generated_text":"Hello! How can I help you today? "}

You can replicate the services in the Docker Compose file to create multiple inference instances if more GPUs are available.

Evaluation

Here are details of our evaluation task, including 6 held-in tasks and 6 held-out tasks.

Held-in Tasks

The 6 held-in tasks are selected from AgentBench. However, since AgentBench is still under active development, the results from the latest branch might not fully reproduce the results reported in the paper. The evaluation code of this project is located in ./AgentBench.old.

Held-out Tasks

Held-out tasks are recompiled from the following frameworks:

TaskAgentTuning SetupOriginal Repo
SciWorld📂 eval_heldout/science-world💻 allenai/ScienceWorld
MiniWoB++📂 eval_heldout/miniwob++💻 Farama-Foundation/miniwob-plusplus
HotpotQA📂 eval_heldout/hotpotQA💻 salesforce/BOLAA
ReWOO📂 eval_heldout/rewoo💻 billxbf/ReWOO
WebArena📂 eval_heldout/webarena💻 web-arena-x/webarena
Digital Card Game💻 AgentBench.old ( Extend Split )💻 THUDM/AgentBench

General Tasks

MMLU Setup:

GSM8k Setup:

MT-Bench Setup:

Citation

If you find our work useful, please consider citing AgentTuning:

@misc{zeng2023agenttuning,
      title={AgentTuning: Enabling Generalized Agent Abilities for LLMs},
      author={Aohan Zeng and Mingdao Liu and Rui Lu and Bowen Wang and Xiao Liu and Yuxiao Dong and Jie Tang},
      year={2023},
      eprint={2310.12823},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}