Awesome

This work presents Chain-of-Action-Thought (dubbed CoAT), which takes the description of the previous actions, the current screen, and more importantly the action thinking of what actions should be performed and the outcomes led by the chosen action. To enable an adaptive learning of CoAT process, we construct a benchmark Android-In-The-Zoo, which contains 18,643 screen-action pairs together with CoAT annotations.

📣 Update

• [2024-10-15] Evaluation code has been released!

• [2024-09-20] Our work has been accepted to EMNLP2024 Findings!

• [2024-07-16] We add the demo code for using CoAT on proprietary models (GPT4V, Gemini-Pro and Qwen-VL-Max)!

• [2024-03-31] We release the first version of our AiTZ dataset!

• [2024-03-05] We have our paper arxived, now you can acess it by clicking here !

Android-in-the-Zoo

The data in AiTZ has 18,643 screens together with 2500+ instructions, all annotated with CoAT-driven semantic labels. The sample format for each time step is

{
  "episode_id": "523638528775825151",
  "episode_length": 4,
  "step_id": 0,
  "coat_screen_desc":   "[observe]",
  "coat_action_think":  "[action think]",
  "coat_action_desc":   "[next action description]",
  "coat_action_result": "[action result]",
  ...
}

You can refer to data-example folder for a more specific example.

Download

Our dataset (GoogleDrive or BaiduNetdisk) contains both the screens (.png) and the annotations (.json), consuming about 2.6G device space.

Statistics

Chain-of-Action-Thought

Comparison with other context modeling methods

We validate the effectiveness of CoAT by conducting a preliminary experiment on 50 episodes randomly sampled from AITW dataset.

The compared baselines are Chain-of-Thought (CoT) and Chain-of-Actions (CoA).

where “hit” means format hit rate, and “acc” means action type prediction accuracy. (One can refer to Table 8 in our paper for more details.)

CoAT demo usage

Here we provide a demo code for anyone who wants to try the CoAT on GPT-4V, Qwen-VL-Max and Gemini-1.0-Pro-Vision.

Firstly, go to coat/config.yaml and add your own api-keys and urls.

Secondly, run the folloiwng code in commad line to generate sematic components of CoAT framework:

python run_coat.py --task "flow" --DEMO_MODE "COAT" --MODEL.NAME "openai/gemini/qwenvl" --num-threads 3

Then, you can obtain the action prediction results by

python run_coat.py --task "predict" --DEMO_MODE "COAT" --MODEL.NAME "openai/gemini/qwenvl" --num-threads 3

Citation

If you find our work helpful, please consider citing our paper.

@misc{zhang2024android,
      title={Android in the Zoo: Chain-of-Action-Thought for GUI Agents}, 
      author={Jiwen Zhang and Jihao Wu and Yihua Teng and Minghui Liao and Nuo Xu and Xiao Xiao and Zhongyu Wei and Duyu Tang},
      year={2024},
      eprint={2403.02713},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}