Home

Awesome

AutoDroid

About

This repository contains the code for the system for the paper: Empowering LLM to use Smartphone for Intelligent Task Automation.

For accessing the dataset DroidTask, you could download it from Google Cloud, and you could refer to the About Dataset Section.

AutoDroid is implemented based on the DroidBot framework.

How to install

Make sure you have:

  1. Python
  2. Java
  3. Android SDK
  4. Added platform_tools directory in Android SDK to PATH

Then clone this repo and install with pip:

git clone git@github.com:MobileLLM/AutoDroid.git
cd AutoDroid/
pip install -e .

How to use

  1. Prepare:

    • If you want to test AutoDroid with the apps used in our paper, please download the apk.zip folder from Google Cloud, and unzip it, and prepare a device or an emulator connected to your host machine via adb.
    • If you want to test AutoDroid with other apps, please download the .apk file to your host machine, and prepare a device or an emulator connected to your host machine via adb.
    • Prepare a GPT API key, and go to tools.py, replace the os.environ['GPT_URL'] with your own API key.
  2. Start:

    droidbot -a <path/to/.apk> -o <output/of/app> -task <your task> -keep_env -keep_app
    

    you can try the scripts in the ./scripts folder, and the tasks from the DroidTask are listed in the form.

About Dataset

Organization of the Dataset,

    DroidTask
    ├── applauncher
    │   ├── states
    │   │   ├── Screenshot 1.png
    │   │   ├── Screenshot 2.png
    │   │   ├── ...
    │   │   ├── View hierarchy 1.json
    │   │   ├── View hierarchy 2.json
    │   │   └── ...
    │   ├── task1.yaml
    │   ├── task2.yaml
    │   ├── ...
    │   └── utg.yaml
    ├── calendar
    │   ├── states
    │   │   ├── Screenshot 1.png
    │   │   ├── Screenshot 2.png
    │   │   ├── ...
    │   │   ├── View hierarchy 1.json
    │   │   ├── View hierarchy 2.json
    │   │   └── ...
    │   ├── task1.yaml
    │   ├── task2.yaml
    │   ├── ...
    │   └── utg.yaml

Known limitations

Welcome to contribute!

Note

Enjoy!