Awesome
Symbol-LLM
This repo contains the official implementation of our paper:
Symbol-LLM: Leverage Language Models for Symbolic System in Visual Human Activity Reasoning (NeurIPS 2023)
Xiaoqian Wu, Yong-Lu Li*, Jianhua Sun, Cewu Lu*
[project page] [paper] [arxiv]
Generate the Proposed Symbolic System
Given an activity, the proposed symbolic system prompts a LLM to generate broad-coverage symbols and rational rules. It is implemented in generate_rule.py.
Visual Reasoning
With generated symbols and rules, we can use it to reason out activities in images. We detail the experiments on HICO, with zero-shot CLIP as baseline.
First, download the DATA folder from this link, with generated rules and symbol predictions. Then run hico_clip+reason.ipynb to get the result.
Alternatively, you can generate rules and predict symbols yourselves :)
- To generate rules, run generate_rule.py. Note that the rules may differ because the evolution of GPT API and the sampling uncertainty.
- To predict symbols, please refer to hico_predict_symbols.py, where BLIP2 is used.
Citation
If you find this work useful, please cite via:
@inproceedings{wu2023symbol,
title={Symbol-LLM: Leverage Language Models for Symbolic System in Visual
Human Activity Reasoning},
author={Wu, Xiaoqian and Li, Yong-Lu and Sun, Jianhua and Lu, Cewu},
booktitle={Thirty-seventh Conference on Neural Information Processing Systems},
year={2023}
}