Home

Awesome

Examining LLMs' Uncertainty Expression Towards Questions Outside Parametric Knowledge

Setup

i.e. export PYTHONPATH="${PYTHONPATH}:/home/genglin2/LLM-hallucination"

datasets

Tasks: FalseQA, NEC, RefuNQ (each task has two modes: answerable and unanswerable)

data_path = f"/data/{task_name}/{task_name}_{mode}.json"  

Results are stored at experiment_outputs/Logits Figures are saved at experiment_outputs/figures

Setup for running the logit experiments on LLama-2

Quick Start

generate LLM responses on FalseQA / NEC / RefuNQ

Our work contains three tasks:

To run different models on these tasks, we have

python src/run_*.py --prompt baseline

Evaluation

To evaluate the outputs of the LLMs and visualize the analysis, see the notebooks in /scripts.

Prompts

The prompts used in this repo can be found in the prompts/ folder.

Citation

@misc{liu2024examining,
      title={Examining LLMs' Uncertainty Expression Towards Questions Outside Parametric Knowledge}, 
      author={Genglin Liu and Xingyao Wang and Lifan Yuan and Yangyi Chen and Hao Peng},
      year={2024},
      eprint={2311.09731},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}