Awesome

ArguGPT: evaluating, understanding and identifying argumentative essays generated by GPT models

If you are confused by any of the following contents or you have any suggestion, please contact us at argugpt@163.com. 【如果有任何疑问以及建议，欢迎通过邮箱联系我们。】

In this repo, you will see 【内容如下】:

• A corpus of 4k argumentative essays written by 7 GPT models. 【我们用7个GPT模型生成了4k议论文。】
• Several classifiers for detecting GPT-generated argumentative essays (99% accuracy at doc-level; 93% at sentence-level). 【训练了相关模型检测议论文是否由GPT模型生成（文章检测达到99%的准确率，句子检测达到93%的准确率）。】

And we make following resources public 【我们公布了以下资源】:

• Models: We fine-tuned RoBERTa to detect machine-generated argumentative essays. We trained both doc-level and sentence-level classifiers. 【我们微调了RoBERTa以检测机器生成的议论文，我们在文档级别和句子级别上都进行了训练。】
• App-demo: We built an app in huggingface space which bases on the sentence-level detector. 【基于句子级别的检测模型，我们搭建了一个App，可以在网页中直接使用。】
• Paper: We released our research paper in arxiv. 【我们的论文预印在了arxiv中。】

Introduction to ArguGPT corpus 语料清单

We compiled a 8k human-machine (4k human v.s. 4k machine) comparison argumentative essays. We also collected a 1k dataset for the out-of-distribution (OOD) test experiment, in which the essays are either generated by ood models or prompted by ood writing tasks. Sub-corpora are listed in the following table. 【ArguGPT包含了8k议论文文章，包括4k人类议论文和4k机器生成议论文。我们也另外收集了1k议论文，作为OOD测试集以验证分类器泛化能力（其中的议论文使用的作文题或生成式模型与ArguGPT中的不同），语料清单如下。】

Note that four OOD models are: gpt-4, claude-instant, bloomz-7b, and flan-t5-11b. More detailed information about the ArguGPT corpus can be seen in our paper. 【有关ArguGPT的更多信息，请参考我们的论文。】

Data split and baseline 数据划分及基准模型

We first split the data into train/dev/test sets. The test split of TOEFL essays are as well evaluated by human participants in the Turing test. Then we established baselines by training detectors based on SVMs and RoBERTa. Moreover, we conducted ablation study to see the effect of reducing training data points. Finally, we evaluated two detectors on our own test set, namely GPTZero and RoBERTa trained by Guo et al. (2023). 【我们将数据集进行了划分。其中，图灵测试（人类测评）中使用的数据为测试集中的托福作文。在此数据集上，我们训练了SVM和RoBERTa模型，作为该数据集的基准模型。此外，我们在不同大小的训练集上进行了训练。】

Accuracy of human evaluators on the TOEFL split of test set is only 64.65%, far lagging behind ML detectors/classifiers. 【对测试集中的托福文章进行图灵测试，人类参与者的准确率仅有64.65%，远低于基于机器学习的检测器。】

Notes: We broke down the dataset of essays into sentences and paragraphs, trained the models from document-, paragraph-, and sentence-level, and evaluated these classifiers. 【我们把文章拆分为段落和句子，并且分别从文章、段落和句子的层次训练了模型，并进行了一些测试。】

Team members 团队介绍

We are a group of students who are interested in language, linguistics, and NLP as well. Led by Hai Hu, an assistant professor from English Department of Shanghai Jiao Tong University (SJTU), whose research interest is computational linguistics, we hope to contribute something interesting to CL and NLP commuities from the perspective of language leaners. 【大家好，我们是一群热爱语言、同时也对NLP技术感兴趣的本科/研究生，团队由交大外院的胡海老师指导。我们会从语言学习者的角度，进行一些有趣的实验与研究。如果有幸的话，希望我们的研究能够对计算语言学和NLP社群做出一些小小的贡献！】

Citation

Please cite our work as

@misc{liu2023argugpt,
      title={ArguGPT: evaluating, understanding and identifying argumentative essays generated by GPT models}, 
      author={Yikang Liu and Ziyin Zhang and Wanyang Zhang and Shisen Yue and Xiaojing Zhao and Xinyuan Cheng and Yiwen Zhang and Hai Hu},
      year={2023},
      eprint={2304.07666},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}