Awesome
DIAC2019-DQD-Based-on-Adversarial-Attack
5st place solution for competition Duplication Question Detection based on Adversarial Attack
Introduction
Although the QA system has made great progress in recent years, how to accurately determine whether the user's input is a semantic equivalent of a given question is still the key of a QA system (such as law, government affairs, etc.). For example, "What departments does the municipal government govern?" and "Which departments are under the jurisdiction of the municipal government?" can be considered as semantically equivalent issues, while "what departments does the municipal government govern?" and "Which departments do the mayor govern?" is different questions. For Duplication Question Detection, in addition to the accuracy of the system, the robustness of the system is also important, but it is often neglected. For example, although a deep neural network model can often achieve satisfactory accuracy on a given train set and test set, a slight change to the test set (Adversarial Attack) may cause a large decrease in overall accuracy.
Analysis
Data Processing:Use the rule below to train.
A: qi is from <EquivalentSentence>, qj is from <EquivalentSentence>, label 1
B: qi is from <EquivalentSentence>, qj is from <NotEquivalentSentence> label 0
Data Augmentation :For each sentence, replace it with synonym or character with similar form randomly to get two new sentences.
For example:
original sentence:民事诉讼什么情况下不能立案
replaced with synonym:官事诉讼什么情况下不能立案
replaced with characters with similar form:民事诉沦什么情况下不能立案
Model:bert+fgm (roberta)
Performance on B Leaderboard
fold:2 performance: 5st yansixiliang 0.79442 1
Environment
python3.7
pip install -i http://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com boto3==1.9.229 regex==2019.8.19 pip install torch==1.3.1+cu92 torchvision==0.4.2+cu92 -f https://download.pytorch.org/whl/torch_stable.html
pyltp
install pyltp
pip install pyltp
download model:ltp_data_v3.4.0
RUN
Data Processing
in dataset
install pyltp, then copy pyltp-master to pyltp-master
dowload ltp_data_v3.4.0, then copy it to pyltp-master
put table character with similar form to pyltp-master/ltp_data_v3.4.0
create data_gitignore to keep data
USE train_origin_data.py TO GET train_origin.csv, KEEP IT IN data_gitignore
USE train_augment_data.py TO GET train_augment.csv, KEEP IT IN data_gitignore
Merge train_origin.csv, train_augment.csv TO GET train_final.csv
USE Kfold_dataset.py TO GET 5 fold data, KEEP IT IN data_gitignore\data_StratifiedKFold_42
Model Trainning
RUN 'model/model.sh' TO train model.
RUN 'model/generate_submission.py' TO get submit file.
Project File Structure
.
├─ 1120_roberta_wwm_large_150_last2embedding_cls
├─ README.md
├─ dataset
│ ├─ Kfold_dataset.py
│ ├─ data_gitignore
│ │ ├─ data_StratifiedKFold_42
│ │ ├─ train_augment.csv
│ │ ├─ train_final.csv
│ │ └─ train_origin.csv
│ ├─ data_origin
│ │ ├─ baifendian_data.zip
│ │ ├─ dev_set.csv
│ │ ├─ sample_submission.csv
│ │ ├─ test_set.csv
│ │ └─ train_set.xml
│ ├─ pyltp-master
│ │ ├─ ..
│ │ ├─ ltp_data_v3.4.0
│ │ │ ├─newShape(table of character with similar form)
│ │ └─ src
│ ├─ train_augment_data.py
│ └─ train_origin_data.py
└─ model
├─ generate_submission.py
└─ model.sh
Data Processing
'dataset/train_origin_data.py' to get original sentence pair 'dataset/train_augment_data.py' to get adversarial sentence pair 'dataset/Kfold_dataset.py' to get cross-validation data set
Model Training
'src/FGM.py' FGM model Class
'src/pytorch_transformers/modeling_bert.py' use fgm in bert to add adversarial
'src/run_bert_base_fgm.py' model train
Future Work
- add fold
- merge with other models
- use synonym extracting algorithm to get synonyms in Law Field
References
How to generate wrong word? (https://github.com/contr4l/SimilarCharacter)
How to generate orginal sentence pair? (https://www.biendata.com/forum/view_post_category/718/)
Some briliant ideas (https://github.com/WenRichard/DIAC2019-Adversarial-Attack-Share)