Awesome

Social Chemistry 101

Project

For an overview of the Social Chemistry project, a live demo of the model, and an interactive dataset browser, check out our project webpage.

Paper

This repository is for code accompanying the paper:

Social Chemistry 101: Learning to Reason about Social and Moral Norms <br/> Maxwell Forbes, Jena D. Hwang, Vered Shwartz, Maarten Sap, Yejin Choi <br/> EMNLP 2020

Data

Download the Social-Chem-101 dataset here.

The dataset schema is given in detail below. See the README in the dataset, as well as the appendix of the paper, for substantially more information about the dataset and its collection.

The dataset is licensed under the CC BY-SA 4.0 license.

Pretrained Models

We provide two pretrained Neural Norm Transformers using the GPT-2 architecture: one for rules-of-thumb (RoTs), and one for actions.

Here are some example commands to download and extract the RoT model.

# Start from repo root. Model checkpoints are conventionally saved in "output/", though
# the downloaded models will also have an "output/" directory, so we'll pull the files
# of them.
mkdir output/
cd output/
wget https://storage.googleapis.com/ai2-mosaic-public/projects/social-chemistry/models/gpt2-xl_rot_64_5epochs.tar.gz
tar -xzf gpt2-xl_rot_64_5epochs.tar.gz
cd output/
mv gpt2-xl_rot_64_5epochs/ ..
cd ..
rmdir output/
rm gpt2-xl_rot_64_5epochs.tar.gz
cd ..

See below for examples of generating using the models.

Code

Installation

# 1. Setup + activate a fresh python3.7+ virtual environment with your method of choice.
#    (We used pyenv and 3.8.3.) Your particular steps will vary.

# 2. Install pytorch, with CUDA if possible. (We tested with pytorch 1.5.1 and 1.7.0
#    using CUDA 10.1.) Follow https://pytorch.org/get-started/locally/

# 3. Install python dependencies
pip install -r requirements.txt

# 4. Download and extract the dataset .tsv file into `data/dataset/`. Here are some
#    example commands for Linux.
mkdir -p data/dataset
cd data/dataset
wget https://storage.googleapis.com/ai2-mosaic-public/projects/social-chemistry/data/social-chem-101.zip
unzip social-chem-101.zip
mv social-chem-101/* .
rmdir social-chem-101
rm social-chem-101.zip
rm -rf __MACOSX  # extra cruft folder
cd ../..

# NOTE: There will also now be a dataset readme .md file in that folder. It contains
#       detailed  information about the dataset schema, collection, splitting, and more.

Training

The Python code is in the sc/ directory. An example script is given in scripts/train_generative_models.sh, which illustrates how to train models.

By default, the following output locations are used:

• output/ (specified in example script, not code) is for saved model / vocab
• data/dataset/ is where cached tokenized/tensorized dataset versions live
• runs/ is where tensorboard-renderable event logs live

... and as such, all are ignored by version control (.gitignore).

Example: Generation

Before you generate, you need pretrained models. You can train the models yourself (previous section), or you can download models we have trained (see above).

We've provided some example scripts for generating. Let's assume you are interested in generating RoTs, and you downloaded the pretrained GPT2-XL RoT model, and it lives at output/gpt2-xl_rot_64_5epochs/. Then you can run:

# See all options for specifying GPU and sampling procedure.
python -m sc.examples.generate --help

# Provide model and use default settings (GPT 0, top-p = 0.9).
python -m sc.examples.generate --model output/gpt2-xl_rot_64_5epochs

This dynamically loads RoT examples from sc/examples/examples.py. You can edit that file to change the generation prompts. It loads from a file (rather than, say, having you type in a situation or attributes directly) to make it easy to experiment with varying an input attribute and seeing how the generations change.

If you'd like to use actions and action attributes (instead of RoTs and RoT attributes), simply specify an action model instead. The example code checks whether "rot" or "action" is in the model's path and loads examples accordingly.

Full Generation and Classification

We've released the code and scripts for training attribute classifiers and running them on model generations. The process is a bit messy. Here are the rough steps:

1. Train generative models (scripts/train_generative_models.sh, as above)

• Or, use a checkpoint we provide.
• This uses sc/generative.py or sc/encoder_decoder.py under the hood

2. Generate (scripts/generate_all.sh)

• You will probably want to use fewer language models (change the lms bash var)
• This uses sc/generate_texts.py under the hood

3. Turn generations (txt) into classifier-friendly format (table) (sc/scripts/output_to_table.py)

4. Move classifier inputs each into their own directory (sc/scripts/move_classifier_inputs.py)

5. Train attribute classifiers (scripts/classifier_runs.sh)

• This uses sc/classify.py under the hood
• Note that here we're training on the ground truth Social Chem 101 data, not the model generations. As such, this step doesn't require any of the previous ones as a prerequisite.

6. Run trained classifiers on model generations (sc/scripts/run_classifier.py)

• This uses sc/classify.py under the hood

Note: sc/model/cleaning.py exists. If memory serves, its primary purpose was for generative outputs so malformed they couldn't be subsequently parsed, but it's also possible it was used more broadly to clean up occasional decoding glitches in all outputs.

Note: This was run during experiments for the paper, but unfortunately, the libraries we've used have since broken: the huggingface transformers v2.11 release hardcodes a version of their tokenizers library (v0.7) that fails to build currently because it uses rust and it contains rust code that errors with a modern rust compiler. As such, we haven't tested the above process using the latest repository structure. So, it's highly unlikely the whole pipeline "just works" without some tweaks. Please feel free to file a GitHub issue.

Citation

@conference{forbes2020social,
    title = {Social Chemistry 101: Learning to Reason about Social and Moral Norms,
    author = {Maxwell Forbes and Jena D. Hwang and Vered Shwartz and Maarten Sap and Yejin Choi},
    year = {2020},
    date = {2020-11-16},
    booktitle = {Conference on Empirical Methods in Natural Language Processing (EMNLP)},
}

Dataset format

The dataset (social-chem-101.v1.0.tsv) is tab-separated with the following columns:

More information about split and m columns:

• split

  • the train/dev/test splits all have 1 worker / RoT breakdown. These are for training models (generative & discriminative).
  • there are additionally dev-extra and test-extra “splits” with 5 (additional) workers / RoT breakdown. These are synchronized so that dev-extra come from a subset of the dev set, same with test-extra from test. If we want, this lets us do a more nuanced scoring for these subsets (e.g., 5 correct answers or majority voting).
  • the analysis split comes from the 50-worker per RoT breakdown annotations

• m (how many worker annotated an RoT) is pretty straightforward, with a few twists:

  • m = 1 is the vast majority (like 99%) of the train/dev/test “main” dataset
  • m = 3 is a super small subset of our data. I did 3 workers early on for a couple batches just to get some agreement numbers. For that subset, we pick 1 breakdown to go towards the main dataset (i.e., then partitioned into train/val/test along with the m=1 annotations), and we mark the other 2 with none as their split.
  • m = 5 RoTs (in {dev,test}-extra) were sampled fully at random across all RoTs from each domain, with the condition that the RoT wasn’t marked as “bad” (unclear) during the 1-worker annotation. (The dev/test sets were then expanded from the situations in these initial subsets.)
  • m = 50 RoTs are for worker (not dataset) analysis. As such, the RoTs they use are not uniformly sampled from our data. (There’s also no constraint all RoTs for a situation make it in.) Instead, we take RoTs from the m = 5 annotation, find ones that <= 1 people marked as “experience” (so they will likely get the full RoT breakdown), and then sort by “most controversial,” i.e. lowest agreement in the m = 5 annotation. We annotate a small number of these maximally controversial RoTs from each domain with 50 workers.

• Other small notes:

  • The train/dev/test splits are are sampled by situation. So, all RoTs for a situation go to the same split. Also, each domain (AITA, rocstories, etc.) is split independently 80/10/10, so the domain proportions are the same across splits.
  • The {dev,test}-extra splits are also sampled by situation (all RoTs for a situation go to the same split). However, they are the same size for each domain.
  • If you want to get the “main” dataset for training models, don’t select by m=1! Instead, select by split = train (or dev or test). This is because a small portion of the dataset has m=3 — but with 1 annotation making it into the data splits, and the other 2 being assigned the none split.

TODO

Docs, code, and data to port over:

• docs: installation instructions
• test: installation on fresh env
• code: train generative (LM + enc/dec) models
• test: training generative models
• data: pretrained models
• code: example of using model (command line "demo")
• code: example of using model to generate full dataset outputs
• code: classifier
• code: baselines
• code: human evaluation