Awesome

German Text Embedding Clustering Benchmark

Shortcut: Datasets - Results - Installation - Usage - Citation

Remarks

This repository contains code to evaluate language models for clustering word embeddings as used in neural topic modelling (see for example BERTopic) specifically for <b>German</b>. This work builds on Massive Text Embedding Benchmark (MTEB), which provides benchmark datasets and results for a wide range of tasks.

More specifically, this work contributes to mteb in the following ways:

• clustering datasets in German (MTEB only consider English datasets)
• the evaluation of more clustering algorithms

:trophy: Note that you can contribute results to the MTEB leaderboard as our datasets are officially part of MTEB (apart from the Reddit datasets, see below)! You can either use this library or MTEB directly to produce results. If you run into any problems, please raise an issue. :trophy:

Datasets

Currently, we provide 4 datasets. The datasets are built similarly to the English clustering datasets in MTEB. Unfortunately, there are fewer datasets available for German and, therefore, we were not able to build as many datasets (e.g. Arxiv only contains very few German papers). However, we plan to add more datasets in the future.

Reddit datasets

We also include two Reddit datasets in the benchmark (similar to MTEB's RedditClustering and RedditClusteringP2P datasets). However, we only provide ids, and if you want to use these datasets, you need to download the data yourself (see Including the Reddit dataset for instructions). The datasets contain "hot" and "top" submissions to 80 popular German subreddits and were extracted using PRAW.

Important: As of June 19, 2023 new Data API Terms become effective for Reddit. Most likely, it will not be allowed anymore to use Reddit data for such purposes (see especially "2.4 User Content" in the terms). Make sure you understand these terms and use Reddit data accordingly.

Results

All results show the V-measure (multiplied by 100 and rounded to two decimal points).

k-means (same as MTEB)

additional clustering algorithms

In addition to k-means, we evaluate the following different clustering algorithms:

• Agglomerative clustering (distance-based, number of clusers is assumed)
• HDBSCAN (density-based, number of clusters is assumed)
• DBSTREAM (streaming algorithm, number of clusters is not assumed)

Inspired by BERTopic, we also evaluate PCA and UMAP as a "preprocessing" step.

If you want add/evaluate more algorithms, please have a look at FlexibleClusteringEvaluator.py on how to achieve that.

UMAP + {k-means, HDBSCAN, DBSTREAM}

For all results have a look at results/tecb-de-full-results.csv.

Installation

If you want to run the code from this repository (for creating the Reddit dataset or model evaluation), clone this repository and move to the downloaded folder

git clone https://github.com/ClimSocAna/tecb-de.git 

and create a new environment with the necessary packages

python -m venv tecb-de

source tecb-de/bin/activate # Linux, MacOS
venv\Scripts\activate.bat # Windows

pip install -r requirements.txt

Usage

Running the evaluation

Simply run python scripts/run_cteb_de.py. This will produce an results folder. You can modify the script to run the evaluation for models and clustering algorithms (and configuration) of your choosing.

Including the Reddit dataset

If you want to use the reddit dataset, you first have to download the data

# move to the reddit_data folder in tecb-de
# make sure you have PRAW and tdqm installed: pip install praw, pip install tqdm

# downloads the data and saves it to submissions.tsv
python download.py

Note that for this to work, you have to edit the reddit_data/praw.ini with your client data. You can find instructions here.

Then you can create the datasets

# creates the splits for both tasks (RedditClusteringS2S and ReddictClusteringP2P)
# and saves them in the reddit_data folder
python create_splits.py

Finally, you can can run the evaluation using the --include-reddit flag

# assuming your position is in the top-level folder
python scripts/run_cteb_de.py --reddit-flag

Adaptive pre-training

If you want to experiment with adaptive pre-training, you can have a look at scripts/run_fine_tuned_cteb_de.py. Basically, it allows you to train models using whole word masking (WWM) and TSDAE and to evaluate on a clustering algorithm during training.

Citation

If you make use of this work, please cite:

@inproceedings{wehrli-etal-2023-german,
    title = "{G}erman Text Embedding Clustering Benchmark",
    author = "Wehrli, Silvan  and
      Arnrich, Bert  and
      Irrgang, Christopher",
    editor = "Georges, Munir  and
      Herygers, Aaricia  and
      Friedrich, Annemarie  and
      Roth, Benjamin",
    booktitle = "Proceedings of the 19th Conference on Natural Language Processing (KONVENS 2023)",
    month = sep,
    year = "2023",
    address = "Ingolstadt, Germany",
    publisher = "Association for Computational Lingustics",
    url = "https://aclanthology.org/2023.konvens-main.20",
    pages = "187--201",
}