Awesome

VQ-Rec

PyTorch implementation of the paper

Yupeng Hou, Zhankui He, Julian McAuley, Wayne Xin Zhao. Learning Vector-Quantized Item Representation for Transferable Sequential Recommenders. TheWebConf 2023.

Updates:

• [Mar. 27, 2023] We fixed two minor bugs in pre-training (Raised by UniSRec#9 and an email from Xingyu Lu, respectively. Thanks a lot!!). We pre-trained VQ-Rec again and the new pre-trained model has been uploaded as pretrained/VQRec-FHCKM-300-20230315.pth. We also evaluated the new pre-trained model on six downstream datasets. Generally, the new pre-trained model performs better. Please refer to Results for more details.

Overview

Recently, the generality of natural language text has been leveraged to develop transferable recommender systems. The basic idea is to employ pre-trained language model~(PLM) to encode item text into item representations. Despite the promising transferability, the binding between item text and item representations might be "too tight", leading to potential problems such as over-emphasizing the effect of text features and exaggerating the negative impact of domain gap. To address this issue, this paper proposes VQ-Rec, a novel approach to learning <ins>V</ins>ector-<ins>Q</ins>uantized item representations for transferable sequential <ins>Rec</ins>ommender. The major novelty of our approach lies in the new item representation scheme: it first maps item text into a vector of discrete indices (called item code), and then employs these indices to lookup the code embedding table for deriving item representations. Such a scheme can be denoted as "text ==> code ==> representation". Based on this representation scheme, we further propose an enhanced contrastive pre-training approach, using semi-synthetic and mixed-domain code representations as hard negatives. Furthermore, we design a new cross-domain fine-tuning method based on a differentiable permutation-based network.

Requirements

recbole==1.0.1
faiss-gpu==1.7.2
python==3.8.13
cudatoolkit==11.3.1
pytorch==1.11.0

Dataset

We use the processed datasets from UniSRec. Please merge (but not replace!!!) the current dataset/ folder and the downloaded folders from UniSRec.

Pre-trained Model and Item Codes

The original pre-trained model is located at pretrained/VQRec-FHCKM-300.pth. This checkpoint was created in Oct. 2022 and used for all our experiments reported in our paper.

We also uploaded a new pre-trained model at pretrained/VQRec-FHCKM-300-20230315.pth. We fixed two bugs in our pre-training scripts and created this checkpoint in Mar. 2023. Associated results can be found at Results

The pre-trained item codes (both on pre-training and downstreaem datasets) are located at dataset/.

Reproduction

To quickly reproduce the reported results, you can run the following scripts.

python finetune.py -d Scientific -p pretrained/VQRec-FHCKM-300.pth -f fix_enc --learning_rate=0.003
python finetune.py -d Pantry -p pretrained/VQRec-FHCKM-300.pth -f fix_enc --learning_rate=0.003
python finetune.py -d Instruments -p pretrained/VQRec-FHCKM-300.pth -f fix_enc --learning_rate=0.001
python finetune.py -d Arts -p pretrained/VQRec-FHCKM-300.pth -f fix_enc --learning_rate=0.003
python finetune.py -d Office -p pretrained/VQRec-FHCKM-300.pth -f fix_enc --learning_rate=0.003
python finetune.py -d OR -p pretrained/VQRec-FHCKM-300.pth -f fix_enc --learning_rate=0.003

Quick Start

Data Preparation

Preparing item codes for pre-training:

python multi_pq.py --gpu_id 0

Preparing item codes for fine-tuning:

python pq.py --dataset Scientific --gpu_id 0

Downstream

Train recommender from scratch (w/o pre-training):

CUDA_VISIBLE_DEVICES=0 python finetune.py -d Scientific --gpu_id=0

Fine-tune pre-trained recommender:

CUDA_VISIBLE_DEVICES=0 python finetune.py -d Scientific -p pretrained/VQRec-FHCKM-300.pth -f fix_enc --gpu_id=0

Pre-train

Pre-train on a single GPU:

CUDA_VISIBLE_DEVICES=0 python pretrain.py --train_batch_size=2048 --gpu_id=0

Pre-train on multiple GPUs:

CUDA_VISIBLE_DEVICES=0,1,2,3 python ddp_pretrain.py

Results

We fixed two bugs in March 2023 and re-trained a new version of pre-trained VQ-Rec model as pretrained/VQRec-FHCKM-300-20230315.pth. The fine-tuned results on six downstream datasets are presented here. Improved metrics (compared to the results in our paper) are denoted as bold.

These results can be reproduced by running the following scripts.

python finetune.py -d Scientific -p pretrained/VQRec-FHCKM-300-20230315.pth -f fix_enc --learning_rate=0.003
python finetune.py -d Pantry -p pretrained/VQRec-FHCKM-300-20230315.pth -f fix_enc --learning_rate=0.003
python finetune.py -d Instruments -p pretrained/VQRec-FHCKM-300-20230315.pth -f fix_enc --learning_rate=0.003
python finetune.py -d Arts -p pretrained/VQRec-FHCKM-300-20230315.pth -f fix_enc --learning_rate=0.001
python finetune.py -d Office -p pretrained/VQRec-FHCKM-300-20230315.pth -f fix_enc --learning_rate=0.003
python finetune.py -d OR -p pretrained/VQRec-FHCKM-300-20230315.pth -f fix_enc --learning_rate=0.003

Acknowledgement

The implementation is based on UniSRec and the open-source recommendation library RecBole.

Please cite the following papers as references if you use our implementations or the processed datasets.

@inproceedings{hou2023vqrec,
  author = {Yupeng Hou and Zhankui He and Julian McAuley and Wayne Xin Zhao},
  title = {Learning Vector-Quantized Item Representation for Transferable Sequential Recommenders},
  booktitle={{TheWebConf}},
  year = {2023}
}

@inproceedings{zhao2021recbole,
  title={Recbole: Towards a unified, comprehensive and efficient framework for recommendation algorithms},
  author={Wayne Xin Zhao and Shanlei Mu and Yupeng Hou and Zihan Lin and Kaiyuan Li and Yushuo Chen and Yujie Lu and Hui Wang and Changxin Tian and Xingyu Pan and Yingqian Min and Zhichao Feng and Xinyan Fan and Xu Chen and Pengfei Wang and Wendi Ji and Yaliang Li and Xiaoling Wang and Ji-Rong Wen},
  booktitle={{CIKM}},
  year={2021}
}

For the implementations of item code learning, thanks the amazing library faiss, thanks Jingtao for the great implementation of JPQ.