Awesome

SynSE - Syntactically Guided Generative Embeddings For Zero Shot Skeleton Action Recognition

Original PyTorch implementation for 'Syntactically Guided Generative Embeddings For Zero Shot Skeleton Action Recognition' , accepted at 'IEEE International Conference on Image Processing (ICIP) 2021'

<b> TL;DR version of the work: </b> <a href="https://threadreaderapp.com/thread/1356893399576580100.html"> HERE </a>

Dependencies

Data Preparation

Creating the test-train splits.

The unseen classes of the various splits are listed below. These splits are also provided under the <code> synse_resources/resources/label_splits</code>, which can be downloaded from here. Place the <code>resources</code> folder in the root <code>synse-zsl</code> directory. Random unseen 5 classes can be found in the ru5.npy file. This naming scheme is used for all splits. R-random, S-seen, U-unseen, V-validation split.

NTU-60:

Unseen Classes (55/5 split):

Unseen Classes (48/12 split):

NTU-120:

Unseen Classes (110/10 split):

Unseen Classes (96/24 split):

Visual Feature Generation:

We provide the visual features generated via SHIFT-GCN for the NTU-120 and NTU-60 dataset for the various splits. They can be found under the <code>synse_resources/ntu_results</code> repository, which is downloadable from here. train.npy contains the visual features of the training data from the seen classes. ztest.npy contains the test data from the unseen classes. gtest.npy contains the test data from all the classes.

If you wish to generate the visual features yourself:

1. Download the NTU-60 and NTU-120 datasets by requesting them from <a href="http://rose1.ntu.edu.sg/Datasets/actionRecognition.asp">here</a>.

2. Create the test-train-val splits for the datasets using the split file created in the previous steps.

3. Train the visual feature generator. Follow this for training Shift-GCN. For each split a new feature generator has to be trained following the zero shot learning assumption. The trained Shift-GCN weights can be found under the repository. <code>synse_resources/ntu_results/shift_5_r/weights/</code>

4. Save the features for train data, unseen test data(zsl) and the entire test data(gzsl).

Text feature generators

We provide the generated language features as well, for the labels in NTU-60, and NTU-120 dataset. They can be found in <code>./synse_resources/resources/</code>. Place the <code>resources</code> folder in the root <code> synse-zsl</code> directory.

If you wish to generate the language features yourself.

1. Word2Vec: Download the <a href="https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit">Pre-Trained Word2Vec Vectors</a> and extract the contents of the archive. Generate the Word2Vec representations by using the gensim python module as described here </li>

2. For Sentence-BERT, we use the sentence-transformers package from here. We use the stsb-bert-large model.

Experiments

We provide the scripts necessary to obtain the results shown in the paper. They include training and evaluation scripts for ReViSE [1], JPoSE[2], CADA-VAE[3] and our model SynSE. The scripts for each of the three models are present in their respective folders (jpose, revise, synse). <br> A README is present in each folder detailing the use of the provided scripts for both training and evaluation.

References: