Home

Awesome

#Structured Matching for Phrase Localization

Created by Mingzhe Wang at University of Michigan, Ann Arbor.

This is released code for:

Mingzhe Wang, Mahmoud Azab, Noriyuki Kojima, Rada Mihalcea, Jia Deng, Structured Matching for Phrase Localization, ECCV 2016. paper

To run this code, make sure the following are installed:

Getting Started

To test with our pretrained model, run:

cd workspace

Download our pretrained model:

./fetch_model.sh

Download features of test data:

./fetch_test_feat.sh

To evaluate the model of bipartite matching:

cd ../src/lua

th test_matching.lua

The predictions of bounding boxes and corresponding scores are stored in workspace/matching/. Open matlab and run evaluation code from Bryan A. Plummer et al:

p=runEval_arg('../../workspace/matching')

To specify a test file, run with -file test-model. To evaluate the model of structured matching:

cd ../lua

th test_pc.lua

The predictions of bounding boxes and corresponding scores are stored in workspace/matching_pc

p=runEval_arg('../../workspace/matching_pc')

In all experiments, we reported Recall@1 in our ECCV paper.

To train your own models, you need to extract features for both phrases and bounding boxes. You can also download our features (63G) with the following commands:

cd workspace/

./fetch_train_feat.sh

To train a model with bipartite matching, in the src/lua folder, run:

th train_matching.lua

To train a model with structured matching, run:

th train_matching_pc.lua

Output models are stored in workspace/model. Please refer to the comments in train_matching_pc.lua to adjust learning parameters.

To generate your own features, you need to

The following tools were used to extract features in our experiments:

Please let me know if you have any issues about our code.

Acknowledgements

Thanks to Geoff Leyland for providing an excellent implementation of simplex algorithms in lua.