Awesome

COO: Comic Onomatopoeia Dataset for Recognizing Arbitrary or Truncated Texts

Official repository of COO (ECCV 2022) | Paper | Dataset, Sample | Codes | Leaderboard | Poster | Video

We provide the COmic Onomatopoeia dataset (COO) and the source codes used in our paper. <br>

1. COO has many arbitrary texts, such as extremely curved, partially shrunk texts, or arbitrarily placed texts. Furthermore, some texts are separated into several parts. Each part is a truncated text and is not meaningful by itself. These parts should be linked to represent the intended meaning. Thus, we propose a novel task that predicts the link between truncated texts.

2. COO is a challenging text dataset. Detecting the onomatopoeia region and capturing the intended meaning of truncated texts are very difficult. If a model can recognize comic onomatopoeias, we expect that the model can also recognize other less difficult texts. We hope our work will encourage studies on more irregular texts and further improve text detection, recognition, and link prediction methods. <br>

3. In text detection and recognition tasks, researchers use various datasets to validate their methods. We hope COO can also help validate their methods. In addition, we hope COO is used to analyze Japanese onomatopoeia.

Dataset: Comic Onomatopoeia (COO)

We provide the annotations of the COO. <br> Several files that help preprocessing, visualization, and data analysis are in COO-data folder. <br> COO has 61,465 polygons and 2,261 links between truncated texts. <br> The below figure shows the COO statistics and character types of COO (182 types in total). <br>

Prerequisites: Download Manga109 images

According to the license of Manga109, the redistribution of the images of Manga109 is not permitted. <br> Thus, you should download the images of Manga109 via the Manga109 webpage. <br>

After downloading, unzip Manga109.zip and then move images folder of Manga109 into COO-data folder. <br> We need images folder in COO-data folder (i.e. COO-data/images) for preprocessing.

Preprocessing for each model

1. Run the following command.

   pip install Flask==2.0.2 Shapely==1.8.0 manga109api==0.3.1 pillow natsort lmdb opencv-python numpy tqdm
   

2. See the section dataset in each model folder.

Codes

The source codes used in our paper are provided in each folder. <br> For text detection, we used ABCNetv2 and MTSv3. <br> For text recognition, we used TRBA. <br> For link prediction, we used M4C-COO (a variant of M4C). <br>

Leaderboard

We will list the results of SOTA methods that provide the official code. <br> For the leaderboard, we report the performance of one pretrained model. <br> Note that we report the average value of three trials in our paper. <br> We welcome the pull requests containing an official code (URL) of other SOTA methods. <br>

Text detection

• P: Precision    R: Recall    H: Hmean    *: only use detection part.
• Based on the hmean, methods are sorted in descending order.

Text recognition

• Based on the accuracy, methods are sorted in descending order.

Link prediction

• P: Precision    R: Recall    H: Hmean
• Based on the hmean, methods are sorted in descending order.

Citation

When using annotations of comic onomatopoeia dataset (COO) or if you find this work useful for your research, please cite our paper.

@inproceedings{baek2022COO,
  title={COO: Comic Onomatopoeia Dataset for Recognizing Arbitrary or Truncated Texts},
  author={Baek, Jeonghun and Matsui, Yusuke and Aizawa, Kiyoharu},
  booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
  year={2022}
}

Contact

Feel free to contact us if there is any question: Jeonghun Baek ku21fang@gmail.com

License

For the dataset, annotation data of COO is licensed under a CC BY 4.0. <br> The license of image data of Manga109 is described here. <br>

For the codes made by us: MIT. <br> After examining the licenses of original source codes of each method used in our work, we found that the redistribution of source codes is permitted. Thus, to facilitate future work, we provide the source codes in this repository. Please let us know if there is a license issue with code redistribution. If so, we will remove the code and provide the instructions to reproduce our work.