Awesome
TransTIC: Transferring Transformer-based Image Compression from Human Visualization to Machine Perception
Accpeted to ICCV 2023
This repository contains the source code of our ICCV 2023 paper TransTIC arXiv.
Abstract
This work aims for transferring a Transformer-based image compression codec from human vision to machine perception without fine-tuning the codec. We propose a transferable Transformer-based image compression framework, termed TransTIC. Inspired by visual prompt tuning, we propose an instance-specific prompt generator to inject instance-specific prompts to the encoder and task-specific prompts to the decoder. Extensive experiments show that our proposed method is capable of transferring the codec to various machine tasks and outshining the competing methods significantly. To our best knowledge, this work is the first attempt to utilize prompting on the low-level image compression task.
Install
git clone https://github.com/NYCU-MAPL/TransTIC
cd TransTIC
pip install -U pip && pip install -e .
pip install timm tqdm click
Install Detectron2 for object detection and instance segementation.
Dataset
The following datasets are used and needed to be downloaded.
- Flicker2W (download here, and use this script for preprocessing)
- ImageNet1K
- COCO 2017 Train/Val
- Kodak
Example Usage
Specify the data paths, target rate point, corresponding lambda, and checkpoint in the config file accordingly.
Base Codec (for PSNR)
python examples/train.py -c config/base_codec.yaml
Classification
python examples/classification.py -c config/classification.yaml
<br>
Add argument -T
for evaluation.
Object Detection
python examples/detection.py -c config/detection.yaml
<br>
Add argument -T
for evaluation.
Instance Segmentation
python examples/segmentation.py -c config/segmentation.yaml
<br>
Add argument -T
for evaluation.
Pre-trained Weights
Tasks | ||||
---|---|---|---|---|
Base codec (TIC) | 1 | 2 | 3 | 4 |
Classification | 1 | 2 | 3 | 4 |
Object Detection | 1 | 2 | 3 | 4 |
Instance Segmentation | 1 | 2 | 3 | 4 |
Citation
If you find our project useful, please cite the following paper.
@inproceedings{TransTIC,
title={TransTIC: Transferring Transformer-based Image Compression from Human Visualization to Machine Perception},
author={Chen, Yi-Hsin and Weng, Ying-Chieh and Kao, Chia-Hao and Chien, Cheng and Chiu, Wei-Chen and Peng, Wen-Hsiao},
booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
pages={},
year={2023}
}
Ackownledgement
Our work is based on the framework of CompressAI. The base codec is adopted from TIC/TinyLIC and the prompting method is modified from VPT. We thank the authors for open-sourcing their code.