Home

Awesome

SQAD: Automatic Smartphone Camera Quality Assessment and Benchmarking

Official webpage for ICCV'23 paper "SQAD: Automatic Smartphone Camera Quality Assessment and Benchmarking"<br>

Dataset download: SQAD

You can download our pre-trained models from this link: ResNet-based / MANIQA-based

Baselines

Here we provide some baselines from sota MANIQA model for testing on sample_crop16 subset and some randomly generated test set from test folder. Within one image, the information levels (i.e. edge details, color vividness) for different parts usually vary, there would be fluctuations for the evaluation metrics if you apply your own cropping.

Results evaluated with sample_crop16 subset:

MetricsResolutionColor AccuracyNoiseDRPSFAliasing
SROCC0.92920.94750.93360.95460.94240.9172
PLCC0.94600.95920.97840.96940.98230.9812

Results evaluated with randomly cropped subset from test folder:

MetricsResolutionColor AccuracyNoiseDRPSFAliasing
SROCC0.94750.92420.87560.93280.91170.8940
PLCC0.94790.94140.93730.93320.97040.9688
ModelsMetricsSET 1SET 2SET 3SET 4SET 5Overall
ResNet50SROCC0.57470.55900.67000.40170.84810.6212
ResNet50PLCC0.81610.42440.68370.65390.61600.6075
MANIQASROCC------
MANIQAPLCC0.65850.40870.83540.63240.75290.4486

Model training

For the quality factor regression with MANIQA model, please follow the guidelines provided by CVPRW 2022: MANIQA to set up the environment. Thanks for their great works on image quality assessment.<br>

For ResNet50-based backbone, there is no specific requirement for the environment settings. Just make sure you have installed pytorch and Pillow properly. If there are some package related issues, follow the error instructions.

Dataset Preparation

SQAD
|—— data
|	|—— processed
|	|—— train
|	|	|—— 01_ASUS_Z00AD
|	|	|	|—— P_20000102_085109.jpg & ...
|	|	|—— 02_...
|	|—— test
|	|	|—— 01_ASUS_Z00AD
|	|	|—— 02_...
|	|—— sample_crop16
|—— prepare_data.py
|—— camera_ground_truth.json
|—— maniqa_plugin
|—— resnet_

To get the image crops for training, simply run:

python prepare_data.py

Or use the following command to specify the crop number and crop size (based on backbone network) per-image. You can also add --crop_test_dataset flag to create a randomly cropped test set for model evaluations.

python prepare_data.py --crop_num 32 --crop_size 240 --crop_test_dataset

Then, the cropped images and corresponding label files for training are stored in ./data/processed.<br>

For model testing, we have provided a pre-cropped image sets in the folder sample_crop16, which is extracted from the test folder. But it is free to use --crop_test_dataset flag to create your own test test as well.

Train with MANIQA

Please clone the MANIQA project and replace the original train_maniqa.py and inference.py by the files with the same name in maniqa_plugin folder.

git clone https://github.com/IIGROUP/MANIQA.git

Then, add camera29.py in maniqa_plugin/data into the original MANIQA data folder. Now you can modify the configurations in train_maniqa.py Line136-141 and inference.py Line75-78 to play with the model training and quality factor inference.

Train with ResNet backbone

To train the regression model with ResNet50 backbone, run train_regress.py file in the resnet_ folder with different quality factors:

python train_regress.py --factor dr(resolution/color_acc/...) --batch_size 64 --train_epochs 200 --save_eval_every 1