Awesome

A Cloud 3D Dataset and Application-Specific Learned Image Compression in Cloud 3D

Tianyi Liu, Sen He, Vinodh, Wei Wang A Cloud 3D Dataset and Application-Specific Learned Image Compression in Cloud 3D, European Conference on Computer Vision (ECCV), 2022

Overview

In Cloud 3D, such as Cloud Gaming and Cloud Virtual Reality (VR), image frames are rendered in the cloud, and then compressed (encoded) and sent to the clients for users to view and interact. To achieve both low latency and high image quality, fast, high compression rate, and high-quality image compression techniques are highly preferable. This paper explores computation time reduction techniques for learned image compression to make it more suitable for cloud 3D. More specifically, we employed slim (low-complexity) and application-specific AI models to reduce the computation time without harming the image quality. Our approach is based on two key insights: (1) as the frames generated by a 3D application are highly homogeneous, application-specific training datasets can improve the rate-distortion performance of the resulting compression model, compared to a general training dataset; (2) many of the computer-generated frames from 3D applications are less complex than natural photos, which also makes it feasible to employ less complex and faster compression models. We evaluated our models on six gaming image datasets. The results show that our approach has similar rate-distortion performance as a state-of-the-art learned image compression algorithm, while obtaining about 5x to 9x speedup and reducing the compression time to be less than 1 second (0.74s), bringing learned image compression closer to being viable for cloud 3D.

This project is based on Yueyu Hu's Coarse-to-fine paper, but the methods should be also suitable for other learned image frameworks.

Yueyu Hu, Wenhan Yang, Jiaying Liu, Coarse-to-Fine Hyper-Prior Modeling for Learned Image Compression, AAAI Conference on Artificial Intelligence (AAAI), 2020

How to use this repo?

Preparation

Clone this repo:

$ git clone this_repo

Before running the python script, you need to compile the arithmetic coder, with:

g++ module_arithmeticcoding.cpp -o module_arithmeticcoding

You need to download the <a href="https://utsacloud-my.sharepoint.com/:u:/g/personal/wei_wang_utsa_edu/EROyq4JeQlxLuhWIFD-RFZ8BR69M1fiQk6HhhaoAE_TZnA?e=iJS5nJ">Cloud 3D datasets (six gaming image datasets)</a> from UTSA Cloud. It is about 23.23GB, and might takes a while for downloading. Create a dataset folder out of current repo:

$ mkdir ../datasets

Please unzip the datasets and the directories look like below:

$ ls ../datasets/
GameImage_dataset

$ ls ../datasets/GameImage_dataset
resize.py  test  train  validation

If you would like to try the compression and decompression directly, you also need to download some <a href="https://utsacloud-my.sharepoint.com/:u:/g/personal/wei_wang_utsa_edu/EbBE6_Gv0PRLghkT9l8r5t0B1abY2cNv9hEZZgWm28PDAg?e=SnWscC">demo trained weights</a> from UTSA Cloud. After downloading and unziping trained weights to 'models' folder, the 'models' folder is like:

$ ls ./models:
large  median  small  small-spruning  sota  xsmall

Help

./evaluate.sh

If you haven't install the conda environment, please install miniconda and create py37 workspace, and install the pytorch. For example: after installing miniconda,

$ conda create py37
$ condo activate py37
$ conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch

Of course, you should have a GPU on your machine with cudatookit 11.3. Then, you can try the following commands to see if GPU works well with pytorch.

$ python
>>> import torch
>>> torch.cuda.is_available()

Encoder & Decoder (GPU Mode)

(1) Test single image with single model:

Test 105.png with large model:

./evaluate.sh -m "./models/large/supertuxkart_lambda0.16_bpp1.52_mse14.6_1600.ckpt" -t "../datasets/GameImage_dataset/test/supertuxkart-720p/105.png" -s 3

Then, you will find a newly generated ".csv" file in current directory, and it contains the metrics we care about.

Similarly, you can try median model, small model, xsmall model with -s = 2, 1, 0

./evaluate.sh -m "./models/median/supertuxkart_lambda0.001.ckpt" -t "../datasets/GameImage_dataset/test/supertuxkart-720p/105.png" -s 2
./evaluate.sh -m "./models/small/supertuxkart_lambda0.001.ckpt" -t "../datasets/GameImage_dataset/test/supertuxkart-720p/105.png" -s 1
./evaluate.sh -m "./models/xsmall/supertuxkart_lambda0.001.ckpt" -t "../datasets/GameImage_dataset/test/supertuxkart-720p/105.png" -s 0

(2) Test all the images with all the models in the GamingImage datasets (for each command, it will take a very long time). For example:

./evaluate.sh -n 0ad -s 1

You can have other combinations:

./evaluate.sh -n supertuxkart/0ad/redeclipse/dota2/inmind/imhotep -s 3/2/1/0

For each run, you will also get a newly generated ".csv" file.

How to train your own models?

The training code is to help developers train their own compression models for a specific range of bit-rate. Please check the train/ sub-directory.