Awesome

Add Temporal Coherence to Single Image AI Upscaling Models in Vapoursynth

Also known as temporal consistency, line wiggle fix, stabilization, temporal denoising, or temporal fix.
Intended for animation.

This will not add extra work to the upscaling on the GPU and instead run in parallel on the CPU.

Requirements

• fftw3.3 (required by mvtools)
  Windows: download and put .dll files in plugin folder or add to windows PATH
  Linux: apt install libfftw3-3 libfftw3-dev or compile from source
• mvtools (release v24 or newer)
• mvtools-sf (optional, only for tr > 6)
• temporalmedian
• motionmask
• fillborders
• retinex
• tcanny (release r14 or newer)
• ctmf
• rgvs

Setup

Put the vs_temporalfix.py file into your vapoursynth scripts folder.
Or install via pip: pip install git+https://github.com/pifroggi/vs_temporalfix.git

Usage

from vs_temporalfix import vs_temporalfix
clip = vs_temporalfix(clip, strength=400, tr=6, exclude="[10 20]", debug=False)

clip
Temporally unstable clip. Should have no black borders.
Must be in YUV or GRAY format. Full range (PC) input is recommended.

strength
Suppression strength of temporal inconsistencies. Higher means more aggressive. No influence on processing speed.
The best way to check is to find a static scene and increase this till details, lines and textures are stable.
400 works great in many cases. If you get blending/ghosting on small movements or blocky artifacts, reduce this.

tr
Temporal radius sets the number of frames to average over. Higher means more stable. Influences processing speed.
The best way to check is to find a slow pan or zoom and increase this till details, lines and textures are stable.
6 works great in many cases. There is no downside to increasing this further, other than speed and RAM usage.

exclude (optional)
Optionally exclude scenes with intended temporal inconsistencies (like TV noise), or in case this doesn't work.
Example setting 3 scenes: exclude="[10 20] [600 900] [2000 2500]"
First number in the brackets is the first frame of the scene, the second number is the last frame (inclusive).

debug (optional)
Shows protected areas, scene changes and exclusions in pink half transparent on top of the clip. Protected areas have motions that are large enough to exclude from processing. This avoids blending/ghosting.

Tips & Troubleshooting

[!CAUTION]

• If fps are much lower than the benchmarks, try adding core.max_cache_size = 20000 (20GB) near the top of your vapoursynth script to allow higher RAM usage. For high tr or resolution, increase further if needed.

[!TIP]

• There is a big drop in performance for tr > 6, due to switching from mvtools to mvtools-sf, which is slower.
• mvtools-sf release r9 and the r10 pre-release will both work, but r9 is faster for me.
• Make sure to check very dark, hazy, or faint scenes for blending/ghosting and reduce strength if necessary.

Benchmarks