--vpp-rtgmc seems to do nothing

[Kuprin2000]

rSystem: Windows 11
Version: 9.19
Bug:
I am trying to deinterlace decoded.mp4

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'decoded.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2mp41
    encoder         : Lavf62.3.100
  Duration: 00:00:30.44, start: 0.000000, bitrate: 9490 kb/s
  Stream #0:0[0x1](und): Video: hevc (Main) (hev1 / 0x31766568), yuv420p(tv, bt470bg/bt470bg/bt709, top coded first (swapped)), 928x576 [SAR 259:311 DAR 7511:5598], 9488 kb/s, 25 fps, 25 tbr, 12800 tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
      encoder         : Lavc62.11.100 hevc_nvenc
      timecode        : 00:00:00:00
  Stream #0:1[0x2](eng): Data: none (tmcd / 0x64636D74), 0 kb/s
    Metadata:
      handler_name    : TimeCodeHandler
      timecode        : 00:00:00:00
Unsupported codec with id 0 for input stream 1

with this command
"C:\Program Files\QSVEnc\NVEncC64.exe" --avhw --interlace tff -i decoded.mp4 --vpp-rtgmc preset=medium -c hevc -o result.mp4
NVEnc runs with no errors

--------------------------------------------------------------------------------
result.mp4
--------------------------------------------------------------------------------
NVEncC (x64) 9.19 (r3716) by rigaya, Jun 15 2026 14:05:49 (VC 1944/Win)
OS Version     Windows 11 x64 (26200) [UTF-8]
CPU            AMD Ryzen 7 7700 8-Core Processor [4.34GHz] (8C/16T)
GPU            #0: NVIDIA GeForce RTX 3070 (5888 cores, 1725 MHz)[PCIe4x16][595.97]
NVENC / CUDA   NVENC API 13.0, CUDA 13.2, schedule mode: auto
Input Buffers  CUDA, 20 frames
Input Info     avcuvid: H.265/HEVC, 928x576, 25/1 fps
Vpp Filters    cspconv(nv12 -> yv12)
               rtgmc: preset medium, tuning none, border false, lossless 0, input_type 0, prog_sad_mask 10.000, prog_sad_mask_gamma 1.000, mv_spatial_refine -1, source_match 0, match_tr1 1, match_tr2 1, match_edi nnedi3, match_enhance 0.500, order auto, tr0 2, rep0-thin 3, rep0-pad 0, search_refine 3, rtgmc-noise: process 0, ezdenoise 0.000, ezkeepgrain 0.000, denoiser fft3d, noise_deint none, sigma 2.000, chroma_noise true, denoise_mc false, noise_tr 0, grain_restore 0.000, noise_restore 0.000, edi nnedi3, nnsize 5, nneurons 1, ediqual 1, chroma_edi none, tr1 thsad 640, rep1-thin 0, rep1-pad 0, retouch smode 2/slmode 2/precise false/sharpness 1.000, tr2 thsad 256, rep2-thin 4, rep2-pad 0
               cspconv(yv12 -> nv12)
Output Info    H.265/HEVC main @ Level auto
               928x576p 259:311 50.000fps (50/1fps)
               avwriter: hevc => mp4
Encoder Preset default
Rate Control   VBR
Multipass      none
Bitrate        0 kbps (Max: 9600 kbps)
Target Quality 25.00
QP range       I:0-51  P:0-51  B:0-51
QP Offset      cb:0  cr:0
VBV buf size   auto
Split Enc Mode auto
Tuning Info    hq
Lookahead      off
GOP length     500 frames
B frames       3 frames [ref mode: middle]
Ref frames     5 frames, MultiRef L0:auto L1:auto
AQ             off
CU max / min   32 / auto
Others         mv:auto

encoded 1522 frames, 356.77 fps, 8121.65 kbps, 29.47 MB
encode time 0:00:04, CPU: 6.0%, GPU: 66.6%, VE: 26.0%, VD: 6.2%, GPUClock: 1554MHz, VEClock: 1431MHz
frame type IDR    4
frame type I      4,  avgQP  25.00,  total size   0.32 MB
frame type P    381,  avgQP  27.10,  total size  17.72 MB
frame type B   1137,  avgQP  30.58,  total size  11.43 MB

and creates file result,mp4

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'result.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 512
    compatible_brands: mp42iso2mp41
    encoder         : NVEncC (x64) 9.19
  Duration: 00:00:30.44, start: 0.000000, bitrate: 8126 kb/s
  Stream #0:0[0x1](und): Video: hevc (Main) (hvc1 / 0x31637668), yuv420p(tv, progressive), 928x576 [SAR 259:311 DAR 7511:5598], 8121 kb/s, 50 fps, 50 tbr, 12800 tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]

But if I play it in VLC player I can still see the lines.

Before (internal deinterlacing of VLC is disabled):

After (slightly different moment, internal deinterlacing of VLC is disabled):

[Kuprin2000]

Here is the video
https://drive.google.com/file/d/1Zo80Z8jHryu6eRM-cFTJx5niEPLAr0bX/view?usp=sharing

[teaching-droid]

I can reproduce this. rtgmc does run and it doubles 25i to 50p, but it leaves residual combing on motion.

I traced it to the EDI stage. The residual combing comes from the nnedi3 interpolator. Every rtgmc preset uses edi=nnedi3 by default, so it shows up out of the box. Switching the EDI to yadif or bob removes it completely.

Test on the sample clip (single frame with motion, native pixels, lossless output so the encoder is not a factor):

edi=nnedi3 (default): comb teeth on moving edges
edi=yadif: clean
edi=bob: clean
The combing does not change with sharpness, source_match or preset, so it is not a tuning question, it is in the nnedi3 EDI path.

Workaround for now (clean output, still keeps the rtgmc temporal processing):

NVEncC64.exe --avhw --interlace tff -i decoded.mp4 --vpp-rtgmc preset=medium,edi=yadif -c hevc -o result.mp4
edi=bob also works if preferred.

Comparison below, same frame and same zoom: left edi=nnedi3 (default) is combed, middle edi=yadif is clean, right edi=bob is clean.

[rigaya]

The root cause of this issue is that, although the interlace flag is set on the container side, the video frames themselves are not reported as being encoded as interlaced, even when using ffprobe.

For example checking your attached input file with ffprobe shows interlaced_frame=0 for all frames.

ffprobe -v error -select_streams v:0 \
  -show_entries frame=coded_picture_number,best_effort_timestamp_time,pict_type,interlaced_frame,top_field_first \
  -of compact=p=0:nk=1 "input.mp4"

0.000000|I|0|0
0.040000|B|0|0
0.080000|B|0|0
0.120000|B|0|0
0.160000|P|0|0
0.200000|B|0|0
0.240000|B|0|0
0.280000|B|0|0
0.320000|P|0|0
0.360000|B|0|0
0.400000|B|0|0
0.440000|B|0|0
0.480000|P|0|0
0.520000|B|0|0
0.560000|B|0|0
0.600000|B|0|0
0.640000|P|0|0
0.680000|B|0|0
0.720000|B|0|0
0.760000|B|0|0
0.800000|P|0|0
0.840000|B|0|0
0.880000|B|0|0
0.920000|B|0|0
0.960000|P|0|0
1.000000|B|0|0
1.040000|B|0|0
1.080000|B|0|0
...

From NVEnc’s design perspective, by default deinterlacing is set to follow the per-frame interlace flags. Therefore in default settings this will not be deinterlaced.

I have updated NVEnc 9.20 to allow to force deinterlace by using --vpp-rtgmc order=tff.

[Kuprin2000]

Thank you very much! I will test the new version today