[TRTLLM-14616][feat] add VisualGen fp8 row-wise quant

[o-stoner]

@coderabbitai summary

Description

Adds FP8 row-wise (FP8_PER_CHANNEL_PER_TOKEN) dynamic quantization support to VisualGen.
This is a feature gap: FP8_PER_CHANNEL_PER_TOKEN is supported in TRT-LLM's general Linear
building blocks but was not wired up in VG's quantization plumbing.

Perf/quality report example for dynamic FP8 row-wise quant on FLUX.2-dev: https://sc.talos.nvidia.com/view/home/scratch.ostoner_coreai_1/TensorRT-LLM/examples/visual_gen/flux2_quant_comparison/FINDINGS.html .

Test Coverage

New unit tests:

• tests/unittest/_torch/visual_gen/test_quant_ops.py - quantize_fp8_rowwise op: output dtype, scale shape (out_features,), round-trip error
• tests/unittest/_torch/visual_gen/test_model_loader.py - config parse ("FP8_PER_CHANNEL_PER_TOKEN" → QuantAlgo, dynamic_weight_quant=True); weight loading on Wan 1.3B with weight_scale.dim() == 1 assertion

Extended existing tests (parametrize additions):

• tests/unittest/_torch/visual_gen/test_flux_pipeline.py - FP8_PER_CHANNEL_PER_TOKEN added to: FLUX.1/FLUX.2 weight loading, single-layer numerical correctness vs BF16, full transformer e2e cosine similarity vs BF16
• tests/unittest/_torch/visual_gen/test_ltx2_pipeline.py - weight loading, single-layer correctness, two-stage pipeline loading
• tests/unittest/_torch/visual_gen/test_cosmos3_transformer.py - transformer weight loading
• tests/unittest/_torch/visual_gen/test_qwen_image_pipeline_config.py - config parsing

Pre-existing tests covering linear.py change:

• tests/unittest/_torch/thop/parallel/test_fp8_rowwise_linear.py - FP8RowwiseLinearMethod forward correctness (covers the reshape code path indirectly via the full forward call)

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[o-stoner]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #61636 [ run ] triggered by Bot. Commit: 3e24bd2 Link to invocation

[tensorrt-cicd]

PR_Github #61636 [ run ] completed with state SUCCESS. Commit: 3e24bd2
/LLM/main/L0_MergeRequest_PR pipeline #49843 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation