-
Notifications
You must be signed in to change notification settings - Fork 2.6k
[TRTLLM-14135][feat] Add Qwen-Image-Edit-2511 support #16095
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
45f3edf
76ae83f
c8f8c88
67adb9f
2458136
8d0aa8f
160e5a9
f6f1272
2ed4aae
a9988a8
9d85816
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -37,6 +37,7 @@ TensorRT-LLM **VisualGen** provides a unified inference stack for diffusion mode | |
| | `Lightricks/LTX-2` | Text-to-Video (with Audio), Image-to-Video (with Audio) | | ||
| | `Qwen/Qwen-Image` | Text-to-Image | | ||
| | `Qwen/Qwen-Image-2512` | Text-to-Image | | ||
| | `Qwen/Qwen-Image-Edit-2511` | Image Editing | | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not sure if we should name like text+image-to-image, and also some other models such as Flux also supports text+image-to-image.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe |
||
| | `nvidia/Cosmos3-Nano` | Text-to-Image, Text-to-Video, Image-to-Video | | ||
| | `nvidia/Cosmos3-Super` | Text-to-Image, Text-to-Video, Image-to-Video | | ||
|
|
||
|
|
@@ -53,6 +54,7 @@ Models are auto-detected from the checkpoint directory. Diffusers-format models | |
| | **Wan 2.2** | Yes | Yes | Yes [^3] | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | | ||
| | **LTX-2** | Yes | Yes | Yes [^4] | Yes | Yes | Yes | No | No | Yes | Yes | Yes | Yes | No | No | | ||
| | **Qwen-Image** [^5] | Yes | Yes | No | No | No | Yes | No | Yes | Yes | Yes | Yes | Yes | No | No | | ||
| | **Qwen-Image-Edit-2511** | Yes | Yes | No | No | Yes | No | No | Yes | Yes | Yes | No | No | No | No | | ||
| | **Cosmos3** | Yes | Yes | No | No | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | No | | ||
|
|
||
| [^1]: FLUX models use embedded guidance and do not have a separate negative prompt path, so CFG parallelism is not applicable. | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| # 1-GPU Qwen-Image-Edit-2511 with selective NVFP4 dynamic quantization. | ||
| # Q/K/V projections remain in BF16 because they preserve accuracy | ||
| # Shared by offline examples (--visual_gen_args) and trtllm-serve. | ||
| quant_config: | ||
| quant_algo: NVFP4 | ||
| dynamic: true | ||
| ignore: | ||
| - "transformer_blocks.*.attn.to_q" | ||
| - "transformer_blocks.*.attn.to_k" | ||
| - "transformer_blocks.*.attn.to_v" | ||
| - "transformer_blocks.*.attn.add_q_proj" | ||
| - "transformer_blocks.*.attn.add_k_proj" | ||
| - "transformer_blocks.*.attn.add_v_proj" | ||
|
Comment on lines
+19
to
+28
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why these settings? When disclose such, we should have some docs to clarify the methodology.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I briefly mentioned in the comment above that Or we could use FP8 quant.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. modified to use FP8 quant config instead. |
||
| attention_config: | ||
| backend: VANILLA | ||
| parallel_config: | ||
| cfg_size: 1 | ||
| ulysses_size: 1 | ||
| cuda_graph_config: | ||
| enable: false | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| # Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| # 1-GPU Qwen-Image-Edit-2511 with FP8 blockwise dynamic quantization. | ||
| # Shared by offline examples (--visual_gen_args) and trtllm-serve. | ||
| quant_config: | ||
| quant_algo: FP8_BLOCK_SCALES | ||
| dynamic: true | ||
| attention_config: | ||
| backend: VANILLA | ||
| parallel_config: | ||
| cfg_size: 1 | ||
| ulysses_size: 1 | ||
| cuda_graph_config: | ||
| enable: false |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| # Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| """Qwen-Image-Edit-2511 image editing. | ||
|
|
||
| Usage: | ||
| python qwen_image_edit.py --image input.png --prompt "Make the image look like a watercolor painting" | ||
| python qwen_image_edit.py \ | ||
| --visual_gen_args ../configs/qwen-image-edit-2511-fp8-1gpu.yaml \ | ||
| --image input.png | ||
| """ | ||
|
|
||
| import argparse | ||
|
|
||
| from tensorrt_llm import VisualGen, VisualGenArgs | ||
|
|
||
|
|
||
| def parse_args() -> argparse.Namespace: | ||
| parser = argparse.ArgumentParser(description=__doc__) | ||
| parser.add_argument( | ||
| "--model", | ||
| default="Qwen/Qwen-Image-Edit-2511", | ||
| help="Hugging Face model id or local checkpoint path.", | ||
| ) | ||
| parser.add_argument( | ||
| "--visual_gen_args", | ||
| dest="visual_gen_args", | ||
| type=str, | ||
| default=None, | ||
| help="Path to YAML config (same as trtllm-serve --visual_gen_args)", | ||
| ) | ||
| parser.add_argument( | ||
| "--image", | ||
| nargs="+", | ||
|
Comment on lines
+44
to
+45
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This support is similar to Flux Kontext cc @karljang |
||
| required=True, | ||
| help="One or more input image paths or URLs.", | ||
| ) | ||
| parser.add_argument( | ||
| "--prompt", | ||
| default="Make the image look like a watercolor painting while preserving the main subject.", | ||
| help="Text edit instruction.", | ||
| ) | ||
| parser.add_argument( | ||
| "--output_path", | ||
| default="qwen_image_edit_output.png", | ||
| help="Edited image output path.", | ||
| ) | ||
| return parser.parse_args() | ||
|
|
||
|
|
||
| def main() -> None: | ||
| args = parse_args() | ||
| extra_args = VisualGenArgs.from_yaml(args.visual_gen_args) if args.visual_gen_args else None | ||
| visual_gen = VisualGen(model=args.model, args=extra_args) | ||
| params = visual_gen.default_params | ||
| params.image = args.image if len(args.image) > 1 else args.image[0] | ||
| output = visual_gen.generate(inputs=args.prompt, params=params) | ||
| saved = output.save(args.output_path) | ||
| print(f"Saved edited image to {saved}") | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| main() | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure if we should name like text+image-to-image, and also some other models such as Flux also supports text+image-to-image.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe Image Editing (text+image-to-image)?