Skip to content

virgl: capture host-rendered virtio-gpu scanouts via GPU EGL readback#20

Merged
fxd0h merged 2 commits into
mainfrom
feat/virgl-egl-readback
Jun 26, 2026
Merged

virgl: capture host-rendered virtio-gpu scanouts via GPU EGL readback#20
fxd0h merged 2 commits into
mainfrom
feat/virgl-egl-readback

Conversation

@fxd0h

@fxd0h fxd0h commented Jun 26, 2026

Copy link
Copy Markdown
Owner

Fixes #15: a virgl-rendered virtio-gpu scanout used to capture as an all-black frame.

Why it was black

A virgl (virtio-gpu 3D) scanout is rendered on the host GPU. The KMS framebuffer handle the guest gets from GetFB2 points at a host-side resource, so any guest CPU read of it (dumb-map V2, or a CPU mmap of the exported DMA-BUF) comes back black, and TRANSFER_FROM_HOST does not populate a guest-readable buffer for it. The pixels are reachable on the guest GPU: importing the DMA-BUF as an EGLImage and glReadPixels() resolves the real host-rendered content (the same backend already used to detile tiled buffers).

Change

  • Helper: when virtio 3D features are present (VIRTGPU_PARAM_3D_FEATURES != 0), export the scanout as a DMA-BUF (V3) and tag the frame FLAG_VIRGL. The export condition is simplified to any virtio, or a tiled non-virtio framebuffer. Plain virtio keeps the existing zero-copy direct-map path (no FLAG_VIRGL).
  • Library: a force_egl flag is threaded into gpu_auto_process(). On HELPER_FLAG_VIRGL the linear-modifier fast path is bypassed and the frame is run through the EGL import + glReadPixels path. Every other path passes force_egl=0 and is unchanged.

Cost on virgl: one GPU textured-quad draw + glReadPixels per frame, the same as the Intel/Nvidia detile path. Plain virtio is untouched (still direct-map, no EGL).

Verification (real virgl VM)

QEMU virtio-gpu-gl-pci + egl-headless, Ubuntu guest, weston --renderer=gl reporting GL renderer: virgl (Mesa Intel Arc):

  • A non-root capture through the helper now returns the live weston desktop (verified visually, incl. the live panel clock), where it was previously black.
  • Helper logs virtio 3D features=1 -> DMA-BUF + GPU readback (virgl); library logs EGL convert: ret=0.

No-regression checks:

  • Plain virtio-gpu (weston --renderer=pixman): still the zero-copy direct-map path (helper logs zero-copy DMA-BUF, no EGL convert), capture correct.
  • Intel i915 (dual-4K, helper V3): unchanged, EGL detile ret=0, correct frame.
  • release+werror, debug+asan+ubsan, buildtype=plain+werror, meson test 5/5, cppcheck clean.

Note: this fixes the helper capture path (the supported unprivileged deployment). Direct/root-mode virgl capture (the library opening virtio-gpu itself) would want the same EGL-readback treatment on its own paths and is left as a follow-up.

Closes #15

A virgl (virtio-gpu 3D) scanout is rendered on the host GPU. The KMS
framebuffer handle the guest gets from GetFB2 points at a host-side
resource, so any guest CPU read of it (dumb-map V2 or a CPU mmap of the
exported DMA-BUF) comes back black, and TRANSFER_FROM_HOST does not populate
a guest-readable buffer for it. The pixels are reachable on the guest GPU,
though: importing the DMA-BUF as an EGLImage and glReadPixels() resolves the
real host-rendered content.

Helper: when virtio 3D features are present, export the scanout as a DMA-BUF
(V3, as for plain virtio) and additionally tag the frame FLAG_VIRGL. The
export condition is simplified to 'any virtio or a tiled non-virtio
framebuffer'; plain virtio keeps the existing zero-copy direct-map path
(no FLAG_VIRGL).

Library: thread a force_egl flag into gpu_auto_process(). On FLAG_VIRGL the
linear-modifier fast path is bypassed and the frame is run through the EGL
import + glReadPixels path (the same backend used to detile tiled buffers),
which reads the host-rendered pixels. Plain virtio and all other paths pass
force_egl=0 and are unchanged.

Verified end to end in a virgl VM (qemu virtio-gpu-gl + egl-headless,
weston --renderer=gl -> 'GL renderer: virgl'): a non-root capture through
the helper now returns the live weston desktop (helper logs 'DMA-BUF + GPU
readback (virgl)', library logs EGL convert ret=0) instead of a black frame.
Re-checked plain virtio-gpu (weston --renderer=pixman): still the zero-copy
direct-map path (helper logs 'zero-copy DMA-BUF', no EGL convert), no
regression.

Closes #15
@coderabbitai

coderabbitai Bot commented Jun 26, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Summary by CodeRabbit

  • New Features

    • Added support for virtio/virgl scanouts, enabling GPU-based readback for DMA-BUF frames when CPU mapping is unavailable or would produce black output.
    • Improved DMA-BUF export metadata so the capture pipeline can choose the correct GPU processing path.
  • Bug Fixes

    • Fixed virgl-related capture failures by ensuring the processing pipeline doesn’t exit early when CPU data is missing.
    • Updated virgl handling to fail closed when GPU readback/conversion does not produce pixels, avoiding misleading black frames.

Walkthrough

The PR adds virgl-specific helper flags, marks virgl DMA-BUF exports, and updates framebuffer processing so virgl scanouts can force EGL-based readback even when CPU mapping is unavailable.

Changes

Virgl scanout capture flow

Layer / File(s) Summary
Virgl flag bits
bindings/rust/libdrmtap-sys/csrc/drmtap_internal.h, src/drmtap_internal.h, helper/drmtap-helper.c, bindings/rust/libdrtap-sys/csrc/drmtap-helper.c
Adds HELPER_FLAG_VIRGL and FLAG_VIRGL definitions plus the associated virgl scanout comment.
Virtio virgl detection
helper/drmtap-helper.c, bindings/rust/libdrmtap-sys/csrc/drmtap-helper.c
Replaces the old virtio plain/virgl split with detection state that probes virtio 3D features and sets FLAG_VIRGL on exported DMA-BUFs.
Force EGL on virgl frames
bindings/rust/libdrmtap-sys/csrc/drm_grab.c, src/drm_grab.c
gpu_auto_process gains force_egl, helper V3 paths derive is_virgl from HELPER_FLAG_VIRGL, and virgl frames call the processor with EGL forcing even when data is missing.
Remaining auto-process callers
bindings/rust/libdrmtap-sys/csrc/drm_grab.c, src/drm_grab.c
The V2, standard mapped, and fast-grab call sites are updated to pass force_egl=0 to the new gpu_auto_process signature.

Sequence Diagram(s)

sequenceDiagram
  participant helper as helper/drmtap-helper.c
  participant grab as src/drm_grab.c
  participant auto as gpu_auto_process
  participant egl as EGL import/convert path

  helper->>grab: send FLAG_VIRGL in helper V3 metadata
  grab->>auto: pass force_egl=is_virgl for V3 DMA-BUF frames
  auto->>egl: run even when data is NULL or modifier is linear
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Poem

I twitched my nose at virgl’s glow,
and hopped where EGL pixels flow.
No black frame now for rabbit eyes,
just DMA-BUF moonlight in the skies.
Thump, thump—happy scanouts! 🐇

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: virgl scanouts now use GPU EGL readback instead of black CPU-mapped frames.
Description check ✅ Passed The description is directly about the virgl black-frame fix and matches the implemented helper and library changes.
Linked Issues check ✅ Passed The PR fixes virgl scanout capture and preserves plain virtio behavior, satisfying the issue's core requirement.
Out of Scope Changes check ✅ Passed The changes stay focused on virgl scanout capture, error handling, and related metadata with no clear unrelated additions.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/virgl-egl-readback

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (4)
bindings/rust/libdrmtap-sys/csrc/drm_grab.c (1)

484-499: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Make forced EGL readback fail closed and propagate the error.

Line 499 can enter gpu_auto_process() with data == NULL. If EGL is unavailable or drmtap_gpu_egl_convert() fails, the current path can still return success with no usable pixels. Return an error when force_egl cannot be satisfied, and check it at the helper V3 call sites.

Proposed direction
-                if (do_mmap) {
-                    gpu_auto_process(ctx, mapped, frame, is_virgl);
+                if (do_mmap) {
+                    ret = gpu_auto_process(ctx, mapped, frame, is_virgl);
+                    if (ret != 0) {
+                        drmtap_frame_release(ctx, frame);
+                        drmModeFreeFB2(fb2);
+                        return ret;
+                    }
                 }
@@
-                if (do_mmap) {
-                    gpu_auto_process(ctx, NULL, frame, is_virgl);
+                if (do_mmap) {
+                    ret = gpu_auto_process(ctx, NULL, frame, is_virgl);
+                    if (ret != 0) {
+                        drmtap_frame_release(ctx, frame);
+                        drmModeFreeFB2(fb2);
+                        return ret;
+                    }
                 }
@@
-    if (!data && !force_egl) return 0;
+    if (!data && !force_egl) {
+        return 0;
+    }
@@
         if (ret == 0 && egl_data) {
@@
             return 0;
         }
+        if (force_egl) {
+            drmtap_set_error(ctx, "virgl EGL readback failed: %d", ret);
+            return ret < 0 ? ret : -EIO;
+        }
         drmtap_debug_log(ctx, "auto-process: EGL failed (%d), trying CPU", ret);
     }
+#else
+    if (force_egl) {
+        drmtap_set_error(ctx, "virgl EGL readback requested but EGL support is unavailable");
+        return -ENOTSUP;
+    }
 `#endif`
+
+    if (force_egl) {
+        drmtap_set_error(ctx, "virgl EGL readback requested but EGL is unavailable");
+        return -ENOTSUP;
+    }
+    if (!data) {
+        return 0;
+    }

Also applies to: 627-717

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@bindings/rust/libdrmtap-sys/csrc/drm_grab.c` around lines 484 - 499, The
helper V3 fallback path in drm_grab.c can call gpu_auto_process() with NULL data
and still continue as success, which leaves force_egl without usable pixels.
Update the gpu_auto_process/drmtap_gpu_egl_convert flow so that when force_egl
is requested and EGL conversion is unavailable or fails, an error is returned
instead of falling through, and make the helper V3 call sites that reach this
path check and propagate that failure. Use the gpu_auto_process(),
drmtap_gpu_egl_convert(), and helper V3 logic to locate the affected branches.
bindings/rust/libdrmtap-sys/csrc/drmtap-helper.c (1)

594-611: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Do not fall back to CPU pixels for confirmed virgl exports.

Line 600 should only tag virtio frames, and Line 608 should fail closed for confirmed virgl. Falling through to the dumb-map pixel path can return the same black frame this PR is fixing.

Proposed fix
-            if (virtio_virgl) {
+            if (is_virtio && virtio_virgl) {
                 meta.flags |= FLAG_VIRGL;  /* parent must GPU-read it back */
             }
@@
-        /* Export failed — fall through to the dumb-map pixel path. */
+        if (is_virtio && virtio_virgl) {
+            return send_error_errno(sock, "virgl DMA-BUF export failed");
+        }
+        /* Export failed — fall through to the dumb-map pixel path. */
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@bindings/rust/libdrmtap-sys/csrc/drmtap-helper.c` around lines 594 - 611, The
fallback in drmtap-helper’s DMA-buf export path is too permissive for confirmed
virgl cases. Update the conditional around the drmPrimeHandleToFD export in
drmtap-helper so that only virtio frames are tagged for DMABUF/FLAG_VIRGL, and
if a confirmed virgl export fails, return an error or fail closed instead of
falling through to the CPU pixel mapping path. Keep the existing DMA-buf success
handling in the same block, but guard the fallback in the code path around the
gem_handle export logic and the send_fd/prime_fd handling.
helper/drmtap-helper.c (1)

594-611: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Do not fall back to CPU pixels for confirmed virgl exports.

Line 600 should only tag virtio frames, and Line 608 should fail closed for confirmed virgl. Falling through to the dumb-map pixel path can return the same black frame this PR is fixing.

Proposed fix
-            if (virtio_virgl) {
+            if (is_virtio && virtio_virgl) {
                 meta.flags |= FLAG_VIRGL;  /* parent must GPU-read it back */
             }
@@
-        /* Export failed — fall through to the dumb-map pixel path. */
+        if (is_virtio && virtio_virgl) {
+            return send_error_errno(sock, "virgl DMA-BUF export failed");
+        }
+        /* Export failed — fall through to the dumb-map pixel path. */
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@helper/drmtap-helper.c` around lines 594 - 611, The fallback in the drmtap
send path is too broad: the drmPrimeHandleToFD branch in drmtap-helper.c
currently falls through to the CPU pixel path even for confirmed virgl exports.
Update the logic around drmPrimeHandleToFD, meta.flags, and the virtio_virgl
handling so only true virtio frames are tagged, and for confirmed virgl cases
fail closed instead of continuing to the dumb-map pixel path. Keep the existing
send_fd path for successful exports, but prevent the fallback for virgl-specific
frames and preserve the current pixel fallback only for non-virgl failures.
src/drm_grab.c (1)

484-499: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Make forced EGL readback fail closed and propagate the error.

Line 499 can enter gpu_auto_process() with data == NULL. If EGL is unavailable or drmtap_gpu_egl_convert() fails, the current path can still return success with no usable pixels. Return an error when force_egl cannot be satisfied, and check it at the helper V3 call sites.

Proposed direction
-                if (do_mmap) {
-                    gpu_auto_process(ctx, mapped, frame, is_virgl);
+                if (do_mmap) {
+                    ret = gpu_auto_process(ctx, mapped, frame, is_virgl);
+                    if (ret != 0) {
+                        drmtap_frame_release(ctx, frame);
+                        drmModeFreeFB2(fb2);
+                        return ret;
+                    }
                 }
@@
-                if (do_mmap) {
-                    gpu_auto_process(ctx, NULL, frame, is_virgl);
+                if (do_mmap) {
+                    ret = gpu_auto_process(ctx, NULL, frame, is_virgl);
+                    if (ret != 0) {
+                        drmtap_frame_release(ctx, frame);
+                        drmModeFreeFB2(fb2);
+                        return ret;
+                    }
                 }
@@
-    if (!data && !force_egl) return 0;
+    if (!data && !force_egl) {
+        return 0;
+    }
@@
         if (ret == 0 && egl_data) {
@@
             return 0;
         }
+        if (force_egl) {
+            drmtap_set_error(ctx, "virgl EGL readback failed: %d", ret);
+            return ret < 0 ? ret : -EIO;
+        }
         drmtap_debug_log(ctx, "auto-process: EGL failed (%d), trying CPU", ret);
     }
+#else
+    if (force_egl) {
+        drmtap_set_error(ctx, "virgl EGL readback requested but EGL support is unavailable");
+        return -ENOTSUP;
+    }
 `#endif`
+
+    if (force_egl) {
+        drmtap_set_error(ctx, "virgl EGL readback requested but EGL is unavailable");
+        return -ENOTSUP;
+    }
+    if (!data) {
+        return 0;
+    }

Also applies to: 627-717

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/drm_grab.c` around lines 484 - 499, The helper V3 path can call
gpu_auto_process() with NULL data and still succeed when force_egl is requested,
which leaves no usable pixels; update the error handling so forced EGL readback
fails closed. In the gpu_auto_process()/drmtap_gpu_egl_convert() flow, propagate
an error whenever EGL is unavailable or the conversion fails under force_egl,
instead of falling back to success. Then update the helper V3 call sites that
currently pass NULL on mmap failure and the related flow in the 627-717 range to
check and return that error immediately.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@bindings/rust/libdrmtap-sys/csrc/drmtap-helper.c`:
- Around line 529-538: Update the stale scanout comment in drmtap-helper.c so it
matches the current virtio detection logic: replace references to the removed
virtio_plain state with the actual virtio_detected and virtio_virgl behavior
used by the scanout handling in the relevant helper path. Keep the explanation
aligned with the existing DMA-BUF export and virgl/GPU readback flow, and ensure
the fallback wording reflects how the code now treats undetected virtio cases.

In `@helper/drmtap-helper.c`:
- Around line 529-538: Update the stale comment in drmtap helper to match the
current virtio-gpu detection logic: the code now uses virtio_detected and
virtio_virgl instead of virtio_plain. Edit the explanatory block near the
scanout handling logic so it describes the current plain vs virgl behavior using
the actual symbols in the code, and remove any reference to the removed
virtio_plain state.

---

Outside diff comments:
In `@bindings/rust/libdrmtap-sys/csrc/drm_grab.c`:
- Around line 484-499: The helper V3 fallback path in drm_grab.c can call
gpu_auto_process() with NULL data and still continue as success, which leaves
force_egl without usable pixels. Update the
gpu_auto_process/drmtap_gpu_egl_convert flow so that when force_egl is requested
and EGL conversion is unavailable or fails, an error is returned instead of
falling through, and make the helper V3 call sites that reach this path check
and propagate that failure. Use the gpu_auto_process(),
drmtap_gpu_egl_convert(), and helper V3 logic to locate the affected branches.

In `@bindings/rust/libdrmtap-sys/csrc/drmtap-helper.c`:
- Around line 594-611: The fallback in drmtap-helper’s DMA-buf export path is
too permissive for confirmed virgl cases. Update the conditional around the
drmPrimeHandleToFD export in drmtap-helper so that only virtio frames are tagged
for DMABUF/FLAG_VIRGL, and if a confirmed virgl export fails, return an error or
fail closed instead of falling through to the CPU pixel mapping path. Keep the
existing DMA-buf success handling in the same block, but guard the fallback in
the code path around the gem_handle export logic and the send_fd/prime_fd
handling.

In `@helper/drmtap-helper.c`:
- Around line 594-611: The fallback in the drmtap send path is too broad: the
drmPrimeHandleToFD branch in drmtap-helper.c currently falls through to the CPU
pixel path even for confirmed virgl exports. Update the logic around
drmPrimeHandleToFD, meta.flags, and the virtio_virgl handling so only true
virtio frames are tagged, and for confirmed virgl cases fail closed instead of
continuing to the dumb-map pixel path. Keep the existing send_fd path for
successful exports, but prevent the fallback for virgl-specific frames and
preserve the current pixel fallback only for non-virgl failures.

In `@src/drm_grab.c`:
- Around line 484-499: The helper V3 path can call gpu_auto_process() with NULL
data and still succeed when force_egl is requested, which leaves no usable
pixels; update the error handling so forced EGL readback fails closed. In the
gpu_auto_process()/drmtap_gpu_egl_convert() flow, propagate an error whenever
EGL is unavailable or the conversion fails under force_egl, instead of falling
back to success. Then update the helper V3 call sites that currently pass NULL
on mmap failure and the related flow in the 627-717 range to check and return
that error immediately.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 35dd3e15-8315-4c80-b618-4abdfcee8ddc

📥 Commits

Reviewing files that changed from the base of the PR and between c5041f4 and b6de568.

📒 Files selected for processing (6)
  • bindings/rust/libdrmtap-sys/csrc/drm_grab.c
  • bindings/rust/libdrmtap-sys/csrc/drmtap-helper.c
  • bindings/rust/libdrmtap-sys/csrc/drmtap_internal.h
  • helper/drmtap-helper.c
  • src/drm_grab.c
  • src/drmtap_internal.h
📜 Review details
🧰 Additional context used
📓 Path-based instructions (3)
**/*.h

📄 CodeRabbit inference engine (AGENTS.md)

**/*.h: Use 4-space indentation, never tabs, and 1TBS brace style in C header files.
Name C functions and variables in snake_case, with public API functions prefixed drmtap_.
Name macros and constants in UPPER_SNAKE_CASE, prefixed DRMTAP_ when project-specific.
Use snake_case_t for internal types and drmtap_* for public types.
Use #ifndef DRMTAP_MODULE_H / #define DRMTAP_MODULE_H style header guards.
Public API functions in include/drmtap.h must use Doxygen comments describing purpose, parameters, return values, and error codes.

Files:

  • bindings/rust/libdrmtap-sys/csrc/drmtap_internal.h
  • src/drmtap_internal.h
**/*.{c,h}

📄 CodeRabbit inference engine (AGENTS.md)

**/*.{c,h}: Keep lines to a 100-character soft limit and 120-character hard limit.
Use // for single-line comments and /* */ for multi-line comments.
Always check malloc returns and free resources in cleanup paths.

Files:

  • bindings/rust/libdrmtap-sys/csrc/drmtap_internal.h
  • src/drmtap_internal.h
  • helper/drmtap-helper.c
  • bindings/rust/libdrmtap-sys/csrc/drmtap-helper.c
  • src/drm_grab.c
  • bindings/rust/libdrmtap-sys/csrc/drm_grab.c
src/**/*.c

📄 CodeRabbit inference engine (AGENTS.md)

src/**/*.c: Use 4-space indentation, never tabs, and 1TBS brace style in C source files.
Name C functions and variables in snake_case, with public API functions prefixed drmtap_.
Name macros and constants in UPPER_SNAKE_CASE, prefixed DRMTAP_ when project-specific.
Use snake_case_t for internal types and drmtap_* for public types.
Return negative errno values for errors and never abort.
Every .c file must start with the standard libdrmtap copyright/license header block, followed by a Doxygen @file comment.
Organize C source files as includes first, then private types/constants, then static functions, then public functions in header-declaration order.
Use simple // comments for internal static functions instead of full Doxygen blocks.

Files:

  • src/drm_grab.c
🪛 Clang (14.0.6)
helper/drmtap-helper.c

[warning] 545-545: variable name 'gp' is too short, expected at least 3 characters

(readability-identifier-length)


[warning] 596-596: variable 'prime_ret' is not initialized

(cppcoreguidelines-init-variables)

bindings/rust/libdrmtap-sys/csrc/drmtap-helper.c

[warning] 545-545: variable name 'gp' is too short, expected at least 3 characters

(readability-identifier-length)


[warning] 596-596: variable 'prime_ret' is not initialized

(cppcoreguidelines-init-variables)

src/drm_grab.c

[warning] 633-633: statement should be inside braces

(readability-braces-around-statements)

bindings/rust/libdrmtap-sys/csrc/drm_grab.c

[warning] 633-633: statement should be inside braces

(readability-braces-around-statements)

🔇 Additional comments (6)
bindings/rust/libdrmtap-sys/csrc/drmtap-helper.c (1)

129-132: LGTM!

helper/drmtap-helper.c (1)

129-132: LGTM!

bindings/rust/libdrmtap-sys/csrc/drmtap_internal.h (1)

122-124: LGTM!

src/drmtap_internal.h (1)

122-124: LGTM!

bindings/rust/libdrmtap-sys/csrc/drm_grab.c (1)

62-62: LGTM!

Also applies to: 528-528, 605-605, 974-974, 1069-1069

src/drm_grab.c (1)

62-62: LGTM!

Also applies to: 528-528, 605-605, 974-974, 1069-1069

Comment thread bindings/rust/libdrmtap-sys/csrc/drmtap-helper.c
Comment thread helper/drmtap-helper.c
Addresses review feedback. A virgl scanout has no CPU-readable pixels, so on
every path where the GPU readback cannot happen we now return an error
instead of emitting an all-black frame as success:

- Helper: if drmPrimeHandleToFD fails for a confirmed-virgl frame, send_error
  instead of falling through to the dumb-map pixel path (which is black).
- Library: gpu_auto_process() returns -ENOTSUP when force_egl is set but the
  EGL path did not produce pixels (EGL unavailable or convert failed), and
  the helper-V3 call sites propagate that error for virgl frames (non-virgl
  frames keep the prior best-effort behaviour).

Also fixes a stale comment that still referenced the removed virtio_plain
variable. Happy path is unchanged: a successful EGL readback returns before
these checks, as verified in the virgl VM.
@fxd0h

fxd0h commented Jun 26, 2026

Copy link
Copy Markdown
Owner Author

Good calls, all three addressed in the latest push:

  • Helper fail-closed: if drmPrimeHandleToFD fails for a confirmed-virgl frame, the helper now send_error()s instead of falling through to the dumb-map path (which would send a black frame for a host-side scanout).
  • Library fail-closed: gpu_auto_process() returns -ENOTSUP when force_egl is set but EGL didn't produce pixels (unavailable or convert failed), and the helper-V3 call sites now propagate that for virgl frames. Non-virgl frames keep the prior best-effort behaviour (return ignored), so tiled-EGL fallbacks are unchanged.
  • Fixed the stale virtio_plain comment.

The happy path is untouched (a successful EGL readback returns before these checks), so the virgl VM capture still works. release/asan/plain + meson test 5/5 + cppcheck all clean.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
bindings/rust/libdrmtap-sys/csrc/drm_grab.c (1)

455-510: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Clean up helper V3 resources before returning the virgl error.

By Line 509, dmabuf_fd, priv, and possibly mapped are owned by frame; returning pret on a failed grab skips normal frame release and leaks those resources. Clean up the partially initialized frame before returning. As per coding guidelines, **/*.{c,h} files must “Always check malloc returns and free resources in cleanup paths.”

Proposed cleanup
             if (is_virgl && pret != 0) {
+                if (priv->sync_started) {
+                    dmabuf_sync_end(dmabuf_fd);
+                }
+                if (priv->mapped != MAP_FAILED) {
+                    munmap(priv->mapped, priv->mapped_size);
+                }
+                close(dmabuf_fd);
+                free(priv);
+                frame->data = NULL;
+                frame->dma_buf_fd = -1;
+                frame->_priv = NULL;
                 return pret;
             }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@bindings/rust/libdrmtap-sys/csrc/drm_grab.c` around lines 455 - 510, The
virgl error return path in the DMA-BUF grab flow exits before the frame-owned
resources are released, leaving the partially initialized `frame`, `priv`, and
`dmabuf_fd`/`mapped` state leaked. Update the `drm_grab` helper V3 cleanup path
so the failed `is_virgl && pret != 0` branch runs the same release/cleanup logic
used for normal frame teardown before returning the error, using the existing
`priv`, `frame`, and `dmabuf_fd` ownership handling.

Source: Coding guidelines

src/drm_grab.c (1)

455-510: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Clean up helper V3 resources before returning the virgl error.

By Line 509, dmabuf_fd, priv, and possibly mapped are owned by frame; returning pret on a failed grab skips normal frame release and leaks those resources. Clean up the partially initialized frame before returning. As per coding guidelines, **/*.{c,h} files must “Always check malloc returns and free resources in cleanup paths.”

Proposed cleanup
             if (is_virgl && pret != 0) {
+                if (priv->sync_started) {
+                    dmabuf_sync_end(dmabuf_fd);
+                }
+                if (priv->mapped != MAP_FAILED) {
+                    munmap(priv->mapped, priv->mapped_size);
+                }
+                close(dmabuf_fd);
+                free(priv);
+                frame->data = NULL;
+                frame->dma_buf_fd = -1;
+                frame->_priv = NULL;
                 return pret;
             }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/drm_grab.c` around lines 455 - 510, The virgl error return path in the
helper V3 grab flow skips the normal frame cleanup, leaving resources owned
through the frame state leaked. Before returning the `pret` failure from this
block, make sure the partially initialized frame is torn down and its owned
resources are released, including the DMA-BUF fd, any `mapped`/`priv->mapped`
mapping, and the `priv` state. Update the cleanup path around the
`gpu_auto_process`, `dmabuf_sync_start`, and `frame->_priv` setup so failed
virgl grabs exit through a release path rather than returning immediately.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@bindings/rust/libdrmtap-sys/csrc/drm_grab.c`:
- Around line 455-510: The virgl error return path in the DMA-BUF grab flow
exits before the frame-owned resources are released, leaving the partially
initialized `frame`, `priv`, and `dmabuf_fd`/`mapped` state leaked. Update the
`drm_grab` helper V3 cleanup path so the failed `is_virgl && pret != 0` branch
runs the same release/cleanup logic used for normal frame teardown before
returning the error, using the existing `priv`, `frame`, and `dmabuf_fd`
ownership handling.

In `@src/drm_grab.c`:
- Around line 455-510: The virgl error return path in the helper V3 grab flow
skips the normal frame cleanup, leaving resources owned through the frame state
leaked. Before returning the `pret` failure from this block, make sure the
partially initialized frame is torn down and its owned resources are released,
including the DMA-BUF fd, any `mapped`/`priv->mapped` mapping, and the `priv`
state. Update the cleanup path around the `gpu_auto_process`,
`dmabuf_sync_start`, and `frame->_priv` setup so failed virgl grabs exit through
a release path rather than returning immediately.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 46cb5c77-3c07-4c5e-9f28-fd203c043d4c

📥 Commits

Reviewing files that changed from the base of the PR and between b6de568 and 408a723.

📒 Files selected for processing (4)
  • bindings/rust/libdrmtap-sys/csrc/drm_grab.c
  • bindings/rust/libdrmtap-sys/csrc/drmtap-helper.c
  • helper/drmtap-helper.c
  • src/drm_grab.c
📜 Review details
⏰ Context from checks skipped due to timeout. (1)
  • GitHub Check: Analyze (rust)
🧰 Additional context used
📓 Path-based instructions (2)
**/*.{c,h}

📄 CodeRabbit inference engine (AGENTS.md)

**/*.{c,h}: Keep lines to a 100-character soft limit and 120-character hard limit.
Use // for single-line comments and /* */ for multi-line comments.
Always check malloc returns and free resources in cleanup paths.

Files:

  • helper/drmtap-helper.c
  • bindings/rust/libdrmtap-sys/csrc/drmtap-helper.c
  • src/drm_grab.c
  • bindings/rust/libdrmtap-sys/csrc/drm_grab.c
src/**/*.c

📄 CodeRabbit inference engine (AGENTS.md)

src/**/*.c: Use 4-space indentation, never tabs, and 1TBS brace style in C source files.
Name C functions and variables in snake_case, with public API functions prefixed drmtap_.
Name macros and constants in UPPER_SNAKE_CASE, prefixed DRMTAP_ when project-specific.
Use snake_case_t for internal types and drmtap_* for public types.
Return negative errno values for errors and never abort.
Every .c file must start with the standard libdrmtap copyright/license header block, followed by a Doxygen @file comment.
Organize C source files as includes first, then private types/constants, then static functions, then public functions in header-declaration order.
Use simple // comments for internal static functions instead of full Doxygen blocks.

Files:

  • src/drm_grab.c
🪛 Clang (14.0.6)
helper/drmtap-helper.c

[note] 611-611: +2, including nesting penalty of 1, nesting level increased to 2

(clang)

bindings/rust/libdrmtap-sys/csrc/drmtap-helper.c

[note] 611-611: +2, including nesting penalty of 1, nesting level increased to 2

(clang)

src/drm_grab.c

[note] 509-509: +3, including nesting penalty of 2, nesting level increased to 3

(clang)


[note] 509-509: +1

(clang)

bindings/rust/libdrmtap-sys/csrc/drm_grab.c

[note] 509-509: +3, including nesting penalty of 2, nesting level increased to 3

(clang)


[note] 509-509: +1

(clang)

🔇 Additional comments (4)
bindings/rust/libdrmtap-sys/csrc/drmtap-helper.c (1)

537-555: LGTM!

Also applies to: 608-615

helper/drmtap-helper.c (1)

537-555: LGTM!

Also applies to: 608-615

bindings/rust/libdrmtap-sys/csrc/drm_grab.c (1)

536-536: LGTM!

Also applies to: 613-613, 635-647, 727-736

src/drm_grab.c (1)

536-536: LGTM!

Also applies to: 613-613, 635-647, 727-736

@fxd0h fxd0h merged commit 4ea9768 into main Jun 26, 2026
9 checks passed
@fxd0h fxd0h deleted the feat/virgl-egl-readback branch June 26, 2026 18:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

virgl-rendered virtio-gpu scanout captures as a black frame (host-side resource, no guest readback)

1 participant