Add Wayland backend, selectable via MLX_BACKEND#5
Open
boulon42 wants to merge 28 commits into
Open
Conversation
Mirrors the XCB backend's services (window management, key/mouse/expose hooks, mouse show/hide/position, flush) on top of wl_seat/xdg-shell, with libxkbcommon for keysym translation and Vulkan/VK_KHR_wayland_surface for rendering. Select it at build time with `make BACKEND=wayland`. Two capabilities have no Wayland equivalent and are documented as such: pointer warping (mlx_mouse_move) and server-side key autorepeat toggling. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Xwayland relies on this same still-in-testing protocol to emulate XWarpPointer, which is why mouse warping worked through XCB->Xwayland. Bind wp_pointer_warp_v1 when the compositor advertises it and the installed wayland-protocols package ships its XML; otherwise keep the previous documented failure (-1). Purely additive and optional, so XCB and older Wayland-protocols installs are unaffected. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…sing Previously a missing/unregistered wayland-protocols package surfaced as a confusing "No rule to make target '/stable/xdg-shell/xdg-shell.xml'" error, since pkg-config silently returned an empty pkgdatadir. Check for that upfront and point at the fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
configure.sh already reports a missing wayland-protocols pkg-config module; the Makefile shouldn't duplicate that responsibility with its own install instructions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Make resolves the full dependency graph before any recipe runs, so a missing wayland-protocols package surfaces as Make's own opaque error unless `make BACKEND=wayland config` (or ./configure.sh) is run first - same constraint as any classic ./configure && make workflow. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fedora suffixes dev packages with -devel; the plain "wayland-protocols" hint doesn't match any installable package there. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
clean previously derived its object-file list from $(OBJ), which is backend-dependent: a plain `make clean` never removed the other backend's stale .o files, which then looked up-to-date to Make (same source, same timestamp) on a later `make BACKEND=wayland` and were silently reused instead of being recompiled with the new -D flags. Also mark `config` .PHONY: without it, Make only re-runs configure.sh based on configure.sh's mtime versus a same-named target, which is fragile - configure.sh must always run and gate the rest of the build. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
$(OBJ): src/backend/mlx__wayland_xdg_shell_protocol.h expands $(OBJ) to a long list of real object filenames, making it the first rule in the file under BACKEND=wayland (all: is defined further down) - and Make's default goal is the target of the first rule when none is set. So a plain `make BACKEND=wayland` was silently building just the first object in that list (plus its now-added header prerequisites) instead of `all`, and reporting success once that tiny, accidental goal was reached - explaining the "stops after one file, exit 0" symptom. Set .DEFAULT_GOAL explicitly so this can't happen again regardless of rule ordering. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
<strings.h> (bzero et al.) is not the same header as <string.h> (strcmp, strlen, memcpy...). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Calling through the raw K&R-style int (*)() stored in win->hook[] is deprecated (and dropped in C23). Match the XCB backend's own pattern (mlx__xcb_loop.c): assign into a properly prototyped local function pointer per call shape before invoking it, instead of calling through the unprototyped storage type directly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
libwayland-client does NOT silently skip unset listener entries for opcodes valid at the bound protocol version - it logs "listener function for opcode N of wl_output is NULL" and aborts. wl_output is bound at version 2 here, which requires geometry/mode/done/scale all be present, even though only mode's payload (pixel size) is used. An earlier cleanup pass removed these based on an incorrect assumption about libwayland's dispatch behavior; restoring them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Plain xdg-shell has no title bar/borders; that's drawn either by the client (CSD, which mlx does not implement) or by the compositor via the xdg-decoration-unstable-v1 extension. Bind zxdg_decoration_manager_v1 optionally (same graceful-degradation pattern as pointer-warp-v1) and request server-side mode for each toplevel before its first commit, as required by the protocol. Compositors without the extension leave the window undecorated, same as before. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
wl_seat was bound in the registry global handler but its listener was
only attached later in mlx__wayland_seat_init(), called after cursor
init. The compositor can (and does) send capabilities as soon as the
seat is bound; libwayland-client silently discards events for objects
with no listener yet ("discarded wl_seat#N.capabilities(N)" in
WAYLAND_DEBUG=1 output), so that event was lost every time, wl->pointer
and wl->keyboard were never created, and no input ever reached mlx.
Move the wl_seat_add_listener() call into the registry global handler,
right where the seat is bound - same pattern already used for
wl_output and xdg_wm_base.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
To compare what surface formats/composite-alpha modes the Wayland vs XCB WSI report on the same machine, diagnosing a reported color washout on the Wayland backend. Will be reverted once diagnosed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
To confirm whether GNOME/Mutter is enforcing client-side decoration despite our server-side request (a known Mutter policy), rather than this being a bug in the binding. Will be reverted once diagnosed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…colors vkGetPhysicalDeviceSurfaceFormatsKHR can report the same VkFormat under several colorspaces when VK_EXT_swapchain_colorspace is exposed (seen on Wayland/GNOME: the same formats repeated under standard, extended sRGB, BT709, BT2020, and HDR10 ST.2084 colorspaces). The format-picking loop never filtered by colorspace and never stopped at the first match, so it kept overwriting format/color_space through the whole list and ended up on whatever the last matching entry was - in this case VK_COLOR_SPACE_HDR10_ST2084_EXT. Since mlx writes plain SDR bytes with no HDR/color management, the compositor then displayed them through the PQ transfer function, badly over-exposing the image. Restrict the picking loop to VK_COLOR_SPACE_SRGB_NONLINEAR_KHR (the only colorspace mlx is designed for) and stop at the first match. Not Wayland-specific code, but only observed to matter there so far. Also revert the temporary debug prints used to diagnose this and the window decoration question. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confirmed on GNOME/Mutter: no zxdg_decoration_manager_v1 global is advertised at all, so windows stay undecorated there - accepted as expected behavior for a minimal, toolkit-less Wayland client rather than something mlx should work around by drawing its own title bar. The xdg-decoration wiring itself stays, for compositors that do support it (e.g. KDE Plasma). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…decoration Real Wayland positioning is client-opaque: there is no protocol to query or set a window's absolute screen position, so two independent top-level windows can't be kept in sync by hand. The correct primitive for a DIY title bar is a wl_subsurface (rigidly attached to the main surface at a fixed offset, so it moves with it automatically) combined with xdg_toplevel_move(), which hands the whole interactive drag over to the compositor in one call - no position tracking needed on our side. mlx__wayland_wm.c implements this: a small solid-colour bar drawn once into a wl_shm buffer, with a close-button region. Pointer enter/leave/ motion/button events for the bar's surface are intercepted in mlx__wayland_seat.c before normal window dispatch (tracked via a new wl->pointer_surface field, since wl_pointer.motion/button don't repeat the surface from .enter). Wired up in two places: immediately at window creation when no xdg-decoration manager is advertised at all, and reactively from the decoration's configure event when the compositor grants client-side mode despite our server-side request (observed on GNOME/Mutter, which advertises no decoration manager in the first place). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ring Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
mlx_hook()'s raw event-number path is inherently backend-native (it forwards whatever number the caller passes straight to the backend), so code written against XCB's raw numbers - e.g. mlx_hook(win, 33, 0, ...) to catch WM_DELETE_WINDOW via XCB_CLIENT_MESSAGE - silently did nothing on the Wayland backend, which used its own compact 1-7 numbering instead. Reuse the exact xcb/xproto.h values (KEY_PRESS=2, KEY_RELEASE=3, BUTTON_PRESS=4, BUTTON_RELEASE=5, MOTION_NOTIFY=6, EXPOSE=12, CLIENT_MESSAGE=33) so existing XCB-oriented advanced hook code keeps working unchanged when switching backends. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confirmed the close-button hit detection was already correct; the reported non-firing was the event-number mismatch fixed in 77b3edf. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… button Corners: per-pixel alpha test against the corner circle, cut pixels outside it to fully transparent (ARGB8888, no opaque-region override so the compositor blends it). Only the top corners round; the bottom stays square where it meets the window content. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ut's font atlas Give mlx_font.c's font_atlas external linkage via a new mlx_font.h instead of textually #include-ing mlx_font.c (previously only done from mlx_string_put.c) - reusing the same ~90KB glyph atlas from two translation units this way avoids compiling it in twice. mlx_font.c is now a normal compiled unit, added to SRC_GENERIC. mlx__wayland_wm.c blits the glyphs directly into the title bar's wl_shm buffer as an alpha mask (same technique mlx_string_put uses via the GPU), tinted light grey. mlx__wayland_win_t now keeps a copy of the window title for this purpose, since it otherwise isn't retained anywhere after xdg_toplevel_set_title(). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
24px vs the font's actual 20px left 2px of dead vertical padding on each side; sizing the bar exactly to font_atlas.height renders the title text at native resolution with no padding/scaling artifacts. Also shrink the close button and its corner radius/X-mark margin to stay proportionate to the smaller bar. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Glyph 0 (mapped from ' ') isn't actually blank in mlx_font.c's atlas - it showed up as a small tick mark between words. Treat space as explicitly blank in the title bar's own drawing code instead of touching the shared atlas or mlx_string_put(), which weren't part of what needed fixing here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
mlx_string_put()'s GPU path blends via outColor = color * texture (the shader), which respects the texture's actual per-pixel alpha, including any partially-transparent edge/anti-aliasing pixels. The title bar's own blit only tested "alpha != 0 -> paint fully opaque", so any faint edge pixel got rendered as solid ink, thickening every stroke compared to the reference rendering. Blend against the existing background colour using the real alpha value instead. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Give the bar a couple pixels of vertical padding around the (still native-size, still alpha-blended) title text; scale the close button, corner radius and X-mark margin back up to match. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confirmed on this Fedora/GNOME setup: Mutter advertises neither pointer-warp-v1 nor grants it to regular clients, yet Xwayland successfully warps the pointer for XCB apps - it does so via zwp_locked_pointer_v1_set_cursor_position_hint() (pointer-constraints- unstable-v1), a much older and far more broadly deployed protocol (originally for FPS-style mouselook) than the newer staging pointer-warp-v1. Lock the pointer (oneshot lifetime), wait for the 'locked' event via a blocking round-trip (lock_pointer() is async, but mlx_mouse_move() is a synchronous call), set the cursor position hint, commit, then destroy the lock - "the compositor may warp the cursor position to the set cursor position hint" on unlock. Tried after pointer-warp-v1 and before giving up with the documented -1/unsupported fallback. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
src/backend/mlx__wayland_*, mirroring every service the XCB backend provides: window creation/destruction (xdg-shell), key/mouse/expose hooks, mouse show/hide/get-pos, key autorepeat toggling, and flush.MLX_BACKEND_WAYLANDmacro inmlx_config.h, toggled withmake BACKEND=wayland(default remains XCB, unchanged).VK_KHR_wayland_surfacebranch alongside the existing XCB one (src/gpu/mlx___vulkan_init.c,mlx___vulkan_window.c).XK_*macros in student code keep working unchanged). Mouse buttons are remapped from Linux evdev codes to the X11 1/2/3/4/5 convention.configure.shchecks the right set per backend.mlx.h:mlx_mouse_move()(no pointer-warp protocol, returns -1) and key-autorepeat toggling (Wayland never server-side auto-repeats key presses;mlx_key_hookfires on release only anyway, so behavior is unaffected).Test plan
make(default XCB) still builds and runs as before on a Linux/X11 hostmake BACKEND=waylandbuilds on a Linux host with a Wayland session (needs the deps above)test/mlxtestopens a window, responds to key/mouse/expose hooks, and closes cleanly under a Wayland compositorNot compiled/run here — built on macOS without Wayland/XCB dev headers available. Flagging for a real build+run pass on Linux before merge.
🤖 Generated with Claude Code