VAAPI encoder crashes on AMD Polaris (RX 590)

[aningQwQ]

Is there an existing issue for this?

• I have searched the existing issues

Is your issue described in the documentation?

• I have read the documentation

Is your issue present in the latest beta/pre-release?

This issue is present in the latest pre-release

Describe the Bug

When using vaapi + kwin, starting streaming crashes immediately on AMD Polaris series graphics cards (RX 590 GME).
Configure vaapi encoder, kwin capture
Start Sunshine streaming
The entire software crashes.

Expected Behavior

The VAAPI encoder should work normally.

Additional Context

Changing n_dmabuf_infos > 0 at pipewire.cpp:883 to 0 forces the va_ram_t path (CPU copy path), and VAAPI encoding works normally again without crashing.
https://github.com/aningQwQ/Sunshine/tree/pr1

Host Operating System

Linux

Operating System Version

fedora44-kde-7.0.12-201.fc44.x86_64 (64 位)

Architecture

amd64/x86_64

Package

Linux - Fedora Copr

GPU Type

AMD

GPU Model

RX 590

GPU Driver/Mesa Version

Mesa 26.0.8

Capture Method

Kwin

Apps

Log output

(Could not obtain due to webui crash, see online logs below)

Online logs

https://gist.github.com/aningQwQ/9f7cdb5834603d4ff8561dae9954ec71

[aningQwQ]

After applying the workaround that sets n_dmabuf_infos = 0 (forcing the CPU‑copy path va_ram_t) in pipewire.cpp:883, VAAPI encoding no longer crashes initially. However, a new crash occurs when switching the display source that Sunshine is actively capturing, specifically by using the hotkey combination Ctrl+Alt+Shift+F1 / F2 to toggle between displays on the fly.
https://gist.github.com/aningQwQ/ada0f3a3108cfb1eacf15171264e1d27

[ReenigneArcher]

#5249

[Kishi85]

After applying the workaround that sets n_dmabuf_infos = 0 (forcing the CPU‑copy path va_ram_t) in pipewire.cpp:883, VAAPI encoding no longer crashes initially. However, a new crash occurs when switching the display source that Sunshine is actively capturing, specifically by using the hotkey combination Ctrl+Alt+Shift+F1 / F2 to toggle between displays on the fly. https://gist.github.com/aningQwQ/ada0f3a3108cfb1eacf15171264e1d27

See #4943

For Fixes see:

#5249

Likely also related: #5257

[aningQwQ]

I would like to add that the issue only occurs with kwin + vaapi; kms does not have this problem.

[Kishi85]

I would like to add that the issue only occurs with kwin + vaapi; kms does not have this problem.

The freeze issue (#5249) can occur on pipewire-based capture methods (likely some racecondition that gets the capture stuck) so it can happen on both kwingrab and portalgrab. #5249 contains a practical workaround, the root cause of this issue is so far still unknown.

The segfault issue (#4943, #5257) can occur with any capture method (including KMS) but is more prevalent on pipewire-based capture methods due to them being event based hence triggering the root cause (trying to flush a never properly started encoder) much easier than others. To trigger this issue with KMS you'd have to force it by button mashing the display switch really quickly to get an encoder shutdown with trying frame flush at frame number 0.

[aningQwQ]

The freeze issue (#5249) can occur on pipewire-based capture methods (likely some racecondition that gets the capture stuck) so it can happen on both kwingrab and portalgrab. #5249 contains a practical workaround, the root cause of this issue is so far still unknown.

The segfault issue (#4943， #5257) can occur with any capture method (including KMS) but is more prevalent on pipewire-based capture methods due to them being event based hence triggering the root cause (trying to flush a never properly started encoder) much easier than others. To trigger this issue with KMS you'd have to force it by button mashing the display switch really quickly to get an encoder shutdown with trying frame flush at frame number 0.

Thanks for the links, but I think there might be a slight misunderstanding regarding the scope of this issue.

To clarify, the primary issue I reported is that VAAPI + Kwin (n_dmabuf_infos > 0) crashes immediately during encoder initialization on my RX 590 (Polaris). The encoder doesn't even get a chance to start streaming.

The workaround I mentioned (forcing n_dmabuf_infos = 0 to use the va_ram_t CPU copy path) was merely a temporary bypass to make the software launchable. It is not a fix, and it introduced the secondary crash when switching displays (which you correctly linked to #4943 / #5249).

However, that secondary crash is a separate symptom that only appears after applying the bypass. My main concern right now is the root cause of the initial crash

https://gist.github.com/aningQwQ/9f7cdb5834603d4ff8561dae9954ec71

I also want to add a clarification: when I previously said “this issue only occurs with the kwin + vaapi combination; kms does not have this problem”, that statement refers specifically to the immediate crash during VAAPI + KWin initialization, not to the other crashes or freezes. Sorry for any misunderstanding.

[Kishi85]

No worries, also no misunderstandings here. Your initial issue seems to be a crash involving avcodec_send_frame if I'm reading the dump correctly. Does applying #5257 fix it by any chance? If not this issue is likely a bug in FFmpeg that we have to work around (like #5257 does) or that has to be fixed upstream. I'm no expert on this so my capabilities in fixing this are unfortunately a bit limited.

Could you also post the journalctl --user -u app-dev.lizardbyte.app.Sunshine.service output for the crash? So we have the full sunshine log up to the coredump and might see better where exactly it is happening.