Very slow framebuffer with hyperv_fb on recent Windows hosts, especially in Gen2 VM

[clouds56]

I'm using manjaro gnome3. With linux version 4.19.
The hyperv_fb is much slower than default efifb. (you could see the buffer is rendered line by line when scrolling with hyperv_fb, but could hardly notice when using )
the mode of hyperv_fb is U:1152x864p-0 (8192kB)
after blacklist the hyperv_fb it is using hardware EFI VGA(told by /var/log/Xorg.0.log)
the mode of efifb is U:1024x768p-75 (3072kB)

I'm not sure what's wrong with hyperv_fb, and the situation goes even worse when set the resolution "video=hyperv_fb:1920x1080", I hadn't found a way to set mode of efifb to 1920x1080 so haven't test for that.

[clouds56]

Forget to mention, the windows 10 version is 18850.1000 and the VM is creating using Hyper-v gen 2.

[clouds56]

I checked the code of hyperv_fb.c and hv/vmbus_drv.c in the kernel, and could not find issues there on my own.
Would it possible caused by change of some part of code in Hyper-V (maybe related to the RemoteFX deprecated?)

[dcui]

Thanks for reporting the issue! Which Linux distribution are you using and are you using the built-in kernel in the distribution ? Is there a .iso from the distribution vendor's website? We'd like to create a VM from the .iso and try to reproduce the issue.

Can you please test Gen-1 VM?

[clouds56]

I'm using manjaro with gnome, and iso here.
When using with Gen-2 VM, please follow the instruction here before install:

# press Ctrl+Alt+F3 to switch to a new tty and login
sudo pacman -Sy
sudo pacman -S xf86-video-fbdev # install the fbdev package
# there's no need restart gdm (and should not)
# just switch back to tty1 using Ctrl+Alt+F1 and continue

[clouds56]

Gen-1 VM seems not to suffer from the issue in Xorg, it is using VESA (vesafb) instead of FBDEV,
When I switch to tty in Gen-1 VM (it would switch to hyperv_fb), it is still slow when scrolling in less.

[dcui]

I can reproduce the exact symptoms on latest Hyper-V build. It looks like recently something on the host side causes this issue. BTW, the issue can not reproduce on a old Windows Server RS2 host.

So far, please blacklist the hyperv_fb driver to work around the issue.

I'm going to report this issue to Hyper-V team, but I'm afraid it can not be resolved soon.

[marcinwiacek]

Any update in this topic ? I observe it on latest Windows 10 Pro (host) with Ubuntu (guest, in fact every version) but only when enable more than 1 CPU vCore for VM. Gen 1 and Gen 2 VMs.

RDC is not option for me, also using standard framebuffers is a little problematic and it would be good to see some progress here.

[dcui]

@marcinwiacek: Can you please share your host version? On the host, please press Win+R and then run "winver.exe", and you should see something like "Version XXXX (OS Build XXXXX.XXX)" . The meaning of the version numbers is explained here: https://en.wikipedia.org/wiki/Windows_10_version_history .

The slowness is introduced by recent host versions (we know RS1 and RS2 are good, and RS5 and newer are bad). Hyper-V team has been working on this, but so far a thorough fix is still not available.

At the same time, we (Linux team) are trying to mitigating the slowness by implementing on-demand framebuffer updates. We have some internal patches and are testing them. We have not finalized the patches yet, and the performance improvement may not be very big on recent hosts, before Hyper-V team fixes the hosts.

We'll keep the link updated once we make more progress.

[marcinwiacek]

10.0.18362.207

I was thinking about RDC (but not very good option like I said) + using vesafb / uvesafb or any other FB (but no luck with this). If you know any workaround, I will be more than happy to test -> like I said the only one option is small resolution or having one CPU core.

[dcui]

10.0.18362 is 19H1, which has the slowness issue, as I mentioned.

It looks you're saying the FB is not slow when the VM is configured with only 1 virtual CPU? We don't see this. In our tests, the FB in an SMP VM (i.e. more than 1 vCPU) is as slow as that in a 1-vCPU VM, when the VM runs on "recent" host builds, including RS5 and 19H1.

If you do need a GUI environment in a Linux VM, I suggest you run vnc server in the VM (which is fast, as it's based on TCP, not Hyper-V VMBus), e.g. https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-vnc-on-ubuntu-18-04. You need a vnc client (e.g. vnc viewer) to connect to the server.

[marcinwiacek]

I confirm - machine with one vCPU has got hyperv_fb working fast, two and more vCPU make it slow. Can it be connected with some Spectre/prediction patches ? And why other FB work always fast ?

I also suggest to make small test - please create VM with hyperv_fb in the Ubuntu guest under older host (Windows not affected by bug), then migrate host to latest version. I had some VM (unfortunately lost), which was working fine for me in this scenario (I'm sure now let's say in 90%).

vnc - if all will fail, I will have to use (thx)

[dcui]

I guess Spectre/prediction patches are not related here.

I'm not sure what you mean by "other FB work always fast". If you can give the detailed instructions to test different FBs and how you measure "fast", I'll try to reproduce it.

Unluckily I don't have a host with exactly the same host build version, and I don't have a host that can upgrade from RS1 (or RS2) to RS5(or 19H1), but I think my build should produce the same result as yours, when we test the access speed of the framebuffers. However, I can not reproduce your symptoms and I can not understand your symptoms (e.g. the FB is not slow with 1 CPU).

In my tests on recent buggy hosts, in a Gen1 VM, the legacy FB and Hyper-V synthetic FB are both slow; in a Gen2 VM, the legacy UEFI FB is fast, but the Hyper-V synthetic FB is slow.

[clouds56]

blacklist the hyperv_fb and you would fallback to efifb (if UEFI enabled in guest), then you might not suffer from the performance issue.

# /etc/modprobe.d/blacklist.conf
blacklist hyperv_fb

[marcinwiacek]

thx, I have checked vesafb, uvesafb, efifb and all of them are fast, but I don't have custom resolution (or at least big one). hyper_fb works fine with 1 CPU only (with more is slow). xrandr doesn't work.

Honestly speaking I don't understand why it's so difficult - is it possible create custom BIOS/UEFI for guests which will have set very big max resolution or many big resolution custom VESA modes? I guess, it would help.

for example 1900x900, 1920x1020, 1440x900, etc.

just predefine them please & it would be good to be able to setup them using "vga=mode" in kernel options.