RFC: CPU profiles

[olivereanderson]

This PR introduces an RFC for adding the concept of CPU profiles to Cloud hypervisor similar to QEMU's CPU models.

RFC???

Note that this is the first time (to the best of my knowledge) we propose changes to CHV in terms of an RFC thus I am not sure about the process. In particular I am not sure whether we should have a folder consisting of RFCs (as I have initially done here), or if there should be a separate repository (the way the Rust programming language does it for instance).

For now I would be happy to receive any feedback on the proposal itself. I am also not sure about the naming CpuProfile as it might be confused with performance profiling. I am definitely up for naming things differently if you think that would be beneficial.

Implementation

I will be happy to start implementing this if you are OK with the idea/concept/design proposed here. I don't expect this to be too difficult given that I have understood the code base somewhat correctly.

[hertrste]

Do you have an idea how a profile is defined and structured? How would a description of a profile look like in Rust or maybe even in json/xml/other markup language?

[olivereanderson]

Do you have an idea how a profile is defined and structured? How would a description of a profile look like in Rust or maybe even in json/xml/other markup language?

I tried to give a flavour of that here: https://github.com/cyberus-technology/cloud-hypervisor/pull/8/files#diff-dae5c7bbf8f599ad431bddf058c50c509887063418c07d6fc793a259ee4db4f2R26-R34

An example of usage in JSON would be something like:

"profile": "skylake-server-v1"

If you omit the "profile" key it will default to the "Host" profile which is the current behaviour.

[hertrste]

Do you have an idea how a profile is defined and structured? How would a description of a profile look like in Rust or maybe even in json/xml/other markup language?

I tried to give a flavour of that here: https://github.com/cyberus-technology/cloud-hypervisor/pull/8/files#diff-dae5c7bbf8f599ad431bddf058c50c509887063418c07d6fc793a259ee4db4f2R26-R34

An example of usage in JSON would be something like:

"profile": "skylake-server-v1"

If you omit the "profile" key it will default to the "Host" profile which is the current behaviour.

That is the usage of the profile. But a profile consists of a list of features a CPU has (avx, avx512, ...) and each of these features maps to some kind of feature bit in a CPUID leaf.

My question now is: How does the implementation of a profile looks like. Is it some kind of simple list: skylake = ["avx", "avx512", "vmx", "..." ] or something more advanced? Do we describe a profile in Rust, do we implement some Domain Specific Description language inside of Rust or do we even allow describing a profile in json and parse it during build time?

We probably want to stay simple while making it "relatively" easy to add new profiles.

How does a profile looks like in the qemu code base?

[olivereanderson]

Do you have an idea how a profile is defined and structured? How would a description of a profile look like in Rust or maybe even in json/xml/other markup language?

I tried to give a flavour of that here: https://github.com/cyberus-technology/cloud-hypervisor/pull/8/files#diff-dae5c7bbf8f599ad431bddf058c50c509887063418c07d6fc793a259ee4db4f2R26-R34
An example of usage in JSON would be something like:

"profile": "skylake-server-v1"

If you omit the "profile" key it will default to the "Host" profile which is the current behaviour.

That is the usage of the profile. But a profile consists of a list of features a CPU has (avx, avx512, ...) and each of these features maps to some kind of feature bit in a CPUID leaf.

My question now is: How does the implementation of a profile looks like. Is it some kind of simple list: skylake = ["avx", "avx512", "vmx", "..." ] or something more advanced? Do we describe a profile in Rust, do we implement some Domain Specific Description language inside of Rust or do we even allow describing a profile in json and parse it during build time?

We probably want to stay simple while making it "relatively" easy to add new profiles.

How does a profile looks like in the qemu code base?

Ah, I think I see what you mean. To me this is more of an implementation detail/question, but I think it is OK to discuss that here for sure (as long as it does not get incorporated into the RFC as that is definitely subject to change).

Is it some kind of simple list: skylake = ["avx", "avx512", "vmx", "..." ]

That would be one possibility, or we can map them directly to hard coded vectors with elements of type CpuIdEntry (a type that already exists internally in the code base). Something like this

let mut cpuid_entries: Vec<CpuIdEntry> = match profile {
      CpuProfile:: Host => // hard coded vector of entries for Host // Basically what we start out with now 
      CpuProfile::SkylakeServerV1 => // hard coded vector of cpu id entries
}

and we should have a solid test suite that confirms that what we expect gets set. Since match is exhaustive we guarantee that adding a new CPU profile forces us to also update this code path where it is transformed to CpuIdEntries.

An important question now though is what about CPU features that require additional configuration/kernel support (i.e. hardware support in of itself is not enough?). It is my understanding that these are not set when selecting a CPU model in QEMU (please correct me if I am wrong?). I would suggest if they are good defaults then we should also configure the VM to support it and do that in the setup phase, otherwise the option goes into CpuFeatures and are disabled by default the way amx support is now.

[hertrste]

An important question now though is what about CPU features that require additional configuration/kernel support (i.e. hardware support in of itself is not enough?). It is my understanding that these are not set when selecting a CPU model in QEMU (please correct me if I am wrong?). I would suggest if they are good defaults then we should also configure the VM to support it and do that in the setup phase, otherwise the option goes into CpuFeatures and are disabled by default the way amx support is now.

As far as I understand it now, selecting a CPU model only subtracts features of the Host CPU features. So there shouldn't suddenly be anything that requires more configuration. Things that require additional configuration (I could imagine stuff like SGX) still need to be added via some other CLI or API parameter and are not part of the profiles.

I would hope looking at qemu makes it more clear which CPU features are relevant for the profiles at all.

[snue]

I like the motivation and that you outline the alternatives and pitfalls. I see a strong possibility for getting these intricate interactions wrong.

Are the CPU feature-flags the only thing you are concerned about? What about the other identifying strings in cpuid? Migrating between different cpu generations can cause all kinds of problems and should be well tested in any case. It looks like the proposal mostly tries to circumvent the existing safety checks so the VMM boots the snapshot. If you really need to migrate to an older CPU generation, a reboot-migration (re-launching the VM on the older host) is probably advisable. This has customer visible impact beyond a short downtime of course, which is what you try to avoid I suppose.

[snue]

It's unfortunate that there are so many existing flags that interact with related options. Adding the profile to that list does not sound appealing. I would strongly favor if CpuFeatures could be changed/redefined just enough to accommodate the use case of making more known platforms compatible for migration. I see listing every single feature bit here is tedious, error-prone, and not future-proof. Maybe you could define a "meta feature" here instead that limits the features according to a "profile"/"generation"?

[snue]

Or maybe you have a concrete set of incompatible feature flags in mind already that block your experiments, for which you could add a "disable" feature toggle instead (e.g. -avx512) and see how far that gets you?

[olivereanderson]

I like the motivation and that you outline the alternatives and pitfalls. I see a strong possibility for getting these intricate interactions wrong.

Are the CPU feature-flags the only thing you are concerned about? What about the other identifying strings in cpuid? Migrating between different cpu generations can cause all kinds of problems and should be well tested in any case. It looks like the proposal mostly tries to circumvent the existing safety checks so the VMM boots the snapshot. If you really need to migrate to an older CPU generation, a reboot-migration (re-launching the VM on the older host) is probably advisable. This has customer visible impact beyond a short downtime of course, which is what you try to avoid I suppose.

I think we also want to include the CPU generation in the profile, but I am also not sure how much you gain from these CPU profiles at this point? Maybe you would know more about that?

[olivereanderson]

An important question now though is what about CPU features that require additional configuration/kernel support (i.e. hardware support in of itself is not enough?). It is my understanding that these are not set when selecting a CPU model in QEMU (please correct me if I am wrong?). I would suggest if they are good defaults then we should also configure the VM to support it and do that in the setup phase, otherwise the option goes into CpuFeatures and are disabled by default the way amx support is now.

As far as I understand it now, selecting a CPU model only subtracts features of the Host CPU features. So there shouldn't suddenly be anything that requires more configuration. Things that require additional configuration (I could imagine stuff like SGX) still need to be added via some other CLI or API parameter and are not part of the profiles.

I would hope looking at qemu makes it more clear which CPU features are relevant for the profiles at all.

I would hope looking at qemu makes it more clear which CPU features are relevant for the profiles at all.

I find it somewhat hard to read the QEMU code base as of now (quite possibly because I don't know C), but if this happens to be where the Skylake server model is defined then it looks like the definition is quite a mouthful.

[snue]

I find it somewhat hard to read the QEMU code base as of now (quite possibly because I don't know C), but if this happens to be where the Skylake server model is defined then it looks like the definition is quite a mouthful.

Yes, that's what a "CPU model" typically looks like, and I find the QEMU declaration surprisingly readable for the exposed complexity. If you really want/need to expose fake information to the guest, you will probably end up with something similar. If you just need to mask out a certain feature to prevent the guest from using a cpu feature that isn't guaranteed to be available after migration, tweaking the CpuFeatures with negative flags may be sufficient?

[hertrste]

If you just need to mask out a certain feature to prevent the guest from using a cpu feature that isn't guaranteed to be available after migration, tweaking the CpuFeatures with negative flags may be sufficient?

Can we formulate a clear negative list that is going to work? How likely is it that we forget something in that list that is going to break our migrations once some new CPU is added to the cloud environment?

It feels like defining a profile with a complete list of available features like QEMU does is the safe bet here. Most likely, we will only have a single concrete profile in the beginning. So I guess maintenance might be okay.

But I am not an expert here and open to discussions. If you think a negative list is enough because it will always just be AVX, then that maybe a viable way to go. In principle, I would agree keeping this one as simple as possible, but I think we shouldn't rely on being lucky here that systems where migrations happen are similar enough so that it just works.

[snue]

Can we formulate a clear negative list that is going to work? How likely is it that we forget something in that list that is going to break our migrations once some new CPU is added to the cloud environment?

Treating individual bits is of course not a future-proof solution that just guarantees the guest always sees the same thing. We can probably formulate that list for a concrete set of platforms to migrate between and test the concept. Whether you can safely migrate a guest between two platforms should be tested in any case. I know of installations that use "fingerprinting" of hosts to ensure migrations happen between compatible machines. That's a higher layer management system, of course.

It feels like defining a profile with a complete list of available features like QEMU does is the safe bet here. Most likely, we will only have a single concrete profile in the beginning. So I guess maintenance might be okay.

This is definitely true if you want to guarantee what the guest sees. QEMU has a very different goal than cloud-hypervisor here, in accurately modeling a specific platform. The current code does bitwise subset checks (or equality) where applicable to allow forward-migrating to newer platforms. We expect issues with migrating back after launching on a new platform. As long as it is just a single profile we expect, a "meta feature" to mask out any known-problematic, unsupported or reserved/future flags could be sufficient? That would fulfill the same goal without the extra profile and not necessarily set expectations that this will become a curated processor list eventually.

[snue]

Thinking about it, going with full negative feature masks is pretty close to the "profile" idea. I'd still prefer if that was just encoded under CpuFeatures maybe as compat or compat=... with settings that do not imply it tries to emulate arbitrary CPU types. The gcc architecture optimization settings for x86 are a good starting point to identify relevant feature sets, but the list is much too exhaustive: https://gcc.gnu.org/onlinedocs/gcc-15.1.0/gcc/x86-Options.html#index-march-15

[olivereanderson]

The current code does bitwise subset checks (or equality) where applicable to allow forward-migrating to newer platforms

The current code checks the following CPUID entries/leaves:
https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/arch/src/x86_64/mod.rs#L396-L508

in my understanding those checks are for the most part about CPU feature flags (and some other things) and leaves out:

• EAX=0: Highest Function Parameter and Manufacturer ID (i.e. what is returned in EAX when calling CPUID with EAX = 0)
• EAX=1: Processor Info. The processor info (Processor Model, Family etc) that gets stored in EAX is not checked. For this leaf only ECX and EDX are considered/compared between the source and destination VM.
• EAX=2: Cache and TLD descriptor information
• EAX=5: Monitor/ MWAIT features
• EAX=6: Thermal and Power Management
• EAX=0Dh: XSAVE Features and State Components

This list of things that are not included in the compatibility check is not exhaustive. I am wondering how much of what is not checked is disabled by CHV anyway though (I should probably check this tomorrow).

I would like to figure out exactly what you need to check in the first place to ensure that a live migration can be assumed to be safe/succeed, but are you of the opinion that this is a super complex topic that can only be checked by running a large test battery @snue?

[olivereanderson]

Thinking about it, going with full negative feature masks is pretty close to the "profile" idea. I'd still prefer if that was just encoded under CpuFeatures maybe as compat or compat=... with settings that do not imply it tries to emulate arbitrary CPU types.

I was also thinking along these lines to begin with myself, but then decided against it (at least as part of CpuFeatures) because it opens a can of worms when you have a set of features that may contradict each other. This is in some sense also the case in the presence of both CpuProfile and CpuFeatures, but there you can at least clearly state that CpuFeatures always takes precedence.

I do however still agree that having profiles

that do not imply it tries to emulate arbitrary CPU types

would be beneficial. At first I considered something like only having
X86_64_V3 (and v1, v2, v4) variant(s) of in the CpuProfile enum which would then limit the available CPU feature flags, but those sets also leave out a lot of stuff you probably do want (even things like hardware support for AES) and may still not even be enough to guarantee safe migrations.

If it happens to be the case however that what is not in the aforementioned microarchitecture levels neatly falls into the following categories:

1. The feature can be set as a feature/option in CpuFeatures and enabled by default (if available). Things like AES feature sets for faster cryptography would fit in this category.
2. The feature needs dedicated configuration. SGX and TDX support land here.
3. The missing feature should not be permitted by CHV for guests under any circumstances thus should unconditionally be disabled.
4. It is safe to just let the feature be whatever the host sets it to be. Even if the migration target has different values here it will not affect functionality (other than possibly performance).

Then I am all for going with this route (i.e. CpuProfiles that are not related to any specific CPU, but only enable a specific set of features that can be extended and/or altered by CpuFeatures). If such a categorization is not possible however, or ensuring migration compatibility is so difficult that migrating from one generation to another (both being restricted to the same subset of CPU features) can cause problems, then I really don't know how to proceed here.

[olivereanderson]

@hertrste and @snue thanks for the comments. They have indeed provided a lot of food for thought!

I think I want to think a bit more about this problem and potentially completely rewrite the proposal. Hence I will close this PR for now and let you know when the new version is ready.