Nova epoxy upgrade batch 2#637
Merged
Merged
Conversation
"nova-manage placement sync_aggregates" doesn't support Ironic hosts, because they have multiple compute nodes. In general, Nova doesn't support aggregates for Ironic hosts. But we still use them for assigning an availability zone to a rack (a building block) of nodes grouped into a Nova host. Therefore, we patch above-mentioned command to handle a list of (in most cases one) compute node UUIDs, only erroring out if the host we find multiple compute nodes for doesn't look like an Ironic host (based on having "ironic" in the name). There can be Ironic hosts, that only have nodes assigned when those nodes are getting repairs or getting build up. Those Ironic hosts would come up empty when searching for ComputeNodes in the sync_aggregates command and would be reported as a problem, which makes the command fail with exit-code 5. Since it's no problem if an Ironic Host doesn't have a ComputeNode, because each node is its own resource provider in placement anyways, we ignore Ironic hosts without nodes now in the error-reporting. Change-Id: I4f7e5fd82c51ce5d6f42089beb5a70e469ec54df Incorporated-Change-Id: I163f3e46f2e375531b870a363b84bba67816954d (cherry picked from commit f899a7c)
Change-Id: I117a4d97ff7e901ce9b590f04a40e58053f0e845 (cherry picked from commit a39dd17)
We expose a capability "resource_scheduling" which enables the `ComputeManager` to decide whether to allow multiple VMs on the same host in the post-scheduling check. We also update the DRS rules when spawning/resizing/migrating a VM. "soft-affinity" is excluded from DRS rule creation and thus only works on cluster-level not on HV-level. The reason is, that DRS doesn't know about "soft" and will require the VMs to run on the same ESXi. Having the VMs run inside the same rack should be good-enough though. For managing DRS settings, we introduce a new module `cluster_util` to the `vmwareapi` driver. Change-Id: Iba9706b7353575fd3f5df5e903e3f254c5e49d73 Co-Authored-By: Johannes Kulik <johannes.kulik@sap.com> Co-Authored-By: Fabian Wiesel <fabian.wiesel@sap.com> Incorporated-Change-Id: I6d44910c420de4c76b6112904ccfebe3ec923098 (cherry picked from commit d1b19da)
We still have to old `_create_vm_group_spec()` to get the same behavior, but nobody outside of `cluster_util` uses it. The new functions `create_vm_group()`, `create_host_group()` and `create_group_spec()` expose a clearer interface and make it possible to overwrite hosts/vms instead of just appending to existing ones. This will be useful if customers can upgrade server-groups via API. Change-Id: I5444318994ac7929a24d357fabb8133410d5bd9d (cherry picked from commit 27a9012) (cherry picked from commit d945c77)
This splits up functionality for creating rules between VMs into `create_vm_rule()` and `create_rule_spec()` and thus enables us to use it more controlled also from the outside, e.g. in the upcoming sync loop for server-groups. Since DRS ignores the "mandatory" attribute for VM-VM rules, we remove setting it here, so it doesn't look like changing the value would make a difference. Change-Id: Ib515e02226e674d0f7cbdc3c354ade5cd77a0b8c (cherry picked from commit 9386f11) (cherry picked from commit 03232ab)
We need to know the compute host to query the instances from the API for sync_server_group(), so we save it onto the VMwareVMOps instance when the VMwareVCDriver instance receives it in the init_host() call. Change-Id: I473539000dde4629e2a251cd9145c8047ce60a41 (cherry picked from commit 22ab4cb) (cherry picked from commit e925801)
We want to create a sync-loop for applying/removing server-groups changed by the user via API. For this we need to be able to distinguish between driver-created DRS groups/rules and admin-created ones. To do this, we introduce a prefix for the DRS group/rule name, which will work as an identifier later on. Since we're now using not only a UUID, but a UUID with a prefix, we change GroupInfo to have a "name" attribute instead of a "uuid" attribute. As we're changing how DRS groups/rules look, we need a migration to run before deploying this to production. Change-Id: I07ecd1953a85d0f53082fa9b0c49b80c2c9bf9d3 (cherry picked from commit 7984cae) (cherry picked from commit 1e5e590)
This function is able to add/remove DRS overrides for a VM. DRS overrides can be used to set an explicit behaviour for a VM that can differ from the default of the cluster. Change-Id: I9022869ad546e28ec9de66e17d639c3314b19832 (cherry picked from commit e3e44a4)
The DRS rules created by Nova do not use those VmGroups and they don't seem to server a purpose otherwise. Therfore, we only update/create a VmGroup that's not based on a server-group, i.e. an admin-defined group. We currently only have this kind of group for the special_spawning case to support a free host for big VMs. Change-Id: Ide011e157ad46037304cfdb52b1db397dde38cc8 (cherry picked from commit 8c3640b) (cherry picked from commit 0f0df08)
These functions are simple helper functions to create, update and delete a DRS rule, so we have to code necessary for that at a single place. Change-Id: I44aeed6f99b9803adca0062b2d7b12cc2e295f03 Incorporated-Change-Id: I86f7ca85d9b0edc1406a54a6f392bfff8f0af00d (cherry picked from commit 94f8c7a)
The VMware driver supports syncing server-groups as DRS rules into the cluster managed by the nova-compute node. The method will be called when a user updates a server-group via API and will later get reused when spawning a VM, too. Instances in certain states need to be excluded from the sync for multiple reasons: 1) when instances are going away during the sync, this can lead to an error 2) VMs contained in a DRS rule cannot be vMotioned - excluding them makes it easier for the live-migration code to update the cluster appropriately We do not sync "soft-affinity" server-groups, mainly because we use "soft-affinity" is our way to give customers the possibility to end up on the same cluster, without the need to end up on the same host, too. We also cannot implement the "soft" part currently and thus spawning a VM will fail if the host is too full, as there are no non-mandatory VM-to-VM rules in VMware DRS. We might be able to optimize it a little more, by keeping a local list of DRS rules instead of querying the cluster in real-time. Tests have shown, that it takes < 500ms to query the cluster, though. Change-Id: I534c035a1e2d962cf5d187d56d104e743f7ade15 Incorporated-Change-Id: I8bc94aebebcd878f60c33fe009f048afeb9a42c0 Incorporated-Change-Id: I745ac6616eefd193ce8c7a9a5cba3c68fc59ac75 (cherry picked from commit 52f42fc) (cherry picked from commit 70aca2b)
Searching for a list of instance groups belonging to a list of instances can be helpful for checking if a server-group update is valid and maybe if a nova-compute wants to sync down server-groups like the VMware HV is able to. Change-Id: Iec93becf0299ec0617e99ce16c06e37c84cb33ee (cherry picked from commit 838142b) (cherry picked from commit 243275e)
We're going to change the API to allow updates of server-group members and need to trigger a sync in the backend, whenever such a change occurs. Therefore, we need a bunch of methods going through the whole stack to call the driver. We use a cast and not a call here, because we cannot let the API wait for all of this to happen. If the cast gets lost somehow, a sync-loop implemented in the driver will pick the change up, eventually The API will have to supply a list of hosts to call, so we only sync the group on the necessary hosts. Change-Id: I00e012ed52ba9fd36b094ecf2dc86b023f2f5a21 (cherry picked from commit f3d2ad7) (cherry picked from commit ae57baa)
This function wasn't previously available, because - as the removed comment says - there was no user-facing API that would allow removal of instance group members. Since we plan on changing the API, we need to add a pulic method. Adding this method, we also need to send notifications like we do for "add_members()" and thus added the appropriate functionality. Change-Id: I4270212b57782e5ffeaf69dc3bd57c7c60a7ffe5 (cherry picked from commit df3bd90) (cherry picked from commit e7c9a01)
In case we missed an update to a server-group, we want to be able to recover at some point in time. Therefore, we implement a sync-loop to call the driver's sync_server_group() for every server-group UUID we find as belonging to our host and also for every DRS rule we find. Since we only start the sync-loop for server-groups once on startup, we have to make it resistant against any exceptions - e.g. if the DB is unreachable for a time or the vCenter is unreachable at the time. Change-Id: I9a633dc87ad1aab7d5f00e5143fac97dd3b87176 Incorporated-Change-Id: I74036b6bfc449b407f687afac1ba4365bcbdc2ee (cherry picked from commit 583b1fd) (cherry picked from commit 4a58936)
We can use sync_server_group() in update_cluster_placement() to use the same mechanism that's used when a user edits a server-group in the DB. This makes sure, that a DRS rule is only created, if it has more than 1 members and also syncs in the other member in case the newly-spawned instance is the second member. Change-Id: I62c1ae3d897f5ccda8788a4d4b23e553be8bc5cf (cherry picked from commit 2dd55ea) (cherry picked from commit f82c01f)
When retrieving multiple - or all - server groups, the code tries to find not deleted members for each server group in every cell individually. This is highly inefficient, which is especially noticable when the number of server groups rises. We change this to query all members of all server-groups we will reply with (i.e. from the already limited list) in advance and pass this set of existing uuids into the function formatting the server group. This is more efficient, because we only do one large query instead of up to 1000 times the number of cells. Change-Id: I3459ce7a8bec9a9e6f3a3b496a3e441078b86af0 (cherry picked from commit e676dd1) (cherry picked from commit f216de6)
We exted "server_groups._get_not_deleted()" function to return a dictionary of instance UUID to host mapping. This is a preliminary step to introducing an update of the server-groups, which will need both the instance UUIDs and the hosts to filter out to-be-removed instances and check the validity of the policy when adding instances. We cannot use "InstanceGroup.get_hosts()" for this, as it's not cell-aware and thus extend this function instead. TODO: * tests Change-Id: I253ef54560c2422baec187b350f05b1b2affc34e (cherry picked from commit a94f6f7) (cherry picked from commit 0b38c20)
This new API endpoint allows to add and/or remove members from a server-group. We found this to be necessary, because instances might get created without a server-group, but later need an HA-partner and re-installing would mean downtime or too much effort. The endpoint checks the validity of the policy is still given after all changes are done and strives for idempotency in that it allows removal/addition of already removed/added instance uuids to accommodate for requests built in parallel. We do not allow a server to join a second server-group if it already joined one. The whole change to the server-group is errored out if any of the servers are already in another group to make it easier in validation and either apply the whole change requested by the user or nothing. When we add/remove a server to/from a server-group, we have to update the server's RequestSpec's `instance_group` attribute, because this is used during scheduling when resizing a server to record a list of appropriate hosts for the instance. Only members/admins can change the server-groups. TODO: * api-request docs Change-Id: I30d5d1dc3a41553b4336aad3877018989159495c Incorporated-Change-Id: Ibc78cbd80ae49df9be60f6d9eda56a47d9ffe39e Incorporated-Change-Id: I6a06e4b4a8c22737c20ab51e85ceb2bf98082b26 Incorporated-Change-Id: Ic193dd3c59bc717ba5329f63054297f44127d76d Co-Authored-By: Jakob Karge <jakob.karge@sap.com> (cherry picked from commit aec6f6e)
Instead of getting all instance groups from DB and after then limit the results with common.limited, we added limit and offset parameters to _instance_group_get_query function to limit the database query itself. SQLAlchemy throws an exception when applying a filter to the query after setting limits and offset that's why project_id parameter added to _instance_group_get_query. Change-Id: I30c9bc43f4b7ff9205f6a9fa6cb947b669a2fbba (cherry picked from commit 6802f80)
When a server-group gets updated, we send an update to all hosts that got an update - which are the hosts the instances we removed/added belong to. Change-Id: Ica1e516358bf6301ab7a2c59631203721dbe658d (cherry picked from commit cfa0696)
If we don't reserve all memory, there's a swap file pre-created for every VM on the ephemeral storage - even if it's unused. But this option comes at the price of not being able to over-commit memory. Related issue: CCM-9589 Change-Id: I916f3ed8d32efc9de36b943546fee7440bbbf7ca Note: This will later be reverted in favor of manual memory allocations. (cherry picked from commit e028cd6)
Add global setting `bigvm_mb` This is necessary in multiple places e.g. the scheduler and compute-driver to handle big VMs differently. We cannot currently use flavors for that, because there are still custom flavors in the system that we don't control. (cherry picked from commit 881889d) [scheduler] Add BigVmFilter The filter filters out hosts having more than `bigvm_memory_utilization_max` memory utilization. This is necessary, because it takes too long to find free space on a host for the big VM otherwise. (cherry picked from commit d284fc3) [vmware] Add DRS override for "big" VMs Since DRS doesn't play well with large differences in VM size and HANA doesn't play well with vMotion, we're disabling automatic re-balancing actions for "big" VMs by the DRS. On spawn, we add a `ClusterDrsVmConfigSpec` for instances having more than by default 1 TB of memory - configurable via `bigvm_mb`` configuration variable. The vCenter deletes the overrides automatically when a VM is deleted. One can view current overrides in the vCenter UI via $cluster (e.g. productionbb098) -> Configure -> VM Overrides. We're always adding the rule in `update_cluster_drs_vm_override`, because we only call it in the driver's `spawn` method which seems to be called only when a new vm object is created. Therefore, there should not be any existing rule for the VM. (cherry picked from commit ae68374) [vmware] Always reserve all memory for big VMs Since there are still custom flavors in use for big VMs, we cannot rely on the flavors to set the reserved memory for big VMs, which is necessary because otherwise small VMs will compete with the big VMs' HANA, which does not work well. We're using `CONF.bigvm_mb` here to identify a big VM and reserve all memory regardless of `CONF.reserve_all_memory`. (cherry picked from commit 706bcd8) [scheduler] Make BigVmFilter more dynamic Instead of having a static threshold for the `BigVmFilter`, we compute the threshold depending on the hypervisor size and the requested RAM of the big VM. The reasoning behind this is, that on bigger hypervisors a small big VM doesn't matter as much as a big big VM, e.g. 1.4 TB on a 6 TB hypervisor vs. 5.9 TB on a 6 TB hypervisor. We basically aim at 50 % of the requested RAM free on average over all hosts in the cluster. The filter thus currently only works if a compute-node is in a host aggregate defining the `_AGGREGATE_KEY` (currently "hv_size_mb") having an integer defining the hypervisor's RAM size as a value. (cherry picked from commit 6ad6a2c) [scheduler] Add BigVmHypervisorRamFilter Since we've got multiple BigVM-filters now, this also renames `BigVmFilter` to `BigVmClusterUtilizationFilter` for clarity. With having more diversity in the hypervisor size (e.g. 6 TB hypervisors right around the corner), we have to make sure big VMs actually fit the hypervisor. For that, we introduce a new scheduler filter - `BigVmHypervisorRamFilter` - that prohibits scheduling of big VMs not fitting the hypervisor. In KVM-deployments, this would be the job of the `RamFilter`, but since we run VMware and the scheduler only sees the cluster's and not a single host's resources, we use a special host aggregate attribute - "hv_size_mb" - to retrieve the hypervisor size as already done for `BigVmClusterUtilizationFilter`. (cherry picked from commit 4b6867a) [scheduler] pep8 cleanup (cherry picked from commit 39231e4) Add `is_big_vm` utility function This function should be used to check whether one is handling a big VM. There, all necessary conditions are gathered in on place so one cannot forget a condition. This also makes it easy to change the conditions later on. The conditions we use to check for baremetal flavors are the same that `nova-scheduler` uses. (cherry picked from commit cc46cf4) [scheduler] Remove BigVmHypervisorRamFilter We actually don't need it, because our filter queries the placement API for resource providers capable of fulfilling the request. With the vmwareapi driver setting the `max_unit` value for `MEMORY_MB` to the cluster's maximum host size, we don't have to add another filter into the scheduler, as too small hosts won't even end up as scheduling options. (cherry picked from commit c2fc281) [scheduler] Use placement API to get HV size We recently discovered, that the hypervisor size is actually available in the placement API. Every compute host has a resource provider with an inventory. There, the vmwareapi virt driver sets the `max_unit` for the `MEMORY_MB` resource class to the biggest host found in the cluster. This means, we don't have to use aggregates to get the hypervisor size in the BigVm filters, but can retrieve it from the placement API. We're building a cache of the value, that expires after 10 min. This guards the case, where a lot of VMs are scheduled in a short period of time - even though tests showed, that querying the inventory of all hosts in QA did not take longer than a second. Cache expiration also guards against stale values for newly-created hypervisors that did not set their inventory, yet. (cherry picked from commit 965d3b4) [vmware] Make `partiallyAutomated` a constant We usually use variables as constants instead of directly embedding strings. (cherry picked from commit 81097b0) (cherry picked from commit fbf0a2d) Change-Id: Idca77aae46d67623f26397ba5ea0dded2892fc05 (cherry picked from commit a3db710)
[vmware] Update memoryReservationLockedToMax on resize We need to update this setting in the case where CONF.vmware.reserve_all_memory is False (the default) and we downsize a big VM or upsize a normal VM to a big VM. To make it simple, we just recompute the setting every time and set it in the reconfig spec no matter what the previous state was. Change-Id: Ic9283af63cb3bf90f0bed007474d7b6583866003 (cherry picked from commit 3bca302) [vmware] Support removal of drs overrides This commit changes `update_cluster_drs_vm_override()` to be able to remove overrides again. This is not necessary for deletion of a VM, but we need to remove it if a VM gets resized under the big VM limit. If there is no override, the task will fail with `InvalidArgument`. Change-Id: If1de5136cbc7c59babac24e8293ddb7a12ea8411 (cherry picked from commit 7ab2765) [vmware] Remove/Add DRS overrides on resize A VM becoming a big VM needs to get an DRS override so it doesn't move around automatically, because that might be bad for performance. A VM no longer being a big VM needs to get its DRS override removed, because this interferes with spawning new VMs - especially the special_spawning code implemented for big VMs. We're currently unable to remove a DRS override as our SOAP library is unable to create an appropriate request accepted by the vCenter. Therefore, we still allow the resize to work for now and have to manually remove the override later on. We add code for the removal, so we get a reminder in the logs and sentry, that this needs fixing. We just don't fail the resize for it. Change-Id: I89aedc29e36c86415a205b7181e06b84316bb82e (cherry picked from commit eaad29e) (cherry picked from commit 21b26df) Incorporated-Change-Id: I4d344347860c7d97d6f4b2e68d9bbac069d71b74 (cherry picked from commit 860c0d5)
After the recent changes of syncing server-groups, the syncing of rules in cluster_util.update_placement has become unused. The code can be simplified as a preparation for live-migration, as it will add complexity. VMops.update_cluster_placement now calls - sync_instance_server_group, which fetches _the_ server-group for the instance and syncs it with the existing VMOps.sync_server_group - update_admin_vm_group_membership handles the membership to special_spawning_vm_group Since that is the only remaining use-case of cluster_util.update_placement, the function has been renamed to update_vm_group_membership and reduced to that use-case Change-Id: I94c311790610fc4658f9b06ac052ad46660f1cea (cherry picked from commit a05999b) (cherry picked from commit 673a3db)
When we log the cluster moref in getting the list of instances, we log the whole moref and thus create multiple newlines in the log-line. To have all data in a single line, we now only log the moref value. Change-Id: I89d3908abdffe60570cec947e9a74b544488b535 (cherry picked from commit 920d080) (cherry picked from commit 0c48be5)
If we disabled the host automatically, because we couldn't reach the vCenter, we set a flag on the `VCState`. This flag is lost with a compute-node restart, but the service stays disabled. We now check the service's state and compare the disabled_reason on startup to detect if we disabled the service automatically. This means, the driver will automatically enable the service again if it can reach the vCenter. For the first start of a compute-node, the compute-node might not be there. To not prohibit the creation of the compute-node, we ignore the error raised when not finding it. It's fine to ignore the error and not print it, because there's another point in the code doing basically the same and printing out a message if the compute-node doesn't exist, yet. Change-Id: I1e49d5962adfd0bb585543c7a5580d343628f687 Incorporated-Change-Id: I25f77a69a86a876be28d31be2c626a66a5df40b3 (cherry picked from commit cc26bc3) (cherry picked from commit fcb1363) (cherry picked from commit c680b79)
These helper-classes allow iterating over the Task/Event objects belonging to another object, hiding the complexity of the vSphere API splitting those Tasks/Events into pages. Change-Id: Ide281bdd613526561324a9368a67a720f580eb4f Partly-Incorporated-Change-Id: I196bb9ab48867f314d9a5f3b7566384fa72778df Incorporated-Change-Id: I86804af29f4c6aff14fa5495e5344377488ba8fe (cherry picked from commit 368f2a7) (cherry picked from commit 52813fa) (cherry picked from commit c5ed5df)
- Import image as VM template per datastore. - First look on other datastores before fetching the image. In previous versions, we also had code in here that would create a VM from these template-VMs. We never used that feature and thus stopped porting it forward. When we try to fetch an image-template from another datastore, it might happen, that the template has an incompatible hardware version and the vcenter raises VirtualHardwareVersionNotSupported if we try to clone it to our cluster. We handle this case by logging a debug message and continuing with the next image-template we find, as this one is unusable for us. Change-Id: Ia806089b244d331003fe6f7e047b5908088ff367 Incorporated-Change-Id: If9dc9b2a13171252e5f0f0b3a99a51be2f28c6eb Incorporated-Change-Id: I0aa58ad11c499e9423c7ecc7998325b05dd9147e (cherry picked from commit a04c8f7)
Order of imports Duplicate imports Spelling mistake Indentation Change-Id: I4ff5594b0a628fee9579761248627099b3f251b8 (cherry picked from commit ec4bb85)
Instead of renaming `vim_util` to `vutil`, we use it directly as `vim_util`, because we have no need to import another `vim_util`. Change-Id: Ic484240f29573cffbbc05742da0231ce17764dca (cherry picked from commit f5f9715)
We use the same functionality for OVA import as for normal image import, which automatically brings handling of duplicate name. In the process, we also fix that OVA import returned a folder path instead of a VMDK path. Change-Id: Iae731490716cc7592d2e99b7ca112fa56d40e4cc (cherry picked from commit 30e15e0)
The advantage of this approach is, that we now properly encapsulate each compute node instead of 'messing' remotely around in another vcenter. So caching and etc should work as expected. The downside is additional hops: Before: compute -> other-vcenter Now: compute -> messaging -> other-compute -> other-vc So more ways of going wrong. Change-Id: I37be358ff7c3bd9f786e6ce086e91ff2b2fc3861 (cherry picked from commit da49100) (cherry picked from commit 7f70e86)
This can be used to have different CPU/memory reservations for hosts depending on the hostgroup they're in. We can also set defaults in percent with this instead of only static values as supported by CONF.reserved_host_memory_mb and CONF.reserved_host_cpus. We also have to change the `os-hypervisors/details` call to adhere to the reservations so both Placement and Nova report the same values. It's important to us, because we get or used to get the quota capacity from there. I opted for changing the resources (for that single call, hence copy.deepcopy) instead of subtracting the reserved values from the totals afterwards, because if an attribute is unset on the `ComputeNode` object, we cannot access it - not even with `getattr`. It just raises a `NotImplementedError` because it cannot fetch the attribute from somewhere (e.g. the DB). But since we need to subtract from the total, we would have to access the attribute. Catching the `NotImplementedError` didn't seem like a good option to me. With subtracting the reserved resources, we have to make sure we do not fall below zero - which could happen for Ironic nodes, where the memory is set to 0. With us disallowing this, one of nova's assumptions failed - that we don't collapse negative values to 0. But since that test only worked with reservations, it couldn't detect that we still allow negative values. We changed the test to test if instance memory can still reduce the free memory to below zero. NOTE: Setting these settings too differently per hostgroup for the same cluster might result in placement API getting confused and scheduling being impossible, as the `max_unit` is still set to the maximum per cluster. In case of our HANA HVs placement would therefore always consider the HANA HVs max unit. That should be no problem as smaller VMs should always fit. (cherry picked from commit 5250436) (cherry picked from commit 28f4eb8) Change-Id: Ie4c9314bd19ec1565f3a380f74054fb1f84a0dcc Incorporated-Change-Id: I191aedb0e3bba7698825771089cf134f320368ec Incorporated-Change-Id: I1c92b193bf867db3a749b09bac57bde500c39e19 (cherry picked from commit 58796bc)
For this to work, the two settings `spawn_on_empty_host_vm_group` and `spawn_somewhere_else_vm_group` need to be defined. The first one specifies a DRS VM-group name, that should have a rule to force the VM to spawn on the prepared host. The second setting specifies a DRS VM-group name where all other VMs should be placed in. This group should have a rule that prohibits all members from spawning/running on the prepared host's hostgroup. (cherry picked from commit 5c80f53) (cherry picked from commit c661179) Fix unit tests for spawn VMs on prepared empty host This is regarding the change in _get_server_groups in inside vm_util.py where we are checking if instance needs special spawning. Commit - 5c80f53 (cherry picked from commit b10324c) (cherry picked from commit f0f7b06) Change-Id: Iac974e16fa46399562e507c2286daa58a9e75c9d (cherry picked from commit db05727)
With supporting memoryAllocation updates also on resize, we don't have to lock the memory reservations to max but can use the memoryAllocation instead. This cleans up the code, as we don't have to pass down the parameter through a lot of functions to set memoryReservationLockedToMax. We still have to deconfigure `memoryReservationLockedToMax` on resize, because there might be some old VMs still running with that setting. Change-Id: I91d892c13b31745e190fcee8b97a2f3fb25fc014 (cherry picked from commit 02aa7dc) (cherry picked from commit 1406e3f) Incorporated-Change-Id: Ic9aa65ce410b42dbdee907361cb56d8b1d254fdc (cherry picked from commit 85300f1)
VMs with reserved memory have better memory allocation
performance and -- it's suspected -- less softlock issues
too. Coincidentally, many larger VMs also have high
performance requirements that puts strict demands on the
quick availability of their memory.
Implement memory reservation and a configurable maximum
number of cluster hosts that could theoretically fail and
still let all VMs with memory reservation boot up.
Nova provides the flavor extra_spec
"quota:memory_reservation"
which reserves the given amount of memory. This change does
not make use of that feature, but instead introduces a
parallel custom resource
"CUSTOM_MEMORY_RESERVABLE_MB".
The reason is that "quota:memory_reservation" does not allow
for limiting the total amount of reservable memory in the
same way the resource provider mechanics do. This is a
requirement for tolerating the above-mentioned partial host
failures, which can occur because a VMware host is a cluster
of hypervisors.
Add a "vm_reservable_memory_ratio" value to the cluster
stats in the VMware driver, and use that to calculate and
return the MEMORY_RESERVABLE_MB resource from the VMware
driver's get_inventory().
The flavor-based memory reservation has less priority than the setting
`reserve_all_memory` and the reservation of all memory for big VMs.
(cherry picked from commit 93e22ef)
Change-Id: I4b1457700428189457b94739d617467248a662df
(cherry picked from commit 7f50b0b)
We already have a function listing all the VMs in a cluster (`list_instances()`) but only returning the uuids. With this change, we add an additional `_list_instances_in_cluster()` method called by `list_instances()` so we do not change its interface. The method will also return the retrieved properties of the VMs, if the `additional_properties` parameter is set. This means we don't have to duplicate the code. This feature became necessary for handling the special-spawning cases of big VMs. We don't have a mapping for VM -> host in nova and have to retrieve it. (cherry picked from commit 1fb23b6) (cherry picked from commit 30daac7) Change-Id: I02a6938c8865c18089bb03a6add80ab7ae7e37b4 (cherry picked from commit 50eb6a0)
We need to be able to spawn big VM and thus have to prepare a host for it i.e. free it from other VMs. This cannot be mapped in nova currently, as nova has no view on the individual hosts in a VMware cluster. Therefore, we implement some function in the VMware driver to actually free up the host and also release it into the cluster again. Placement is used for keeping a view over all clusters, as we need one host free per HV size, per Shard, per AZ. We have to make sure that we run the functionality only once. Placing it into the conductor will - at least in regions with cells - run it multiple times. So we could add some leader-election into the process or create our own service that we only spawn once per region. We opted for the latter, as it should be simpler. We have a threshold in place up to which memory usage in a cluster we try to spawn big VMs. Since trying to free a host in a too full cluster would lead to memory overcommit on the remaining hosts, we by default only keep or create a free host until the cluster has by default 80 % memory usage. If a RP is marked as only accepting hana_* flavors by the `CUSTOM_HANA_EXCLUSIVE_HOST` trait, we do not need to adhere to this threshold. These clusters have explicit failover hosts to keep enough resources available. Additionally, we can also prohibit freeing up a host in a cluster or giving back a free host to the cluster by adding the `CUSTOM_BIGVM_DISABLED` trait to its resource provider. Since spawning on disabled compute nodes doesn't work and Nova adds the `COMPUTE_STATUS_DISABLED` trait to the RP, when a compute node gets disabled, we can and should ignore those nodes when looking for a new big VM cluster. The placement-client has an internal cache for resource-providers, which we need to fill to be able to use a couple of functions. Since we're using private functions, the client doesn't call `ensure_resource_provider()` everywhere and thus might not have the resource-provider in the cache and thus refuses to work with it. If we change something by a direct request to Placement's API, we also have to update/refresh the cache. We clean up a provider if we get an "error" returned as state. This can happen, if the host we decided on earlier e.g. got disconnected or became failover host. We then need to find another host - maybe in another cluster - to be able to spawn VMs for that hypervisor size. We also have to remove the host from the hostgroup, so it can properly join the cluster later on. We check already freed up hosts in every run to ensure that they are still suitable. In calling `free_host()` again, the `nova-compute` service should return FREE_HOST_STATE_DONE if the host is still suitable. We also clean providers that match our naming scheme but for which we do not find a vmware provider anymore, because they can lead to `KeyError`s late in the code. These could be created with changing settings or manually allowing big VMs on some cluster. inside nova-compute =================== If the DRS policy is not set to "fullyAutomated", i.e. if DRS will not take action on existing, running VMs, we first check if there's already a host free (e.g. because an operator freed one manually). If we don't find one, we have no chance to free up a host without DRS and thus we error out to tell nova-bigvm to search for another cluster. We ignore failover hosts and hosts in disconnected state or maintenance mode. This is the same as `vm_util.get_stats_from_cluster()` does for reporting capacity cluster capacity. Reasoning: We can't spawn a VM on any of those hosts. These filters are not only applied when searching for a new host to free, but also for checking on the host already destined to be freed up. If its state changes, we return an error-state, which should let nova-bigvm restart the search for that host's hypervisor-size. Instead of adding up the configured memory of the VMs on a host, we add up the used memory of the VMs. The reasoning behind this is, that it would be faster/easier to free up the host if the VMs are not using all their RAM, because there's less to copy around and hopefully less memory activity. We remove the hostgroup, because having an empty host list is not possible. At the same time, we also have to delete the matching DRS rule, because DRS does not update its internal state for a rule, if the rule has become an invalid configuration. This effectively means, that the rule stays in place if we remove the hostgroup but not the rule using it. Previously, we had a check in place that would return an error if no VMs were found on the cluster. This prohibited using newly-built clusters, that did not have a VM, so we do not want such a check anymore. We had a case, where the vCenter discovered some random VM, which didn't even have a `config` attribute when retrieved from the API. We now ignore these VMs instead of letting the `KeyError` bubble up, because they are pretty much unusable and should not matter for our decision-making process. Additionally, with vSphere 7 there are `vCLS` VMs providing (parts of) the DRS service, which we need to ignore. The reason is that DRS doesn't move those VMs, even if they violate a DRS rule. Therefore, we would never see a host getting freed up, if it contained a `vCLS` VM. They take up 100 MiB of reserved RAM and thus should fit next to a big VM. Therefore, we ignore them. With large VMs being set to partiallyAutomated, compute nodes stayed in "waiting" state more and more often and never came out of it. Since we still need to deploy big VMs to those compute nodes and they are usually just stopped from being free by a large VM that cannot move, we often mark those compute nodes free by hand. Since this doesn't scale, we want to automate this behavior by letting the nova-compute take the same decision. Therefore, if there are only partiallyAutomated VMs left on a host, we still report that host as freed up. This is done by ignoring all `partiallyAutomated` VMs during the check of running VMs on the host. When we spawn or resize a VM on the prepared empty host in the cluster, we have to mark the resource as reserved on the special resource provider. This allows the code handling the emptying of hosts to recognize a used host and start the clean-up operations necessary after successful spawning/resizing. Before marking the host as used, we have to refresh the RP inventory, because the RP generation in our cache can be out of sync. For resizes, we have to apply it for both big->big as well as small->big resizes. scheduler ========= Ignore hana/BigVM flavors by default in the ShardFilter, allowing such flavors to be deployed on any shard. This behaviour can be disabled globally by setting [filter_scheduler] sharding_ignore_hana to False It can also be disabled per project, by adding the tag `use_individual_shard_tags_hana` to the project. We can tell the scheduler how to determine if a flavor is hana and/or BigVM by setting the [filter_scheduler] hana_detection_strategy configuration to either `hana_exclusive_host` or `memory_mb`. This has to be set to `memory_mb` for region where there are no hana exclusive BBs and nova-bigvm is still used to free up hosts. Co-Authored-By: Marius Leustean <marius.leustean@sap.com> Co-Authored-By: Jakob Karge <jakob.karge@sap.com> Change-Id: I2e6e7ecf5348ac7d32e23349c7196b65a73b4cf8 Incorporated-Change-Id: I65c2be9781b60e31d9a6c6dd84a02c7fe4aa1e72 Incorporated-Change-Id: I6e994281ae2e62096c2f5241dd493fe1f9c9d515 Incorporated-Change-Id: Idbdfe82b4057844401e710fb9d87141478bb3353 Incorporated-Change-Id: I5ceb4d9338f6d94f49cc2deff25eefb19df2030f Incorporated-Change-Id: Ie033332436674f4fe792f4aa3f83f33b12a6d9ed Incorporated-Change-Id: Ic9d707c59a4ea405f3a982dbe269cdfea0d03aa5 Incorporated-Change-Id: I2329db43adc00e04b07d16d312361ae5e669d298 Incorporated-Change-Id: I36813ed3d95fd8572c6b75544ebb2fc1936f6bdb Incorporated-Change-Id: I737f312db0e156fa971a189d47efd227c666b178 Incorporated-Change-Id: I43d3d060f7c51841a8561a29d3189e37e4f87fb1 Incorporated-Change-Id: I2883c5c713ee657006c80d61ebc59a086ec22411 Incorporated-Change-Id: Ibd576211ac33f38ef0e1b9016381a955916d7c1c Incorporated-Change-Id: I1b18a7b7db0452a10f0adc499be2df26d923f936 Incorporated-Change-Id: Ie96a41bd9151e869fabd18d69777f38db85d0ca6 Incorporated-Change-Id: Iaa1d18eb0a3e78bf1e361c8e8d1040aa07344448 Incorporated-Change-Id: Ic9b86aebbae7796fc17db3bb69f6f9ee9fffb9c9 Incorporated-Change-Id: I1eec50a5b1b6206e5b3eab8d5f9fa891fecb4b25 Incorporated-Change-Id: Ic57a71bc4e69c57833396690fc3fb5453aa122b3 Incorporated-Change-Id: Iac3b8c183f633bf5ddc59acf340b477bd1eb88cc Incorporated-Change-Id: I47feb6a34c0e210f0ebb0edd4479550750e605d7 Incorporated-Change-Id: I22dbdcc9f135bbfc9ef05e13c801e88a78e64236 Incorporated-Change-Id: I250f203b3bb24e084ec1b499a923f7f66e638102 Incorporated-Change-Id: If3986df022273f20e109816f2752ce0254db4f10 Incorporated-Change-Id: I5a4c6c5a1894d1f6f5cff6e3475670c27bb97f28 Incorporated-Change-Id: I2905b470e91770d5951260e69a81d1910cb15972 Incorporated-Change-Id: I57cad032fc9002cb4b5013e4868215a60ebe412a Incorporated-Change-Id: I6801fce6cf95c37096a045799d30cbde843c509a Incorporated-Change-Id: I840e5d671c3df24cd01952b522288982ddfde80e Incorporated-Change-Id: I71f0324395e0135d8facdbf45c74c9682093104d Incorporated-Change-Id: I43fe1c4dc40caf5781457a0d4de83c6cb53ebc8f Incorporated-Change-Id: Iabeae6e01f9615be7c122d1e3fd719a1e53762d9 Incorporated-Change-Id: Ieb4ad968657ef8d2bf19e08ec954da5acc51372d Incorporated-Change-Id: I71e856382453343d8f593b330e9268d3b36b4dfa (cherry picked from commit 21502fe)
In various places, own version of iterating over the results are implemented, sometimes even faulty. The following functions were only getting up to vmware.maximum_objects objects (100 by default) vm_util.get_all_cluster_mors, vm_util.get_stats_from_cluster Previously, the results were fetched in batches of up to vmware.maximum_objects items. Using WithRetrieval yields an iterator to the results, which pages transparently to the next request. Receiver of the output of the results where changed to consume an iterator, where easily possible. Replace the quadratic algorithm in `ds_util._filter_datastores_matching_storage_policy` with one of O(n log(n)) runtime (cherry picked from commit fcabb04) Change-Id: I2ac02a0cf4569398455dd63cfa4e2380a71b8ab9 (cherry picked from commit 84ee36d)
This extracts getting the hosts and reservations from a cluster in get_stats_from_cluster into its own function get_hosts_and_reservations_for_cluster. For cross-vcenter vmotion, we need to specify hosts, and we want the same ones as we use elsewhere. (cherry picked from commit 05cd96b) Change-Id: I4f738e76553c3afefc735fb4b05f1a496c3c028f (cherry picked from commit fb3d5db)
Joined the collection of the cpu-info with the host-stats in order to avoid calling twice the property collector in different places to get information over all the host. Split the code into getting the data and aggregating it over the cluster. Partly to split the logic into more easily consumable parts, but it is also a preparation for exposing the individual ESXi hosts as hypervisors. Host in certain states (such as disconnected ones) do not report `quickStats` about memory usage, so we have to assume values here, which is better than failing updating the stats for all hosts. Since they are not available, both the usage as well as the free memory will not be added to the total for the cluster Disconnected hosts also don't provide other properties, so we handle that gracefully by defaulting them to 0. Change-Id: I383854bb0e956519e3bdc42121b59d43ca54743d Incorporated-Change-Id: I05b3a158a058034d9a50a6948cdf302769f2d0eb Incorporated-Change-Id: I1c53477e891b5b95859ca267fcad8cd1bff260ef (cherry picked from commit f6279b5)
The resource tracker iterates over all instances in _update_usage_from_instances, where it will the fetch for each instance the block-device-mappings in an rpc call. By fetching the block-device-mappings up-front, we get a single rpc/db call, instead of several small ones, where the latencies then add up If a block-device-mapping list cannot be retrieved for an instance, it will fall back to the old behaviour and fetch it individually. Change-Id: Ice9ab0a9c1b783c059687e1d992eea1f97cb3193 (cherry picked from commit 6caf72a) (cherry picked from commit ae68565)
Clients such as the cli expect cpu_info to be a json dict, so returning a string is breaking them. By aggregating the individual attributes, we also work around having different cpu models with the same flags in a cluster not yielding any information. Now we can see that the model mismatch, but the cpu feature flags are matching. Change-Id: Id552121b642ec90f6e06be09825ab7339531f9d6 (cherry picked from commit a84a785) (cherry picked from commit 273e2e7)
joker-at-work
approved these changes
Jul 2, 2026
fwiesel
approved these changes
Jul 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.