Skip to content

Enabling Kernel Caching - Intern Project - #6

Open
t-sakshsingh wants to merge 34 commits into
mainfrom
personal/t-sakshsingh/next
Open

Enabling Kernel Caching - Intern Project#6
t-sakshsingh wants to merge 34 commits into
mainfrom
personal/t-sakshsingh/next

Conversation

@t-sakshsingh

Copy link
Copy Markdown
Collaborator

This change introduces the following:

Transformers :

  1. kernel_source_packager: To build, package and store .deb kernel images from source in a local directory on a VM. These packages can be fetched anytime to be reused using this transformer. It searches the package with respect to the latest commit id and returns the path to the package.
  2. file_transfer_transformer: To upload the cache directory to a desired blob container from "builder VM" and download it on any "Test VM". This transformer can be used for any kind of file transfer.
  3. deb_package_installer: To install any .deb package. It automatically recognizes if there is a bootable kernel image package in it. If desired, it will reboot the machine with the new image after installing it.

Tools :

  1. Dpkg: To check validity of .deb packages and recognize bootable kernel images.

Runbook - transformers_azfiles_xfstests.yml :

  1. Runbook usage of the new transformers.
  2. End to End flow for fetching a kernel package from cache and running test on a custom kernel image on a new test VM.

Copilot AI and others added 30 commits June 4, 2025 15:03
* Initial plan for issue

* Add private network configuration to azure.yml runbook

Co-authored-by: LiliDeng <10083705+LiliDeng@users.noreply.github.com>

* Fix use_public_address default value to true

Co-authored-by: LiliDeng <10083705+LiliDeng@users.noreply.github.com>

* Add create_public_address variable to azure.yml runbook

Co-authored-by: LiliDeng <10083705+LiliDeng@users.noreply.github.com>

* Add fallback to default values for empty subnet and vnet names

Co-authored-by: LiliDeng <10083705+LiliDeng@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: LiliDeng <10083705+LiliDeng@users.noreply.github.com>
Add hook `azure_update_vm_capabilities` to allow editing of Azure VM SKU capabilities before being saved to cache.

This will allow capability information to be inserted by extensions for internal SKUs or the processing of additional SKU information such as disk and network performance. Alternatively, SKUs can be removed from the results to restrict the SKUs that LISA will deploy.
…soft#3839)

* Initial plan for issue

* Add source_address_prefixes parameter to azure.yml runbook with empty default

Co-authored-by: LiliDeng <10083705+LiliDeng@users.noreply.github.com>

* Support string input for source_address_prefixes parameter

Co-authored-by: squirrelsc <27178119+squirrelsc@users.noreply.github.com>

* Fix flake8 errors: remove trailing whitespace and use double quotes

Co-authored-by: LiliDeng <10083705+LiliDeng@users.noreply.github.com>

* Update documentation for source_address_prefixes parameter

Co-authored-by: LiliDeng <10083705+LiliDeng@users.noreply.github.com>

* Improve source_address_prefixes parsing and simplify documentation

- Enhanced _get_ip_addresses method with ast.literal_eval for robust string parsing
- Added proper error handling for malformed list strings
- Simplified documentation examples to show string and list formats only
- Removed variable file example as requested

Co-authored-by: LiliDeng <10083705+LiliDeng@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: LiliDeng <10083705+LiliDeng@users.noreply.github.com>
Co-authored-by: squirrelsc <27178119+squirrelsc@users.noreply.github.com>
Co-authored-by: LiliDeng <lildeng@microsoft.com>
If the capabilities dictionary is empty for any reason, the hook implementation would have no way to know which region is being queried.
If Modprobe fails to remove a module, run some
additional commands to debug why it failed
make detection accurate for root not allowed in /bin/sh
Add testcase to verify crash functionality having crash generated from mshv.

Separate the util functions used for kdump and Re-use kdump setup of
LISA to run the test and verify if crashdump is generated.

Signed-off-by: Smit Gardhariya <sgardhariya@microsoft.com>
* Added openSSL tool and testsuites to lay ground work for encryption and decryption tests

* Fixing tuple type error in actions

* made helper method private and added more to test description

* Removed duplicated line

* fixed comment
…kages (microsoft#3851)

* tool: add createrepo tool

Signed-off-by: Thien Trung Vuong <tvuong@microsoft.com>

* test/cvm_boot: optionally add package repository before upgrading packages

Signed-off-by: Thien Trung Vuong <tvuong@microsoft.com>

---------

Signed-off-by: Thien Trung Vuong <tvuong@microsoft.com>
With older baremetal server, lscpu wont expose virtualisation
feature details (VT-x or AMD-V)

This will cause CH test to be skipped when runs on Ubuntu-KVM
with older generation windows machine.

Fix it by checking if /dev/kvm exist

Signed-off-by: Smit Gardhariya <sgardhariya@microsoft.com>
* reboot stress test

* indentations

* flake

* median, info changed to debug

* flake - long line

* flake

* reboot stress test

* indentations

* flake

* median, info changed to debug

* flake - long line

* flake

* Review comment resolve
…t that skips the make when set to true so that kernel_source_packager doesn't run make twice when making the deb package.
…d a storage blob container.

1) upload - vm to blob
2) download-  blob to vm

It generates sas tokens and uses azcopy for file transfer
If a path is provided in the "cache_path" parameter, that path will be validated if it exists, if not, a directory will be created with the specified path.

Examples:
cache_path: "/my/storage/area"
# Result: Uses "/my/storage/area/cache" after validation

If no path is specified , it uses "/default/cache" to store cache artifacts.
* Added import of Dpkg tool in __init__.py in tools.

* modified file_transfer_transformer to accept container_sas_token from runbook to facilitate multi-file transfers.

* Created a runbook called 'transformers_azfiles_xfstests.yml' to :
       1. Use a build vm (new or existing) to build the kernel or fetch from cache using kernel_source_packager.
       2. upload the cache entries from build vm to a storage account and download them from storage account to test_vm using azure_file_transfer transformer.
       3.  Install the packages on the test vm using deb package installer and reboot.
       4. Run xfstests on the test vm
…ng deb package.

* Recognising the new bootable kernel image and set it as default boot kernel.
* fixed kernel source packager to return deb package directory instead of kernel image package.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.