Redesign packagegroup-avocado-extra into opt-in feature groups#229
Open
jetm wants to merge 35 commits into
Open
Redesign packagegroup-avocado-extra into opt-in feature groups#229jetm wants to merge 35 commits into
jetm wants to merge 35 commits into
Conversation
Contributor
|
This looks great, I am building locally. In the meantime, could you add something in the top level docs with notes for maintainers on how to build various combinations of this from bare minimal to full featured. |
Contributor
Author
Done, in 7358674. New page at It walks the composition model first - the |
packagegroup-avocado-extra ships System/Base packages unconditionally, mixing always-on content with feature-gated bundles in one recipe. Introduce packagegroup-avocado-feature-system-base carrying the same set. The optee-client member keeps its MACHINE_FEATURES guard verbatim so unsupported targets resolve to an empty set. Signed-off-by: Javier Tia <javier@peridio.com>
The monolith bundles networking daemons (networkmanager, openssh, bluez5, wireguard-tools) unconditionally alongside feature-gated content, making selective composition impossible. Introduce packagegroup-avocado-feature-networking carrying the same set of System/Daemons + Applications/Internet members from the ungated bulk. Signed-off-by: Javier Tia <javier@peridio.com>
The monolith's GSTREAMER_PACKAGES bundle and opencv/v4l-utils are always-on but buried alongside feature-gated content in a 270-line recipe, making it impossible to opt into multimedia independently. Introduce packagegroup-avocado-feature-multimedia with the exact member list from the GSTREAMER_PACKAGES variable plus opencv and v4l-utils. Signed-off-by: Javier Tia <javier@peridio.com>
The monolith's PYTHON_PACKAGES block (22 python3-* entries) installs unconditionally on every distro build but is buried alongside qt, browser, and ai bundles, making python impossible to opt into independently. Introduce packagegroup-avocado-feature-python carrying the complete python3 runtime set from lines 183-206 of the monolith. Signed-off-by: Javier Tia <javier@peridio.com>
The monolith gates OPENGL_PACKAGES (weston, wayland, wpewebkit, cog, cage) on DISTRO_FEATURES "opengl" but this guard is invisible to callers who only see IMAGE_INSTALL append to the monolith. Add packagegroup-avocado-feature-graphics carrying the OPENGL_PACKAGES bundle with the "opengl" DISTRO_FEATURES guard preserved verbatim. Signed-off-by: Javier Tia <javier@peridio.com>
The monolith embeds QT5_PACKAGES and QT5_OPTIONAL_PACKAGES with nested guards for "opengl wayland", "x11", and "bluetooth" DISTRO_FEATURES inside its 270-line recipe, making Qt composition opaque. Add packagegroup-avocado-feature-qt migrating the full Qt5 block with all nested bb.utils.contains guards preserved verbatim from the source. Signed-off-by: Javier Tia <javier@peridio.com>
chromium-ozone-wayland is gated on "opengl wayland" DISTRO_FEATURES in the monolith's BROWSER_PACKAGES block, but the constraint is invisible to image composition without reading the full recipe. Add packagegroup-avocado-feature-browsers carrying chromium-ozone-wayland behind the same "opengl wayland" guard copied verbatim from the source. Signed-off-by: Javier Tia <javier@peridio.com>
The monolith carries BASLER_PACKAGES behind a TARGET_ARCH aarch64 guard nested inside an "opengl" guard, and librealsense2 behind its own "opengl" guard - both constraints are invisible without reading the full recipe. Add packagegroup-avocado-feature-cameras preserving both guards verbatim so camera packages resolve to the same set on unsupported targets. Signed-off-by: Javier Tia <javier@peridio.com>
The monolith's DEEPX_PACKAGES are gated on MACHINE_FEATURES "deepx" but the constraint is opaque to callers, and the four dx-* packages install on every machine that has any matching feature. Add packagegroup-avocado-feature-ai carrying dx-driver, dx-rt, dx-stream, dx-stream-sample behind the "deepx" MACHINE_FEATURES guard verbatim. Signed-off-by: Javier Tia <javier@peridio.com>
The monolith's AWS_PACKAGES (greengrass-bin, aws-iot-device-client) carry a TARGET_ARCH guard for aarch64/x86_64 only, but this constraint is opaque to image composition. Add packagegroup-avocado-feature-cloud-aws preserving the arch guard so non-aarch64/x86_64 targets still resolve to an empty set. Signed-off-by: Javier Tia <javier@peridio.com>
The monolith's JAVA_PACKAGES (openjdk-17-jdk, openjdk-17-jre) carry a TARGET_ARCH guard for aarch64/x86_64 only, buried inside the kitchen-sink recipe alongside unrelated bundles. Add packagegroup-avocado-feature-java preserving the arch guard so Java packages resolve to empty on unsupported architectures. Signed-off-by: Javier Tia <javier@peridio.com>
The monolith bundles docker, podman, podman-compose, k3s behind a DISTRO_FEATURES "virtualization" guard (stored in a misspelled variable CONATINER_PACKAGES), but the gate is invisible to image composition. Add packagegroup-avocado-feature-containers with the same four members behind the "virtualization" guard copied verbatim from the source. Signed-off-by: Javier Tia <javier@peridio.com>
The four always-on groups (system-base, networking, multimedia, python) use oe-core/meta-oe packages already in bblayers and need no vendor layer include, but they still need a kas fragment to append their token to AVOCADO_FEATURE_GROUPS so the composing packagegroup can resolve them. Add minimal local_conf_header-only fragments for each group, modeled on kas/feature/tpm.yml. Each appends its token with a leading space. Signed-off-by: Javier Tia <javier@peridio.com>
Existing client targets (nvidia, raspberrypi, rockchip, etc.) must keep receiving their full image content after the feature-group migration. A "complete" aggregator that includes every per-group fragment satisfies this without requiring each client to list all 12 fragments individually. Add kas/feature/complete.yml including all 12 kas/feature/<group>.yml files via the repo:/file: form, matching the include convention used by kas/machine/*.yml and kas/extra.yml. Signed-off-by: Javier Tia <javier@peridio.com>
kas/extra/wayland.yml provides meta-wayland through the all-layers kas/extra.yml aggregator, making it impossible to opt into graphics (the Wayland compositor stack) without pulling every other vendor layer. Add kas/feature/graphics.yml migrating the meta-wayland repo from extra/wayland.yml and appending the "graphics" token to AVOCADO_FEATURE_GROUPS. Signed-off-by: Javier Tia <javier@peridio.com>
kas/extra/qt5.yml provides meta-qt5 and its PACKAGECONFIG block through the all-layers aggregator, bundling Qt support into every distro build regardless of whether any Qt package is installed. Add kas/feature/qt.yml migrating the meta-qt5 repo and the qtbase/qtmultimedia PACKAGECONFIG lines from extra/qt5.yml, appending the "qt" token to AVOCADO_FEATURE_GROUPS. Signed-off-by: Javier Tia <javier@peridio.com>
kas/extra/browser.yml provides meta-browser (meta-chromium layer) through the all-layers aggregator, pulling Chromium build infrastructure into every distro build even when the browser packagegroup is not installed. Add kas/feature/browsers.yml migrating the meta-browser repo from extra/browser.yml and appending the "browsers" token to AVOCADO_FEATURE_GROUPS. Signed-off-by: Javier Tia <javier@peridio.com>
meta-basler-tools (requiring ACCEPT_BASLER_EULA) and meta-intel-realsense live in separate kas/extra overlays pulled through the all-layers aggregator. Opting into cameras requires both layers together with the EULA preserved. Add kas/feature/cameras.yml combining both vendor layers from extra/basler.yml and extra/realsense.yml, preserving ACCEPT_BASLER_EULA, and appending the "cameras" token to AVOCADO_FEATURE_GROUPS. Signed-off-by: Javier Tia <javier@peridio.com>
kas/extra/deepx.yml provides meta-deepx-m1 and a MACHINE_FEATURES deepx append through the all-layers aggregator, pulling the DeepX NPU layer into every build regardless of whether any deepx package is installed. Add kas/feature/ai.yml migrating the deepx vendor repo and its MACHINE_FEATURES local_conf_header from extra/deepx.yml and appending the "ai" token to AVOCADO_FEATURE_GROUPS. Signed-off-by: Javier Tia <javier@peridio.com>
kas/extra/aws.yml provides meta-aws through the all-layers aggregator, bundling AWS IoT build infrastructure into every distro build regardless of whether any AWS package is installed. Add kas/feature/cloud-aws.yml migrating the meta-aws repo from extra/aws.yml and appending the "cloud-aws" token to AVOCADO_FEATURE_GROUPS. Signed-off-by: Javier Tia <javier@peridio.com>
kas/extra/openjdk.yml chains through a vendor file to meta-openjdk-temurin via the all-layers aggregator, pulling the OpenJDK layer into every build. Add kas/feature/java.yml inlining the meta-openjdk-temurin repo from the vendor file, preserving the PREFERRED_VERSION local_conf_header, and appending the "java" token to AVOCADO_FEATURE_GROUPS. Signed-off-by: Javier Tia <javier@peridio.com>
meta-virtualization is already pulled by kas/feature/virtualization.yml for the DISTRO_FEATURES virtualization append, but a separate feature fragment is needed to co-locate the opt-in token with the layer include, matching the kernel-config fragment model used by all other feature groups. Add kas/feature/containers.yml mirroring virtualization.yml's repo/layer block and appending the "containers" token to AVOCADO_FEATURE_GROUPS. Signed-off-by: Javier Tia <javier@peridio.com>
The monolith recipe mixed always-present base content (avocado-img-* wrappers, linux-firmware) with feature-specific bundles in one 270-line recipe, making it impossible to opt into features selectively. Rewrite as a composing packagegroup: keep only the always-present base items as direct RDEPENDS, remove all bundle variables and migrated bulk members, and add a python inline expansion that maps each token in AVOCADO_FEATURE_GROUPS to a packagegroup-avocado-feature-<token> RDEPEND. The avocado-pkg-extra.bb wiring and PACKAGES/PACKAGE_ARCH remain unchanged. Signed-off-by: Javier Tia <javier@peridio.com>
kas/target/distro.yml and kas/target/sdk.yml included kas/extra.yml which aggregates all 14 vendor-layer overlays, meaning every distro/SDK build cloned meta-qt5, meta-browser, meta-basler-tools, and others regardless of whether any package from those layers was installed. Remove the kas/extra.yml include from both target files. Client vendor/machine compositions now include kas/feature/complete.yml to pull the layers they need; minimal compositions include only their fragments. Signed-off-by: Javier Tia <javier@peridio.com>
Ten client kas/vendor/*.yml and kas/machine/*.yml files set "opengl wayland"
DISTRO_FEATURES_EXTRA today and must keep receiving full feature-group content
after kas/target/{distro,sdk}.yml stops pulling kas/extra.yml.
Add a kas/feature/complete.yml include to each of the 10 files (nvidia,
raspberrypi, rockchip, renesas, synaptics, rubikpi, nxp-frdm, intel,
imx8mp-evk, imx93-evk). Existing DISTRO_FEATURES_EXTRA and PKG_EXTRA_INSTALL
lines are unchanged.
Signed-off-by: Javier Tia <javier@peridio.com>
The feed-validation suite (PR #223) exercises package feed contents and boot behavior, but had no cases for the new opt-in feature groups, making it impossible to assert that each group composes and installs correctly. Add one representative case per feature group (12 cases), selecting each package directly from its group recipe's RDEPENDS so every case references a real feed package. boot=yes for graphics, qt, and browsers (require a running compositor); boot=no for always-on and arch/feature-gated groups. Signed-off-by: Javier Tia <javier@peridio.com>
meta-avocado-distro carries recipes and bbappends that customize recipes from optional vendor layers (qtwebengine inherits meta-qt5's qmake5; the wpewebkit, cxxopts, openjdk-17, aws-iot-device-client and hpp-fcl bbappends extend recipes from meta-webkit, meta-wayland, meta-openjdk-temurin, meta-aws and meta-ros). With those layers always pulled, the coupling was invisible. Once a build opts out of a feature, the layer is gone and these abort the build: a missing inherited class halts parsing, a bbappend with no base recipe halts collection. Move each vendor-extending file under dynamic-layers/<collection>/ and add a BBFILES_DYNAMIC entry keyed on that vendor layer's BBFILE_COLLECTIONS name, so each is parsed only when its layer is present. A build that opts into the feature gets identical behavior; one that does not skips the file instead of failing. Signed-off-by: Javier Tia <javier@peridio.com>
Dropping kas/extra.yml from the targets removed every vendor layer, but three categories of layer must come back. Base layers back always-on content or unconditional meta-avocado-distro customizations (meta-fwup→fwup, meta-erlang→livebook, meta-atmel→cryptoauthlib in system-base; meta-swupdate for OTA), so provision them in base.yml on every build. meta-webkit provides wpewebkit, which is graphics content, so add it to the graphics fragment alongside meta-wayland. Add python-ai, clang and ros as layer-only fragments and include them, plus the existing feature fragments, in complete.yml so client targets reproduce the kas/extra layer set. deepx is deliberately excluded: it was never in kas/extra (it is MACHINE_FEATURES-gated for deepx hardware), so pulling it onto clients would add an unbuildable dx-driver bbappend. Signed-off-by: Javier Tia <javier@peridio.com>
kas/extra.yml and the per-vendor overlays it included are now unreferenced: the targets stopped including the aggregator, and the layers it provisioned moved to base.yml (base layers) and the per-feature fragments (gated content). Remove the aggregator and its 15 included overlays. kas/extra/deepx.yml stays: kas/machine/grinn-astra-1680-sbc.yml still includes it directly. Migrating that deepx machine to the ai feature fragment is a follow-up tracked separately. Signed-off-by: Javier Tia <javier@peridio.com>
The qemuarm64 avocado-complete build failed on a non-java composition with nativesdk-openjdk-17-jre having no provider. The cause is a runtime coupling the parse-level checks cannot see: kabtool and publishtool (meta-avocado KOS tools in the always-on system-base group) RDEPEND on openjdk-17-jre, so their nativesdk variants pull nativesdk-openjdk-17-jre into every SDK. openjdk is therefore base, not an opt-in feature. Provision meta-openjdk-temurin in base.yml and reduce the java fragment to a token-only opt-in (the full JDK content). Move the openjdk bbappends back to the static recipe tree and drop their BBFILES_DYNAMIC entry, since the layer is now always present. An authoritative RDEPENDS sweep of meta-avocado confirmed openjdk is the only always-on dependency on a vendor layer. Signed-off-by: Javier Tia <javier@peridio.com>
avocado-pkg-sdk-extra unconditionally pulled nativesdk-packagegroup-qt5- toolchain-host, which lives in meta-qt5. Once a build opts out of the qt feature meta-qt5 is absent, so avocado-complete failed with no provider for that toolchain. Gate it on AVOCADO_FEATURE_GROUPS containing qt, mirroring the target-side qt gating, so the qt5 SDK toolchain is pulled only when the qt feature is selected (and meta-qt5 is therefore present). Signed-off-by: Javier Tia <javier@peridio.com>
run-qemuarm64 never worked end to end. It called a find-library-path helper that does not exist, then ran qemu-system-native directly via LD_LIBRARY_PATH; the native binaries use the uninative glibc loader as their ELF interpreter, so on a non-Yocto host they fail with "cannot execute: required file not found". It also pointed at a stale image name and the wrong bios directory, and never passed -L, so QEMU could not find its option ROMs. Port the working approach from run-qemux86-64: invoke the native binaries through the uninative loader with --library-path, resolve the image as deploy/stone/_build/avocado-os-qemuarm64.img and the bios as deploy/stone/flash.bin, and pass -L for the QEMU data dir. The arm64 image now boots through TF-A, U-Boot and the kernel to a login prompt. Signed-off-by: Javier Tia <javier@peridio.com>
grinn-astra-1680-sbc still included kas/extra/deepx.yml and was never wired to complete.yml. Once kas/target/distro.yml stopped pulling kas/extra.yml, the board's AVOCADO_FEATURE_GROUPS resolved empty, so the composing packagegroup-avocado-extra reduced to base content only: it dropped the system-base, networking, multimedia, python, containers, java and cloud-aws groups the board shipped before, plus the dx-* deepx packages (deepx.yml set MACHINE_FEATURES deepx but never the ai token the new model requires). Include kas/feature/complete.yml (the 11 non-ai groups; graphics, qt, browsers and cameras resolve empty here without opengl/wayland) and kas/feature/ai.yml (the ai token plus the deepx layer and MACHINE_FEATURES, which complete.yml deliberately excludes). Together they reproduce the full monolith content the board built on scarthgap. Remove the now-unreferenced kas/extra/deepx.yml, the last kas/extra file. Signed-off-by: Javier Tia <javier@peridio.com>
The opt-in feature-group model this series introduces had no top-level documentation, so a maintainer had no single reference for which fragments exist or how to compose a build from bare minimal to full featured. Add docs/feature-groups.md covering the composition mechanics (AVOCADO_FEATURE_GROUPS tokens, kas colon-overlay syntax, the complete.yml umbrella), a per-group table with each group's vendor layer and DISTRO_FEATURES / MACHINE_FEATURES / arch gating, and worked kas build recipes spanning a bare machine, a few stacked groups, opengl-gated graphics/Qt/browsers, the full set, and deepx boards. Link it from the README build section. Signed-off-by: Javier Tia <javier@peridio.com>
Rebasing onto scarthgap pulled in CompuLab UCM-iMX8M-Plus SOM support, which factored the shared NXP i.MX base into kas/vendor/nxp.yml and added kas/vendor/compulab.yml and the ucm-imx8m-plus machine. Those files were written against the monolithic packagegroup-avocado-extra, which received full image content through the kas/extra.yml include in kas/target/distro.yml. This series removed that include, so ucm-imx8m-plus resolved an empty AVOCADO_FEATURE_GROUPS and would have regressed to base content only. Include kas/feature/complete.yml from kas/vendor/compulab.yml, mirroring the sibling kas/vendor/nxp-frdm.yml so the CompuLab family ships the same full set; ucm-imx8m-plus inherits it transitively and has no deepx, so it needs no ai fragment. Also drop the stray blank line the rebase merge left in nxp-frdm.yml's includes block. Signed-off-by: Javier Tia <javier@peridio.com>
jetm
force-pushed
the
eng-2003-packagegroup-feature-groups
branch
from
7358674 to
0f1531d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
packagegroup-avocado-extrais a single monolithic packagegroup that pulls everyoptional capability (Qt, Chromium/WebKit, GStreamer, Python, ROS, Java, containers,
AWS, cameras, AI) into every
avocado-completebuild. The only way to slim an imagewas to hand-roll
RDEPENDS:...:removeoverlays (the opt-out approach in #222). That isbackwards: a dev image pays for the heaviest recipes by default and has to subtract
them, and the pattern only scales to the few packages someone writes a remove for.
Solution
Invert the model to opt-in. Split the monolith into one
packagegroup-avocado-feature-<group>.bbper RPM category, and rewritepackagegroup-avocado-extraas a composing group that expands a kas-setAVOCADO_FEATURE_GROUPSvariable into the matching feature packagegroups. Each groupships a self-contained
kas/feature/<group>.ymlfragment that pulls its vendor layerand appends its token. Stack the fragments you want; a
complete.ymlumbrellareproduces today's full content for existing clients.
Key changes
packagegroup-avocado-feature-*.bbrecipes (system-base, networking, multimedia,python, graphics, qt, browsers, cameras, ai, cloud-aws, java, containers), migrated
from the monolith with every
DISTRO_FEATURES/MACHINE_FEATURES/TARGET_ARCHguard preserved verbatim.
packagegroup-avocado-extra.bbrewritten as always-on base content plus the tokenexpansion.
kas/feature/<group>.ymlper group, acomplete.ymlumbrella, and layer-onlyfragments for clang, python-ai and ros.
kas/target/{distro,sdk}.ymlno longer pullkas/extra.yml, and the deadkas/extra.ymlaggregator and all its overlays are removed (the grinn-astra deepxboard was the last
kas/extra/consumer; it now usescomplete.yml+ai.yml).meta-avocado-distrorecipes and bbappends that extend optional vendor recipes(qtwebengine, wpewebkit, cxxopts, openjdk-17, aws-iot-device-client, hpp-fcl) move
under
dynamic-layers/<collection>/gated byBBFILES_DYNAMIC, so a compositionwithout the layer parses cleanly instead of failing.
meta-swupdate, meta-openjdk-temurin) are provisioned unconditionally in
kas/base.yml. openjdk is base because the always-on KOS tools RDEPEND a JRE.kas/vendor/*.ymlandkas/machine/*.ymlfiles includecomplete.yml(deepx boards add
ai.ymlalongside it) so existing targets keep identical content.meta-avocado-qemu/scripts/run-qemuarm64so the arm64 image bootson a non-Yocto host (needed to validate the boot path).
Usage
Minimal dev image, opt into only what you need:
Full content for existing clients is unchanged (the vendor/machine files include
complete.yml):Validation
resolves with no dangling bbappends and no missing inherited classes, checked before
building.
system-base:networking:multimedia):gstreamer1.0-*present in the package feed;chromium-ozone-waylandandqtbaseabsent. Opt-in proven in both directions.
kernel to a login prompt.
imx8mp-evkandgrinn-astra-1680-sbc(bothinclude
complete.yml) the resolved kas composition is identical toscarthgap(samelayers and commits, same target, same
bblayers/env), and the feature-grouppartition is exact: base plus all groups RDEPEND the same package set as the old
monolith, zero packages added or dropped. Default client image content does not change.
Reviewer notes
Supersedes #222 (ENG-1975). That PR added opt-out overlays to subtract chromium/Qt from
dev images; this reverses to opt-in feature groups, which scales to every capability
rather than the two heavy ones. Per @mobileoverlord's point on #222, clients keep
building the full set by default through
complete.yml. Implements ENG-2003.