Add Wrynose (Yocto 6.0) support and IoT Edge 1.6.0-rc.1 recipes#211
Closed
jlian wants to merge 9 commits into
Closed
Add Wrynose (Yocto 6.0) support and IoT Edge 1.6.0-rc.1 recipes#211jlian wants to merge 9 commits into
jlian wants to merge 9 commits into
Conversation
Introduce build infrastructure for the Wrynose (Yocto 6.0) release so the
layer can target it in parallel with the existing Scarthgap (5.0) and
Kirkstone setups, using a per-template model rather than separate branches.
- conf/layer.conf: extend LAYERSERIES_COMPAT to include wrynose.
- conf/templates/wrynose/: new bblayers.conf.sample + local.conf.sample.
Wrynose splits poky into separate oe-core/bitbake/meta-yocto repos, so the
layer set and PREFERRED_VERSION pins differ from the combo-poky templates.
- conf/templates/{scarthgap,kirkstone}/local.conf.sample: pin
PREFERRED_VERSION for the IoT Edge packages to the 1.5.x recipes so the
existing releases keep selecting 1.5 while wrynose selects 1.6.
- scripts/fetch.sh: support the wrynose split-repo layout (oe-core + bitbake
+ meta-yocto under poky/, meta-poky/meta-yocto-bsp symlinked), retaining the
combo-poky path for scarthgap/kirkstone.
- README.md: document the supported Yocto/IoT Edge version matrix.
Add a parallel recipe set for IoT Edge 1.6.0-rc.1 (aziot-edge daemon + aziot-identity-service 1.6.0-rc.1) that coexists with the existing 1.5.x recipes. Wrynose selects 1.6 via PREFERRED_VERSION; Scarthgap/Kirkstone keep 1.5. Version-specific bits live in the versioned .inc/.bb; truly-common bits stay in the shared .inc. - New per-package recipes for iotedge, aziot-edged, aziot-keys, aziotctl, aziotd: versioned .bb + .inc + auto-generated -crates.inc (1.6.0-rc.1 from the prerelease channel; SRCREVs pinned to the 1.6.0-rc.1 tags). - Regenerated the panic=abort -> unwind patches against the restructured 1.6 Cargo.toml ([workspace.dependencies]/[workspace.lints]); dropped the obsolete Rust-1.78 dead_code workaround that no longer applies to 1.6. - aziotd: pre-generated keys.generated.rs (bindgen 0.72.1) for the 1.6 C headers, placed in the version-specific FILESPATH dir. - 1.5 .inc files: move version-divergent patch SRC_URI into the versioned includes so 1.5 and 1.6 recipes can coexist cleanly.
Resolve the build breakages that surface when building the 1.6.0-rc.1 recipes
against the Wrynose (Yocto 6.0) toolchain. Each fix is scoped to avoid
regressing the Scarthgap/1.5 build.
iot-identity-service.inc (1.6 IIS split-crate layout):
- Synthesize a minimal IIS workspace root and point EXTRA_OECARGO_PATHS at
${UNPACKDIR} (Yocto 6.0 sets UNPACKDIR=${WORKDIR}/sources for fetched
crates). 1.6 uses { workspace = true } deps/lints across detached-fetched
crates, which otherwise fail do_compile with "failed to find a workspace
root" / "failed to read directory".
- Extend DEBUG_PREFIX_MAP to remap the ${UNPACKDIR} prefix so sibling-crate C
build-script debug paths (openssl-sys2, aziot-key-openssl-engine) don't leak
into the -dbg package and fail do_package_qa [buildpaths]. Longest-prefix
match keeps the existing ${S} mapping intact.
aziot-keys: set CARGO_INSTALL_LIBRARIES=1 so the libaziot_keys.so cdylib is
installed to ${rustlibdir} (Yocto 6.0 gates .so/.rlib install behind this);
otherwise cargo_do_install finds nothing and fails.
aziotd: add -Wno-unused-command-line-argument to CFLAGS. aziotd builds with
TOOLCHAIN="clang"; the OE rust-cc wrapper bakes linker flags (-Wl,...) into the
C compiler, and clang errors on them under -Werror during cc-rs compile-only
steps (gcc, used by aziot-keys, ignores them). Keeps -Werror=format-security.
Also point the service-install paths at ${UNPACKDIR} for Yocto 6.0.
edgelet (iotedge + aziot-edged): pin sysinfo to 0.38 (MSRV 1.88). The RC pulls
sysinfo 0.39, which requires rustc 1.95, but Wrynose poky ships rustc 1.94.1.
0.38 is the latest release before the 1.95 bump and is API-compatible with the
edgelet call sites (no source changes). 0002 patch rewrites edgelet
Cargo.toml + Cargo.lock to match the --frozen cargo build.
Also fixes the lack-of-whitespace-around-assignment QA warnings (export VAR=...
-> export VAR = "...") across the shared .inc files.
Wire the GitHub Actions workflows to cover the new Wrynose/1.6 target in addition to the existing Scarthgap/1.5 build. - ci-build.yml: add the wrynose template to the build matrix. - release.yml: produce 1.6.0-rc.1 artifacts for wrynose alongside the 1.5 release artifacts. - build-devcontainer.yml: publish the (Yocto-release-agnostic) devcontainer image under both scarthgap and wrynose tags so the build workflow can reference a per-template tag.
The "Recipe consistency check" CI regenerates the 1.5 recipes with
scripts/update-recipes.sh and fails on any drift. The coexistence change
hand-edited the generated 1.5 incs to add edgelet patch SRC_URI and an
aziotd keys.generated.rs do_compile:prepend, which the generator does not
emit, so the check failed.
Make the generator the source of truth instead of hand-editing its output:
- update-recipes.sh now wires the edgelet source-compatibility patches into
each generated aziot-edged/iotedge version inc. It does not hardcode the
list. It wires whatever .patch files exist in the recipe's patch dir for
that version, mirroring bitbake FILESPATH precedence: the version dir
recipes-core/<pkg>/<pkg>-<ver>/ first, then the shared files/ dir. That is
the same order the build uses to resolve file://, so the generated wiring
matches what the build picks up. Today 1.5.x patches live in files/ and
1.6.x patches live in the version dir, so each series gets its own set
(1.5 keeps the Rust 1.78 dead_code workaround, 1.6 does not).
- The aziotd keys.generated.rs prepend moves into the shared aziotd.inc using
${UNPACK_ROOT}, which already resolves to ${WORKDIR} on Scarthgap and
${UNPACKDIR} on Wrynose. It is the same install for every aziotd version, so
it belongs in the shared inc and the generated version inc stays at just the
export VERSION line. This also removes the duplicate prepend the 1.5 and 1.6
incs each carried.
Patch and asset files stay on disk in their version dirs by hand. The
generator only writes the SRC_URI wiring, so committed recipes match the
generator output and the consistency check stays green. The 1.6.0-rc.1
recipes remain hand-authored because the generator only fetches released
versions; CI regenerates 1.5.35 only, so they do not trip the check.
After running update-recipes.sh --iotedge-version 1.5.35 --skip-validate,
git diff --exit-code recipes-core/ is clean.
The "Recipe consistency check" enforces that recipes are generated, never
hand-edited. The 1.5 (Scarthgap) recipes already come from
scripts/update-recipes.sh, but the 1.6.0-rc.1 (Wrynose) recipes were
hand-authored because the generator only knew the old single-version
(Scarthgap) recipe shape. 1.6 is headed for the LTS channel, so it must be
generator-managed too: then the consistency check covers it and the daily
watch-upstream bot keeps working when LTS flips to 1.6.
Teach the generator the Wrynose shape via a template-driven switch:
- Add 'wrynose' to the allowed --template set. The recipe shape is decided by
series: scarthgap/kirkstone keep the legacy shape (shared <pkg>-crates.inc
name, explicit S = "${WORKDIR}/git"); wrynose uses the new shape (per-version
<pkg>-<ver>-crates.inc so 1.5 and 1.6 coexist in one recipe dir, and no S
line because Yocto 6.0 sets S = ${UNPACKDIR}/${BP} by default and the old
explicit value hard-errors). Default template stays scarthgap, so the CI
consistency step (no --template) is unchanged.
- Resolve the release channel by version instead of hardcoding lts. At the
1.6.0-rc.1 tag the 1.6 daemon lives in the prerelease channel (lts still
points at 1.5.21), so the generator now picks the channel whose aziot-edge
product version matches the requested release, preferring lts and falling
back to lts when nothing matches. 1.5.35 still resolves to lts -> 1.5.21 /
1.5.6 exactly as before.
- Compute LIC_FILES_CHKSUM md5s for LICENSE and THIRDPARTYNOTICES from the
fetched source instead of hardcoding the 1.5 values. THIRDPARTYNOTICES is
regenerated each release, so its md5 differs by version (1.6 is c613cff9...,
1.5 stays 11604c61...). Computing it makes the recipe correct for any release.
- Apply each version's source-compatibility patches to the throwaway source
checkout before reading Cargo.lock for crates.inc. The cargo build runs with
--frozen, so the fetched crate set is the one in the patched lock. The 1.6
sysinfo-0.38 pin rewrites edgelet/Cargo.lock (drops the macOS-only objc2
stack), so the generated crates.inc must reflect that. The 1.5 patches only
touch Cargo.toml / source, so 1.5 crates.inc is byte-identical.
- Emit the aziot-keys CARGO_INSTALL_LIBRARIES wiring for the wrynose shape
only. aziot-keys builds a cdylib, and the Yocto 6.0 cargo bbclass needs that
flag to install the .so. Scarthgap installs it without the flag, so its
recipe is unchanged.
The 1.6 patch FILES (0001-Remove-panic-abort, 0002-Pin-sysinfo-0.38) and the
keys.generated.rs asset stay hand-crafted in their version dirs. The generator
only wires and applies them, so committed recipes match generator output and
the consistency check stays green.
Normalizations applied to the previously hand-authored 1.6 recipes so they
match generator output:
- iotedge-1.6.0-rc.1.inc / aziot-edged-1.6.0-rc.1.inc: replace the bespoke
per-patch comment + two separate SRC_URI lines with the generator's standard
combined patch-wiring block (same two patches, same order, same form 1.5
uses).
- aziotd-1.6.0-rc.1.inc: drop the informational keys.generated.rs comment; the
install logic already lives in the shared aziotd.inc, so the version inc is
just the export VERSION line the generator emits.
- iotedge-1.6.0-rc.1-crates.inc / aziot-edged-1.6.0-rc.1-crates.inc: remove the
five stale objc2 crates (dispatch2, objc2, objc2-encode, objc2-foundation,
objc2-open-directory). The committed crates.inc had sysinfo pinned to 0.38.4
but still listed the objc2 stack that the 0002 patch removes from the lock,
so it was internally inconsistent with its own patch. The generated file is
derived from the actual patched lock, which references only
objc2-core-foundation and objc2-io-kit.
Verified: update-recipes.sh --iotedge-version 1.6.0-rc.1 --template wrynose and
update-recipes.sh --iotedge-version 1.5.35 (default scarthgap) both reproduce
recipes-core/ byte-identically (git diff --exit-code clean).
The build job built only one template per run, chosen from the branch or tag. On a Wrynose pull request it still built Scarthgap, so a green "Build and validate" check did not prove that the 1.6 line fetches, builds, and boots. With 1.5 and 1.6 now shipping as overlapping LTS lines, the check must cover both. Make the build job a matrix over Yocto templates: - Add a set-matrix step in check-changes that outputs the templates to build. Pull requests and pushes build both LTS lines (scarthgap for 1.5, wrynose for 1.6). Release tags, which release.yml passes as inputs.ref, collapse to the single template that matches the tag, so a release does not double-build or build the wrong line. - The build job uses strategy.matrix.template with fail-fast: false and passes matrix.template to fetch.sh, build.sh, and validate-qemu.sh, and to the per-template artifact names. - status-check keeps the needs.build.result == 'success' check. A matrix job aggregates to success only when every leg succeeds, so a single failed line fails the overall check. Keep kirkstone out of the pull request matrix. It builds the same 1.5 recipe as scarthgap, so a third leg adds build time without testing a distinct recipe shape. The single template output still maps kirkstone for any branch or release that needs it. Name the devcontainer image with a release-neutral tag. The image only provides host build tools, and OE-core fetches the per-template uninative and cross toolchain at build time, so the image content does not depend on the Yocto release. Publish it as :latest and reference :latest for every matrix leg, which also removes the :wrynose chicken-and-egg (that tag never existed). build-devcontainer.yml still pushes :scarthgap as a transitional alias for one publish cycle so existing references keep working during the rename. Note: :latest must be published once before the matrix build can pull it. The .devcontainer files are unchanged here, so build-devcontainer does not run on this change. A maintainer should run build-devcontainer once via workflow_dispatch (or merge any later .devcontainer change) to publish :latest. The :scarthgap alias keeps current builds working until then.
The daily upstream watcher assumed a single recipe line. With 1.5 and 1.6 recipes coexisting in recipes-core/ during LTS overlap, it had three failures and pulled the wrong devcontainer. Fix the version comparison crash. compare_versions split versions on "." and ran arithmetic on each field. A prerelease like 1.6.0-rc.1 produced a patch field of "0-rc.1", which raised a bash arithmetic syntax error. Under set -e that killed the whole job. Replace the arithmetic with a small dependency-free Python comparator that orders by the numeric release first, then applies the semver rule that a prerelease sorts below its final release (1.6.0-rc.1 < 1.6.0) and rc.1 < rc.2. Pure 1.5.x numeric comparisons are unchanged. GNU sort -V is not used for the decision because it orders a prerelease above its final release, which would make the bot miss an rc to final upgrade. Scope the recipe lookup to the tracked line. get_recipe_version and get_recipe_release picked the highest recipe on disk, which is now 1.6.0-rc.1 even while the lts channel still ships 1.5.x. Derive the major.minor from the lts channel daemon version and compare against that line only, so the bot tracks 1.5 today and 1.6 once the channel advances, without the other line interfering. Scope update-recipes.sh --clean to the line being regenerated. The clean deleted every *_*.bb and *-[0-9]*.inc across all versions, which would wipe the other LTS line on a single-line update. Restrict the delete to the major.minor derived from the daemon and IIS versions, so regenerating 1.5 never removes 1.6 recipes and vice versa. The recipe consistency check does not use --clean, and a normal regeneration overwrites the same files, so generated output stays byte-identical for both lines. Make the watcher template-aware. check-upstream now emits a template output (1.6.* maps to wrynose, otherwise scarthgap), and watch-upstream.yml passes it as --template to update-recipes.sh. The container moves to the release-neutral :latest tag, since the image provides host tools only and the Yocto line comes from --template. Verified with the live product-versions.json (exits 0, tracks the 1.5 line), a unit harness over compare_versions across rc and numeric inputs, and a simulation of the 1.5 to 1.6 channel transition that confirms the rc to final upgrade is detected and the correct template is chosen.
Member
Author
|
Superseded by #212. Re-opened as a same-repo branch (Azure/meta-iotedge:feat/wrynose-iotedge-16-rc1) so the self-hosted build runner and the devcontainer image publish work without fork-PR restrictions. Same commits, same content. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
This adds Wrynose (Yocto 6.0) support to the layer and a parallel IoT Edge 1.6.0-rc.1 recipe set, without disturbing the existing Scarthgap/1.5 and Kirkstone builds. Wrynose selects 1.6 through PREFERRED_VERSION; Scarthgap and Kirkstone keep 1.5. Version-specific bits live in the versioned
.inc/.bb, and the shared.incfiles stay common to both versions.The full Wrynose build is green (iotedge + aziot-edged RPMs produced,
libaziot_keys.soinstalls), and both the wrynose and scarthgap templates parse with 0 errors.The commits
The branch is organized so each commit stands on its own and bisects cleanly:
kirkstone scarthgap wrynose..inccoexistence tweaks). No build fixes here on purpose, so it reads as "here are the recipes."The Yocto 6.0 build fixes (commit 3)
Each fix is scoped so it can't regress the Scarthgap/1.5 build, since several of these files are shared:
{ workspace = true }deps/lints, sodo_compileneeds a synthesized workspace root and the cargo paths pointed at the unpack dir. Yocto 6.0 introducedUNPACKDIR(${WORKDIR}/sources); Scarthgap doesn't have it. So the paths go through aUNPACK_ROOTfallback that resolves toUNPACKDIRon Wrynose andWORKDIRon Scarthgap. The debug-prefix-map for the sibling crate build paths is guarded the same way (it's a no-op on Scarthgap).CARGO_INSTALL_LIBRARIES = "1"so thelibaziot_keys.socdylib actually installs. Yocto 6.0 gates.so/.rlibinstall behind this knob, and aziot-keys has no binary, so without it cargo install finds nothing.-Wno-unused-command-line-argumentto CFLAGS. aziotd builds withTOOLCHAIN="clang", and clang errors on the OE linker flags that the rust-cc wrapper bakes into the C compiler during compile-only cc-rs steps (gcc, used by aziot-keys, just ignores them). This keeps-Werror=format-securityintact. Service-install paths also go throughUNPACK_ROOT.Cargo.toml+Cargo.lockto match the--frozenbuild.Also cleans up the lack-of-whitespace-around-assignment QA warnings across the shared
.incfiles.Notes
${WORKDIR}paths whenUNPACKDIRisn't defined.