[ci]: use shared buildenv_setup for the build environment#1974
Draft
theasianpianist wants to merge 9 commits into
Draft
[ci]: use shared buildenv_setup for the build environment#1974theasianpianist wants to merge 9 commits into
theasianpianist wants to merge 9 commits into
Conversation
Replace the inline apt-get / DownloadPipelineArtifact / dpkg / redis / rsyslog
steps in build-template.yml with a single invocation of the shared
`buildenv_setup` tool (hosted in sonic-swss-common/ci/), driven by declarative
config under build-env/:
- build-env/packages/base.yaml core build+link deps (dbus/pcsc/hiredis/
swig/zmq/autoconf-archive, arm-only libnl,
redis-server + configure-redis post_install)
- build-env/packages/tooling.yaml docbook/aspell/rsyslog + sswsyncd/rsyslog
- build-env/upstream-artifacts.yaml libswsscommon, common-libs (libnl+libyang3),
vpp -- downloaded via the Azure DevOps REST
API with the same run-selection semantics
- build-env/configure-redis-for-tests.sh, build.sh, Dockerfile, compose.yaml
build.sh is the single source of truth for the build command (autogen +
dpkg-buildpackage -Psyncd,vs,nopython2, honouring ASAN), used by both CI and
local dev (docker compose). The post-build steps (azsyslog rewrite, setcap,
make check, gcov, syslog publish) stay inline in the template as they run after
the build. Verified end-to-end in a sonic-slave-bookworm container against live
Azure artifacts.
TEMPORARY (PoC) pins, to be removed before merge: the sonic-swss-common clone
and sonic-slave image are pinned to master / lawlee/ci-unify, and upstream
artifacts are pinned to branch master (feature-branch artifacts don't exist).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Lawrence Lee <lawlee@microsoft.com>
Collaborator
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
…cade Remove the common-libs upstream-artifacts entry AND the libnl apt packages from sonic-sairedis. Both libyang3 and the SONiC-patched libnl family (including libnl-cli) are now inherited transitively via the sonic-swss-common cascade: buildenv_setup downloads the sonic-swss-common artifact, reads its build-env/, and walks sonic-swss-common's upstream-artifacts.yaml, which declares common-libs -> libnl + libyang3 (all downloaded from the common-libs artifact, never apt). This replaces sonic-sairedis' old split of amd64-libnl-from-common-lib + arm-libnl-from-apt. Flip the sonic-swss-common upstream to cascade_optional: false so a missing build-env/ in the resolved artifact WARNS loudly. NOTE: depends on the resolved sonic-swss-common artifact shipping a build-env/, which only happens once sonic-swss-common is built from a branch that carries the tool. The sonic-swss-common upstream is still pinned to branch:master here, so this commit is intentionally incomplete and MUST NOT be pushed until that pin is lifted (to lawlee/ci-unify / BUILD_BRANCH) AND a green sonic-swss-common lawlee/ci-unify run has published an artifact with build-env/. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Lawrence Lee <lawlee@microsoft.com>
Same fix as sonic-swss-common: .artifactignore was '**/* + !*.deb', publishing only debs. Un-ignore build-env/ so sonic-swss can cascade sonic-sairedis' dependency declarations via buildenv_setup once it is onboarded. (Not exercised in the current swss-common -> sairedis PoC, but keeps the repos consistent.) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Lawrence Lee <lawlee@microsoft.com>
Remove the TEMPORARY 'branch: master' pin on the sonic-swss-common upstream so it resolves on ctx.branch (--branch $(BUILD_BRANCH)). On a lawlee/ci-unify branch build that is lawlee/ci-unify, whose sonic-swss-common artifact now ships build-env/ (commit 309a2bb) -- so the cascade activates and sonic-sairedis inherits the SONiC-patched libnl + libyang3 from common-libs transitively. NOTE (PoC): a PR build uses BUILD_BRANCH=<target>=master, whose artifact has no build-env/ yet, so the cascade would no-op there; validate this PoC via a lawlee/ci-unify branch build, not the PR build. common-libs/vpp pipelines have no lawlee/ci-unify branch, so those upstreams stay pinned to master. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Lawrence Lee <lawlee@microsoft.com>
Collaborator
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Collaborator
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
…h build Get a manual sairedis lawlee/ci-unify branch build green: - Pin the sonic-slave container tag, sonic-swss checkout, and the common_libs / sonic-dash-api / sonic-buildimage-ubuntu22.04 runBranches to master in the BuildSwss/BuildDocker/Test templates (none of those exist on lawlee/ci-unify). - Keep the swss-common cascade artifacts on lawlee/ci-unify, but pin them EXPLICITLY (upstream-artifacts.yaml branch: + the 3 runBranches) rather than via BUILD_BRANCH: on a branch build BUILD_BRANCH = Build.SourceBranchName = 'ci-unify' (Azure returns only the last ref segment), which can't resolve the lawlee/ci-unify swss-common run. - Add the SONiC libnl runtime debs to the vstest host install (the jammy swss-common deb depends on the +b1sonicN libnl, absent from the host apt). All temporary; marked 'TEMPORARY (PoC)' with the value to restore before merge. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Lawrence Lee <lawlee@microsoft.com>
Collaborator
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Collaborator
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Add a 'repo' parameter (default: self) so a consumer pipeline can reuse this template to build sonic-sairedis by passing the sonic-sairedis repository resource. When overridden, the checkout uses path 's' so $(Build.SourcesDirectory) still points at the sairedis checkout and the rest of the template is unchanged. sonic-sairedis's own pipeline (repo: self) is unaffected. Enables sonic-swss-common's BuildSairedis stage to reuse this template + the buildenv_setup dependency cascade instead of maintaining its own copy (PoC). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Lawrence Lee <lawlee@microsoft.com>
Collaborator
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Two additions to the reusable build-template.yml, both no-ops for sairedis's own pipeline (repo=self, stage_swss_common=false): - Branch-match the resource repo to $(BUILD_BRANCH) when reused (repo != self), so a consumer builds the matching sairedis branch. Gated off for self (a self PR build is on the merge ref and must not be switched); falls back to the checked-out ref when the branch doesn't exist here. - Add stage_swss_common: when set, download the consumer's current-run swss-common artifact to <workspace>/staged/sonic-swss-common/ and pass --upstream-staged-dir + --required-staged-upstream so buildenv_setup uses THIS run's swss-common instead of the latest completed external run. Absent the staged dir, it falls back to download. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Lawrence Lee <lawlee@microsoft.com>
Align the reusable build-template.yml with two design follow-ups from the PoC review:
- Replace the single stage_swss_common bool with a staged_upstreams list param
({artifact, name} entries). A per-entry DownloadPipelineArtifact loop stages each
to <workspace>/staged/<name>/, and --required-staged-upstream flags are built from
the same list (passed as JSON via convertToJson) so any expected-but-missing
current-run bundle fails loudly. Lets any stage stage any set of upstreams (§6.6).
- Drop the runtime 'git checkout $(BUILD_BRANCH)' branch-match (gated repo!=self):
it contradicts the fail-closed design and is redundant -- the resource ref (resolved
at compile time) already checks out the reused repo on the right branch.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Lawrence Lee <lawlee@microsoft.com>
Collaborator
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
…ript
Two changes to sonic-sairedis' build-env config:
1. vpp upstream artifact: add install_env { VPP_INSTALL_SKIP_SYSCTL: 1 }. vpp's
maintainer scripts call sysctl, which fails in the unprivileged CI container;
this makes them skip it. buildenv_setup installs the vpp DEBs in their own
dpkg -i group as 'sudo env VPP_INSTALL_SKIP_SYSCTL=1 dpkg -i ...', after the
empty-env library providers.
2. redis dedup: delete sonic-sairedis' local configure-redis-for-tests.sh. The
post_install entry now resolves the script from the cascaded sonic-swss-common
bundle (single source of truth). This applies notify-keyspace-events AKE to
sonic-sairedis too; verified harmless by this pipeline.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Lawrence Lee <lawlee@microsoft.com>
Collaborator
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replace the inline apt-get / DownloadPipelineArtifact / dpkg / redis / rsyslog steps in build-template.yml with a single invocation of the shared
buildenv_setuptool (hosted in sonic-swss-common/ci/), driven by declarative config under build-env/:swig/zmq/autoconf-archive, arm-only libnl,
redis-server + configure-redis post_install)
vpp -- downloaded via the Azure DevOps REST
API with the same run-selection semantics
build.sh is the single source of truth for the build command (autogen + dpkg-buildpackage -Psyncd,vs,nopython2, honouring ASAN), used by both CI and local dev (docker compose). The post-build steps (azsyslog rewrite, setcap, make check, gcov, syslog publish) stay inline in the template as they run after the build. Verified end-to-end in a sonic-slave-bookworm container against live Azure artifacts.
TEMPORARY (PoC) pins, to be removed before merge: the sonic-swss-common clone and sonic-slave image are pinned to master / lawlee/ci-unify, and upstream artifacts are pinned to branch master (feature-branch artifacts don't exist).
Description of PR
Summary:
Fixes # (issue)
Type of change
Approach
What is the motivation for this PR?
Work item tracking
How did you do it?
How did you verify/test it?
Any platform specific information?
Documentation