test(#3928): add behaviour scenarios for URL-sourced harness dispatch by fullsend-ai-coder[bot] · Pull Request #5407 · fullsend-ai/fullsend

fullsend-ai-coder · 2026-07-21T10:14:58Z

Summary

Add behaviour test scenarios for URL-sourced custom agent harness dispatch, verifying that FetchAgentHarness URL resolution works end-to-end in the harness-dispatch pipeline.

Related Issue

Fixes #3928

Changes

New feature file (e2e/behaviour/features/dispatch/url-dispatch.feature): Three scenarios covering URL-sourced harness CEL trigger dispatch, mixed URL+local harness enumeration, and graceful skip on integrity hash failure.
New step definitions (pkg/behaviourtest/steps/url_dispatch.go): givenURLSourcedCustomHarness commits harness YAML to the config repo, computes SHA256 integrity hash, constructs raw.githubusercontent.com URL, and registers the agent in config with the URL source and allowed_remote_resources update. Supports badHash and skipAllowlist option variants for failure testing.
New fixture (e2e/behaviour/fixtures/dispatch/url-harness.yaml): Reference harness YAML for URL-sourced tests.
Unit tests (pkg/behaviourtest/steps/url_dispatch_test.go): Covers input validation, URL format construction, bad hash injection, allowlist skip, and containsPrefix helper.
Registry update (pkg/behaviourtest/steps/registry.go): Registers new URL dispatch step definitions.

Testing

Unit tests pass: go test ./pkg/behaviourtest/steps/...
All existing behaviour test step tests continue to pass
go vet clean
Secret scan clean
E2E behaviour tests require pool org infrastructure (run in CI)

Checklist

PR title follows Conventional Commits (correct type, ! for breaking changes)
Commits are signed off (DCO) — human and human-directed agent sessions only
I wrote this contribution myself and can explain all changes in it

Closes #3928

Post-script verification

Branch is not main/master (agent/3928-url-dispatch-behaviour-tests)
Secret scan passed (gitleaks — d9185cb4c47c583de659ff76e15a7ba167899d3a..HEAD)
PR body secret scan passed (gitleaks — no-git)
Pre-commit hooks passed (authoritative run on runner)
Tests ran inside sandbox

Add three behaviour test scenarios that verify URL-sourced custom agent harness dispatch via the harness-dispatch pipeline: 1. URL-sourced harness with CEL trigger: registers a harness YAML at a raw.githubusercontent.com URL with sha256 integrity hash, dispatches on matching label event, and verifies agent execution. 2. Mixed URL + local harnesses: registers both a local-path and a URL-sourced harness in the same config, verifies the local harness enumerates and matches independently. 3. Bad integrity hash: registers a URL-sourced harness with a deliberately wrong sha256 hash alongside a valid local harness. Verifies the failed URL agent is skipped at runtime while dispatch continues for the local agent. New step definitions in url_dispatch.go handle URL-sourced harness setup: commit harness YAML to config repo, compute sha256 hash, construct raw.githubusercontent.com URL, register in config with URL source and allowed_remote_resources update. Closes #3928

fullsend-ai-review · 2026-07-21T10:16:48Z

🤖 Finished Review · ✅ Success · Started 10:16 AM UTC · Completed 10:31 AM UTC
Commit: 2ff4c11 · View workflow run →

github-actions · 2026-07-21T10:17:04Z

Site preview

Preview: https://32857044-site.fullsend-ai.workers.dev

Commit: 2ff4c11a7c9a13d328de0eaecc054455b3362f58

codecov · 2026-07-21T10:20:06Z

Codecov Report

❌ Patch coverage is 52.72727% with 26 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
pkg/behaviourtest/steps/url_dispatch.go	53.70%	19 Missing and 6 partials ⚠️
pkg/behaviourtest/steps/registry.go	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

fullsend-ai-review · 2026-07-21T10:31:25Z

Review

Findings

Low

[dead-code] e2e/behaviour/fixtures/dispatch/url-harness.yaml:1 — Fixture file is not referenced anywhere in the codebase. It duplicates the inline harness YAML from Scenario 1's docstring but serves no purpose.
Remediation: Either remove the file or wire it into a test step.
[scope-question] e2e/behaviour/features/dispatch/url-dispatch.feature:58 — The third scenario tests hash failure but not allowlist failure, though issue test(behaviour): harness-dispatch URL-sourced custom agents #3928 says "fails allowlist or integrity hash." The step definition for allowlist failure (not in allowlist with:) is already registered but no scenario exercises it.
Remediation: Confirm whether a fourth scenario for allowlist failure is intended for this PR or a follow-up.
[naming-consistency] pkg/behaviourtest/steps/url_dispatch.go:112 — Function name containsPrefix suggests prefix matching but implementation uses exact match via slices.Contains.
Remediation: Rename to containsExact or inline the slices.Contains call.
[error-wrapping-consistency] pkg/behaviourtest/steps/url_dispatch.go:103 — Bare return err from yaml.Marshal differs from the wrapping pattern used elsewhere in the file. Note: dispatch.go uses the same bare pattern.
[missing-documentation] docs/guides/dev/behaviour-testing.md — The behaviour testing guide does not document the new Given a URL-sourced custom harness step, though existing steps like Given a custom harness are also not individually documented.
Remediation: Consider adding a brief mention of URL-sourced harness testing.

Labels: PR adds e2e behaviour test scenarios for URL-sourced harness dispatch

fullsend-ai-review · 2026-07-21T10:31:26Z

@@ -0,0 +1,9 @@
+agent: agents/triage.md


[low] dead-code

Fixture file is not referenced anywhere in the codebase. It duplicates the inline harness YAML from Scenario 1's docstring but serves no purpose. May be leftover scaffolding from development.

Suggested fix: Either remove the file or wire it into a test step if it was intended to be served as the URL-sourced harness content.

fullsend-ai-review · 2026-07-21T10:31:26Z

+      | Prove local execution    | write_fixture| output/dispatch-local-ok.json, fixtures/dispatch/ok.json |
+    And an issue
+    When the issue is labeled "ready-for-mixed-ping"
+    Then the harness "local-ping" workflow completes successfully


[low] scope-question

The third scenario tests hash failure but not allowlist failure, though issue #3928 says 'fails allowlist or integrity hash'. The code implements the allowlist failure path (skipAllowlist option with a registered step 'not in allowlist with:'), but no scenario exercises it yet.

Suggested fix: Confirm whether a fourth scenario for allowlist failure is intended for this PR or a follow-up. The step definition is already wired.

fullsend-ai-review · 2026-07-21T10:31:26Z

+}
+
+// containsPrefix reports whether the list already contains the given prefix.
+func containsPrefix(list []string, prefix string) bool {


[low] naming-consistency

Function name containsPrefix suggests prefix matching but implementation uses exact match via slices.Contains.

Suggested fix: Rename to containsExact or inline the slices.Contains call at the usage site.

fullsend-ai-review · 2026-07-21T10:31:26Z

+
+	merged, err := yaml.Marshal(cfg)
+	if err != nil {
+		return err


[low] error-wrapping-consistency

Bare return err from yaml.Marshal differs from the consistent error wrapping pattern used throughout the rest of the file. dispatch.go uses the same bare pattern.

ifireball · 2026-07-21T23:13:51Z

/fs-fix The new step functions assume the existence of the .fullsend repo in the test org, that repo, belongs to the deprecated per-org install mode while the behaviour tests are based on per-repo mode. A step for creating an arbitrarily named new repo should be introduced instead (if no such step exists already, similar to the fork repo creation step) and called from the "background" section. A supplementry step for pushing a harness file to that repo should then be introduced as well (again, if we don't already have a step for pushing a harness to a given arbitrary repo)

Fix the comments by the review agent as well.

fullsend-ai-coder Bot requested a review from a team as a code owner July 21, 2026 10:14

fullsend-ai-coder Bot added the ready-for-review Agent PR ready for human review label Jul 21, 2026

github-actions Bot deployed to site-preview July 21, 2026 10:17 View deployment

fullsend-ai-review Bot approved these changes Jul 21, 2026

View reviewed changes

fullsend-ai-review Bot added ready-for-merge All reviewers approved — ready to merge component/e2e End-to-end tests component/dispatch Workflow dispatch and triggers testing labels Jul 21, 2026

ifireball self-assigned this Jul 21, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

test(#3928): add behaviour scenarios for URL-sourced harness dispatch#5407

test(#3928): add behaviour scenarios for URL-sourced harness dispatch#5407
fullsend-ai-coder[bot] wants to merge 1 commit into
mainfrom
agent/3928-url-dispatch-behaviour-tests

fullsend-ai-coder Bot commented Jul 21, 2026

Uh oh!

fullsend-ai-review Bot commented Jul 21, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 21, 2026

Uh oh!

codecov Bot commented Jul 21, 2026

Uh oh!

fullsend-ai-review Bot commented Jul 21, 2026

Uh oh!

fullsend-ai-review Bot Jul 21, 2026

Uh oh!

fullsend-ai-review Bot Jul 21, 2026

Uh oh!

fullsend-ai-review Bot Jul 21, 2026

Uh oh!

fullsend-ai-review Bot Jul 21, 2026

Uh oh!

ifireball commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

fullsend-ai-coder Bot commented Jul 21, 2026

Summary

Related Issue

Changes

Testing

Checklist

Post-script verification

Uh oh!

fullsend-ai-review Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 21, 2026

Site preview

Uh oh!

codecov Bot commented Jul 21, 2026

Codecov Report

Uh oh!

fullsend-ai-review Bot commented Jul 21, 2026

Review

Findings

Low

Uh oh!

fullsend-ai-review Bot Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

fullsend-ai-review Bot Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

fullsend-ai-review Bot Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

fullsend-ai-review Bot Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

ifireball commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

fullsend-ai-review Bot commented Jul 21, 2026 •

edited

Loading