feat: triage variable to control auto-coding by deboer-tim · Pull Request #331 · fullsend-ai/agents

deboer-tim · 2026-07-21T11:54:33Z

Adds a new variable to the triage agent that allows you to control whether it automatically triggers the coding agent. With the variable set to true (default) you get the current behaviour: the issue is labelled as ready-to-code, which triggers the coding agent automatically. When the variable is set to false, issues will be marked with triaged instead, and a human (or alternate agent) can decide when to add the ready-to-code label or /fs-code comment.

Fixes #1750.

Adds a new variable to the triage agent that allows you to control whether it automatically triggers the coding agent. With the variable set to true (default) you get the current behaviour: the issue is labelled as `ready-to-code`, which triggers the coding agent automatically. When the variable is set to false, issues will be marked with `triaged` instead, and a human (or alternate agent) can decide when to add the `ready-to-code` label or `/fs-code` comment. Fixes #1750. Signed-off-by: Tim deBoer <git@tdeboer.ca>

github-actions · 2026-07-21T11:54:43Z

Functional tests did not run

The ok-to-test label was removed because new commits landed after it was applied. A maintainer must re-apply ok-to-test after reviewing the latest changes.

Note: ok-to-test was cleared due to new commits.

qodo-code-review · 2026-07-21T11:55:37Z

PR Summary by Qodo

Add TRIAGE_AUTO_CODE flag to gate ready-to-code promotion

✨ Enhancement 🧪 Tests 📝 Documentation ⚙️ Configuration changes 🕐 20-40 Minutes

AI Description

• Add TRIAGE_AUTO_CODE to toggle automatic ready-to-code labeling for low-risk issues.
• Wire the variable through the triage harness/env and update post-triage label application.
• Document the new setting and add regression tests for the disabled auto-code path.

Diagram

graph TD
  A["CI workflow env"] --> B["env/triage.env"] --> C["harness/triage.yaml"] --> D["Triage sandbox run"] --> E["scripts/post-triage.sh"] --> F{Auto-code?}
  F -->|"TRIAGE_AUTO_CODE=true"| G["Apply ready-to-code"] --> H{{"Code agent trigger"}}
  F -->|"TRIAGE_AUTO_CODE=false"| I["Apply triaged"]
  E --> J{{"GitHub issue labels"}}

  subgraph Legend
    direction LR
    _file["File/config"] ~~~ _dec{Decision} ~~~ _ext{{External/trigger}}
  end

High-Level Assessment

The following are alternative approaches to this PR:

1. Triage-result field (schema-driven)

➕ Makes the auto-code decision explicit in agent output, not inferred from env
➕ Potentially easier to audit/debug per-issue decisions
➖ Requires schema changes + agent prompt updates + validation loop updates
➖ More invasive than an env-level feature flag for a workflow concern

2. Configurable allowlist of auto-code categories

➕ More flexible than a boolean (e.g., allow auto-code for docs but not bugs)
➕ Avoids future PRs for category-by-category policy tweaks
➖ More configuration surface area (parsing/validation) for a small initial need
➖ Harder to document and test exhaustively compared to a single flag

Recommendation: Keep the env-var feature flag approach: it’s the lowest-friction control point for CI policy and preserves current default behavior (true). If teams ask for finer policy control, evolve this into a category allowlist; avoid a schema-level change unless the decision must be persisted in the triage result for downstream consumers.

Files changed (5) +110 / -6

Enhancement (1) +26 / -6

post-triage.shGate ready-to-code promotion behind TRIAGE_AUTO_CODE +26/-6

Gate ready-to-code promotion behind TRIAGE_AUTO_CODE

• Adds 'TRIAGE_AUTO_CODE' (defaulting to true) and updates the sufficient-action label logic so bug/documentation/performance categories apply 'triaged' instead of deferring 'ready-to-code' when auto-coding is disabled.

scripts/post-triage.sh

Tests (1) +72 / -0

post-triage-test.shAdd tests for TRIAGE_AUTO_CODE=false label outcomes +72/-0

Add tests for TRIAGE_AUTO_CODE=false label outcomes

• Introduces a helper to run post-triage with 'TRIAGE_AUTO_CODE=false' and asserts that bug/docs/perf categories receive 'triaged' (and not 'ready-to-code') while still applying their category labels where applicable.

scripts/post-triage-test.sh

Documentation (1) +10 / -0

triage.mdDocument TRIAGE_AUTO_CODE variable and behavior +10/-0

Document TRIAGE_AUTO_CODE variable and behavior

• Adds a Variables section documenting 'TRIAGE_AUTO_CODE', its default ('true'), and how it switches low-risk issues between 'ready-to-code' and 'triaged'. Includes guidance on setting it in the CI workflow env and how it propagates to the pipeline.

docs/triage.md

Other (2) +2 / -0

triage.envExport TRIAGE_AUTO_CODE with default true +1/-0

Export TRIAGE_AUTO_CODE with default true

• Exports 'TRIAGE_AUTO_CODE' from the env file with a 'true' default, ensuring it is available to the sandbox/harness without extra configuration.

env/triage.env

triage.yamlPass TRIAGE_AUTO_CODE into the runner environment +1/-0

Pass TRIAGE_AUTO_CODE into the runner environment
• Wires 'TRIAGE_AUTO_CODE' into the forge runner env so the post-triage script can read it from the runner environment.
harness/triage.yaml

qodo-code-review · 2026-07-21T12:01:35Z

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📜 Skill insights (1)

Context used

✅ Compliance rules (platform): 55 rules

✅ Skills: 4 invoked

code-review

code-implementation

pr-review

docs-review

1. Protected paths modified 📜 Skill insight § Compliance

Description

This PR changes protected governance/infrastructure paths (scripts/, harness/) which require
explicit human review and must not be auto-approved. Merging without the required approval controls
increases risk of unintended workflow/automation changes.

Code

scripts/post-triage.sh[R315-332]

+    #
+    # Set TRIAGE_AUTO_CODE=false to disable the automatic ready-to-code
+    # promotion. When disabled, low-risk categories receive "triaged" instead,
+    # requiring human review before the code agent runs.
+    TRIAGE_AUTO_CODE="${TRIAGE_AUTO_CODE:-true}"
    CATEGORY=$(jq -r '.triage_summary.category // "unknown"' "${RESULT_FILE}")
    echo "Category: ${CATEGORY}"
    case "${CATEGORY}" in
      bug)
        echo "Applying bug label..."
        add_label "bug"
-        echo "Deferring ready-to-code label (${CATEGORY}) until after label_actions..."
-        DEFERRED_LABEL="ready-to-code"
+        if [[ "${TRIAGE_AUTO_CODE}" == "false" ]]; then
+          echo "TRIAGE_AUTO_CODE=false — applying triaged label instead of ready-to-code (${CATEGORY})"
+          add_label "triaged"
+        else
+          echo "Deferring ready-to-code label (${CATEGORY}) until after label_actions..."
+          DEFERRED_LABEL="ready-to-code"
+        fi

Relevance

⭐⭐ Medium
Similar protected-path review/gating suggestions were only partially accepted or undetermined (e.g.,
workflow protected-path note in PR184).
PR-#184
PR-#59
PR-#29

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
PR Compliance ID 1538392 requires raising a finding whenever protected governance/infrastructure
paths are modified. The diff includes changes under scripts/ and harness/, which are explicitly
listed as protected paths.
scripts/post-triage.sh[311-352]
harness/triage.yaml[37-48]
scripts/post-triage-test.sh[495-566]
Skill: pr-review

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The PR modifies protected governance/infrastructure paths (e.g., `scripts/`, `harness/`), which must not be auto-approved and require explicit human review per policy.

## Issue Context
Protected paths can impact CI/CD, automation, and repo governance. Ensure the PR is gated by the required human/owner review controls before merging.

## Fix Focus Areas
- scripts/post-triage.sh[315-332]
- harness/triage.yaml[37-48]
- scripts/post-triage-test.sh[495-566]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

2. Unvalidated auto-code flag 🐞 Bug ☼ Reliability

Description

scripts/post-triage.sh treats any TRIAGE_AUTO_CODE value other than the exact string "false" as
enabled, so typos/case variants (e.g., "FALSE", "0", "fasle") will still apply "ready-to-code" and
can unintentionally trigger the coding agent.

Code

scripts/post-triage.sh[R319-332]

+    TRIAGE_AUTO_CODE="${TRIAGE_AUTO_CODE:-true}"
    CATEGORY=$(jq -r '.triage_summary.category // "unknown"' "${RESULT_FILE}")
    echo "Category: ${CATEGORY}"
    case "${CATEGORY}" in
      bug)
        echo "Applying bug label..."
        add_label "bug"
-        echo "Deferring ready-to-code label (${CATEGORY}) until after label_actions..."
-        DEFERRED_LABEL="ready-to-code"
+        if [[ "${TRIAGE_AUTO_CODE}" == "false" ]]; then
+          echo "TRIAGE_AUTO_CODE=false — applying triaged label instead of ready-to-code (${CATEGORY})"
+          add_label "triaged"
+        else
+          echo "Deferring ready-to-code label (${CATEGORY}) until after label_actions..."
+          DEFERRED_LABEL="ready-to-code"
+        fi

Relevance

⭐⭐⭐ High
Team has accepted env-var parsing hardening to avoid fail-open behavior (SCRIBE_MIN_CONFIDENCE
validation in PR10).
PR-#10

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The documentation constrains TRIAGE_AUTO_CODE to true/false, but the implementation only checks
for the exact lowercase string false and otherwise proceeds with the auto-coding path.

docs/triage.md[131-139]
scripts/post-triage.sh[315-352]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`TRIAGE_AUTO_CODE` is documented as a boolean (`true`/`false`) but is only checked via `[[ "${TRIAGE_AUTO_CODE}" == "false" ]]`. Any other value (including common misconfigurations like `FALSE`) silently behaves like `true`, potentially auto-applying `ready-to-code`.

### Issue Context
- Docs state valid values are strictly `true` and `false`.
- The post-triage script currently defaults to enabled for all non-exact-false values.

### Fix Focus Areas
- scripts/post-triage.sh[319-352]
- docs/triage.md[131-139]

### Proposed fix
1. Normalize the value (e.g., lowercase) and validate explicitly:
  - `TRIAGE_AUTO_CODE="${TRIAGE_AUTO_CODE:-true}"`
  - `TRIAGE_AUTO_CODE="${TRIAGE_AUTO_CODE,,}"`
  - `case "${TRIAGE_AUTO_CODE}" in true|false) ;; *) echo "ERROR: TRIAGE_AUTO_CODE must be true|false"; exit 1 ;; esac`
2. Keep the existing branching logic, but now it can rely on a known-good value.
3. (Optional) Mention in docs that values are case-sensitive unless normalization is implemented.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

qodo-code-review · 2026-07-21T12:01:37Z

+    # Set TRIAGE_AUTO_CODE=false to disable the automatic ready-to-code
+    # promotion. When disabled, low-risk categories receive "triaged" instead,
+    # requiring human review before the code agent runs.
+    TRIAGE_AUTO_CODE="${TRIAGE_AUTO_CODE:-true}"


2. Unvalidated auto-code flag 🐞 Bug ☼ Reliability

scripts/post-triage.sh treats any TRIAGE_AUTO_CODE value other than the exact string "false" as enabled, so typos/case variants (e.g., "FALSE", "0", "fasle") will still apply "ready-to-code" and can unintentionally trigger the coding agent.

Agent Prompt

### Issue description `TRIAGE_AUTO_CODE` is documented as a boolean (`true`/`false`) but is only checked via `[[ "${TRIAGE_AUTO_CODE}" == "false" ]]`. Any other value (including common misconfigurations like `FALSE`) silently behaves like `true`, potentially auto-applying `ready-to-code`. ### Issue Context - Docs state valid values are strictly `true` and `false`. - The post-triage script currently defaults to enabled for all non-exact-false values. ### Fix Focus Areas - scripts/post-triage.sh[319-352] - docs/triage.md[131-139] ### Proposed fix 1. Normalize the value (e.g., lowercase) and validate explicitly: - `TRIAGE_AUTO_CODE="${TRIAGE_AUTO_CODE:-true}"` - `TRIAGE_AUTO_CODE="${TRIAGE_AUTO_CODE,,}"` - `case "${TRIAGE_AUTO_CODE}" in true|false) ;; *) echo "ERROR: TRIAGE_AUTO_CODE must be true|false"; exit 1 ;; esac` 2. Keep the existing branching logic, but now it can rely on a known-good value. 3. (Optional) Mention in docs that values are case-sensitive unless normalization is implemented.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

rh-hemartin · 2026-07-21T13:18:48Z

/fs-review

fullsend-ai-review · 2026-07-21T13:20:13Z

🤖 Finished Review · ✅ Success · Started 1:20 PM UTC · Completed 1:35 PM UTC
Commit: 7ada4e0 · View workflow run →

fullsend-ai-review · 2026-07-21T13:34:56Z

Review

Findings

Medium

[stale-doc] docs/code.md:41 — The control labels table describes ready-to-code as unconditionally "Applied by the triage post-script for low-risk categories (bug, documentation, performance)." This is now conditional on TRIAGE_AUTO_CODE=true (the default). When TRIAGE_AUTO_CODE=false, low-risk categories receive triaged instead.
Remediation: Update the ready-to-code row in docs/code.md to note the conditional behavior and cross-reference the triage docs' Variables section.
[protected-path] harness/triage.yaml, scripts/post-triage.sh, scripts/post-triage-test.sh — This PR modifies files under protected paths (harness/, scripts/). The PR body explains the rationale and references #1750. Human approval is always required for protected-path changes regardless of context.

Low

[test-inadequate] scripts/post-triage-test.sh — The new run_test_auto_code_disabled tests cover TRIAGE_AUTO_CODE=false for all three low-risk categories, but no test explicitly sets TRIAGE_AUTO_CODE=true to confirm the comparison logic. The existing tests provide implicit coverage via the default value, so this is low risk.

Labels: PR modifies triage agent configuration and post-script behavior.

The review agent flagged docs/code.md as an issue because it says the 'ready-to-code' label was "unconditionally applied" and this PR changes that. That's not really an issue b/c the triage agent already applied the label only in certain cases, but it does highlight that the code agent docs contain unnecessary details of triage (which will require changes when triage changes; agents and their docs should be self-sufficient) and internal details (post-script). This just simplifies to clean that up and only contain what the user needs to know. Signed-off-by: Tim deBoer <git@tdeboer.ca>

deboer-tim · 2026-07-21T21:34:05Z

The [stale-doc] above wasn't accurate, but it did highlight that the code agent docs had an overly detailed and unnecessary reference to triage agent. Second commit added to fix that.

rh-hemartin

LGTM, but you need to resolve the last Qodo comment

deboer-tim requested a review from a team as a code owner July 21, 2026 11:54

qodo-code-review Bot reviewed Jul 21, 2026

View reviewed changes

rh-hemartin self-assigned this Jul 21, 2026

rh-hemartin added the ok-to-test label Jul 21, 2026

fullsend-ai-review Bot added requires-manual-review Review requires human judgment triage-agent labels Jul 21, 2026

github-actions Bot removed the ok-to-test label Jul 21, 2026

rh-hemartin approved these changes Jul 22, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: triage variable to control auto-coding#331

feat: triage variable to control auto-coding#331
deboer-tim wants to merge 2 commits into
fullsend-ai:mainfrom
deboer-tim:triage-auto-code

deboer-tim commented Jul 21, 2026

Uh oh!

github-actions Bot commented Jul 21, 2026 •

edited

Loading

Uh oh!

qodo-code-review Bot commented Jul 21, 2026

Uh oh!

qodo-code-review Bot commented Jul 21, 2026

Uh oh!

Uh oh!

qodo-code-review Bot Jul 21, 2026

Uh oh!

rh-hemartin commented Jul 21, 2026

Uh oh!

fullsend-ai-review Bot commented Jul 21, 2026 •

edited

Loading

Uh oh!

fullsend-ai-review Bot commented Jul 21, 2026

Uh oh!

deboer-tim commented Jul 21, 2026

Uh oh!

rh-hemartin left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

deboer-tim commented Jul 21, 2026

Uh oh!

github-actions Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Functional tests did not run

Uh oh!

qodo-code-review Bot commented Jul 21, 2026

PR Summary by Qodo

Uh oh!

qodo-code-review Bot commented Jul 21, 2026

Code Review by Qodo

Uh oh!

Uh oh!

qodo-code-review Bot Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

rh-hemartin commented Jul 21, 2026

Uh oh!

fullsend-ai-review Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fullsend-ai-review Bot commented Jul 21, 2026

Review

Findings

Medium

Low

Uh oh!

deboer-tim commented Jul 21, 2026

Uh oh!

rh-hemartin left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Jul 21, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jul 21, 2026 •

edited

Loading