Skip to content

feat: triage variable to control auto-coding #331

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
deboer-tim wants to merge 2 commits into
base: main
Choose a base branch
from
Open

feat: triage variable to control auto-coding #331

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c9792b0
feat: triage variable to control auto-coding
deboer-tim Jul 14, 2026
e1c03fb
chore: clean up code docs
deboer-tim Jul 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/code.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ on issues (not PRs). The code agent is also triggered automatically when the

| Label | Meaning |
|-------|---------|
| `ready-to-code` | Triggers the code agent. Applied by the [triage](triage.md) post-script for low-risk categories (bug, documentation, performance), or manually by a human for feature work after prioritization. |
| `ready-to-code` | Triggers the code agent. May be applied by the [triage](triage.md) agent, or manually by a human for feature work after prioritization. |
| `ready-for-review` | Applied by the code agent's post-script after pushing a PR. In per-repo installs, triggers review when applied to a PR; also marks workflow state for humans and the retro agent. |

## Configuration and extension
Expand Down
10 changes: 10 additions & 0 deletions docs/triage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,16 @@ This gives the triage agent the subtlety it needs to distinguish between
controller-runtime code, without adding label documentation to `AGENTS.md`
where every agent would pay the context cost.

### Variables

| Variable | Description | Default | Valid values |
|----------|-------------|---------|--------------|
| `TRIAGE_AUTO_CODE` | When `true`, low-risk categories (bug, documentation, performance) automatically receive the `ready-to-code` label, triggering the [code agent](code.md). When `false`, these categories receive `triaged` instead, requiring human review before coding begins. | `true` | `true`, `false` |

Set this in the CI workflow `env:` block. The env file passes it to the
sandbox automatically, and the post-script reads it from the runner
environment directly — no separate configuration is needed.

## Source

[`harness/triage.yaml`](../harness/triage.yaml)
1 change: 1 addition & 0 deletions env/triage.env
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
export GITHUB_ISSUE_URL="${GITHUB_ISSUE_URL}"
export GH_TOKEN=${GH_TOKEN}
export TRIAGE_AUTO_CODE="${TRIAGE_AUTO_CODE:-true}"
1 change: 1 addition & 0 deletions harness/triage.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ forge:
runner:
GITHUB_ISSUE_URL: ${GITHUB_ISSUE_URL}
GH_TOKEN: ${GH_TOKEN}
TRIAGE_AUTO_CODE: ${TRIAGE_AUTO_CODE}
sandbox:
GITHUB_ISSUE_URL: "${GITHUB_ISSUE_URL}"
GH_TOKEN: "${GH_TOKEN}"
72 changes: 72 additions & 0 deletions scripts/post-triage-test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -492,6 +492,78 @@ run_test "label-category-consistent-passes" \
'{"action":"sufficient","reasoning":"all clear","clarity_scores":{"symptom":0.9,"cause":0.85,"reproduction":0.9,"impact":0.8,"overall":0.87},"triage_summary":{"title":"Fix crash","severity":"high","category":"bug","problem":"Crash","root_cause_hypothesis":"Buffer overflow","reproduction_steps":["step 1"],"environment":"Linux","impact":"All users","recommended_fix":"Fix buffer","proposed_test_case":"test_crash"},"comment":"## Triage Summary\n\nReady.","label_actions":{"reason":"Area label applies.","actions":[{"action":"add","label":"area/api"}]}}' \
"gh api repos/test-org/test-repo/issues/42/labels -f labels[]=area/api --silent"

# --- TRIAGE_AUTO_CODE=false tests ---

# When TRIAGE_AUTO_CODE=false, bug category should get "triaged" instead of "ready-to-code".
run_test_auto_code_disabled() {
local test_name="$1"
local json_content="$2"
local expected_pattern="$3"
local forbidden_pattern="${4:-}"

local run_dir="${TMPDIR}/run-${test_name}"
mkdir -p "${run_dir}/iteration-1/output"
echo "${json_content}" > "${run_dir}/iteration-1/output/agent-result.json"
: > "${GH_LOG}"

local exit_code=0
(cd "${run_dir}" && TRIAGE_AUTO_CODE=false bash "${POST_SCRIPT}") > "${TMPDIR}/stdout.log" 2>&1 || exit_code=$?

if [[ ${exit_code} -ne 0 ]]; then
echo "FAIL: ${test_name} — exit code ${exit_code}"
cat "${TMPDIR}/stdout.log"
FAILURES=$((FAILURES + 1))
return
fi

if ! grep -qF "${expected_pattern}" "${GH_LOG}"; then
echo "FAIL: ${test_name} — expected '${expected_pattern}' not found"
echo "Actual calls:"
cat "${GH_LOG}"
FAILURES=$((FAILURES + 1))
return
fi

if [[ -n "${forbidden_pattern}" ]] && grep -qF "${forbidden_pattern}" "${GH_LOG}"; then
echo "FAIL: ${test_name} — forbidden '${forbidden_pattern}' was found"
echo "Actual calls:"
cat "${GH_LOG}"
FAILURES=$((FAILURES + 1))
return
fi

echo "PASS: ${test_name}"
}

SUFFICIENT_BUG='{"action":"sufficient","reasoning":"all clear","clarity_scores":{"symptom":0.9,"cause":0.85,"reproduction":0.9,"impact":0.8,"overall":0.87},"triage_summary":{"title":"Fix crash","severity":"high","category":"bug","problem":"Crash","root_cause_hypothesis":"Buffer overflow","reproduction_steps":["step 1"],"environment":"Linux","impact":"All users","recommended_fix":"Fix buffer","proposed_test_case":"test_crash"},"comment":"## Triage Summary\n\nReady."}'

SUFFICIENT_DOCS='{"action":"sufficient","reasoning":"all clear","clarity_scores":{"symptom":0.9,"cause":0.85,"reproduction":0.9,"impact":0.8,"overall":0.87},"triage_summary":{"title":"Update docs","severity":"low","category":"documentation","problem":"Outdated docs","root_cause_hypothesis":"Not updated","reproduction_steps":["step 1"],"environment":"Linux","impact":"Contributors","recommended_fix":"Update README","proposed_test_case":"test_docs"},"comment":"## Triage Summary\n\nDocs issue."}'

SUFFICIENT_PERF='{"action":"sufficient","reasoning":"all clear","clarity_scores":{"symptom":0.9,"cause":0.85,"reproduction":0.9,"impact":0.8,"overall":0.87},"triage_summary":{"title":"Slow query","severity":"medium","category":"performance","problem":"Slow","root_cause_hypothesis":"Missing index","reproduction_steps":["step 1"],"environment":"Linux","impact":"All users","recommended_fix":"Add index","proposed_test_case":"test_query_speed"},"comment":"## Triage Summary\n\nPerf issue."}'

run_test_auto_code_disabled "auto-code-disabled-bug-gets-triaged" \
"${SUFFICIENT_BUG}" \
"labels[]=triaged" \
"labels[]=ready-to-code"

run_test_auto_code_disabled "auto-code-disabled-bug-still-gets-bug-label" \
"${SUFFICIENT_BUG}" \
"labels[]=bug"

run_test_auto_code_disabled "auto-code-disabled-docs-gets-triaged" \
"${SUFFICIENT_DOCS}" \
"labels[]=triaged" \
"labels[]=ready-to-code"

run_test_auto_code_disabled "auto-code-disabled-docs-still-gets-documentation-label" \
"${SUFFICIENT_DOCS}" \
"labels[]=documentation"

run_test_auto_code_disabled "auto-code-disabled-perf-gets-triaged" \
"${SUFFICIENT_PERF}" \
"labels[]=triaged" \
"labels[]=ready-to-code"

# --- Summary ---

echo ""
Expand Down
32 changes: 26 additions & 6 deletions scripts/post-triage.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -312,24 +312,44 @@ ${FAILED_CREATES}"
# ready-to-code, which triggers the code agent. Feature work and anything
# else receives the triaged label and waits for human prioritization
# (per #561, only feature issues should require human review before coding).
#
Comment thread
rh-hemartin marked this conversation as resolved.
# Set TRIAGE_AUTO_CODE=false to disable the automatic ready-to-code
# promotion. When disabled, low-risk categories receive "triaged" instead,
# requiring human review before the code agent runs.
TRIAGE_AUTO_CODE="${TRIAGE_AUTO_CODE:-true}"

@qodo-code-review qodo-code-review Bot Jul 21, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

2. Unvalidated auto-code flag 🐞 Bug ☼ Reliability

scripts/post-triage.sh treats any TRIAGE_AUTO_CODE value other than the exact string "false" as
enabled, so typos/case variants (e.g., "FALSE", "0", "fasle") will still apply "ready-to-code" and
can unintentionally trigger the coding agent.
Agent Prompt 
### Issue description
`TRIAGE_AUTO_CODE` is documented as a boolean (`true`/`false`) but is only checked via `[[ "${TRIAGE_AUTO_CODE}" == "false" ]]`. Any other value (including common misconfigurations like `FALSE`) silently behaves like `true`, potentially auto-applying `ready-to-code`.

### Issue Context
- Docs state valid values are strictly `true` and `false`.
- The post-triage script currently defaults to enabled for all non-exact-false values.

### Fix Focus Areas
- scripts/post-triage.sh[319-352]
- docs/triage.md[131-139]

### Proposed fix
1. Normalize the value (e.g., lowercase) and validate explicitly:
   - `TRIAGE_AUTO_CODE="${TRIAGE_AUTO_CODE:-true}"`
   - `TRIAGE_AUTO_CODE="${TRIAGE_AUTO_CODE,,}"`
   - `case "${TRIAGE_AUTO_CODE}" in true|false) ;; *) echo "ERROR: TRIAGE_AUTO_CODE must be true|false"; exit 1 ;; esac`
2. Keep the existing branching logic, but now it can rely on a known-good value.
3. (Optional) Mention in docs that values are case-sensitive unless normalization is implemented.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

CATEGORY=$(jq -r '.triage_summary.category // "unknown"' "${RESULT_FILE}")
echo "Category: ${CATEGORY}"
case "${CATEGORY}" in
bug)
echo "Applying bug label..."
add_label "bug"
echo "Deferring ready-to-code label (${CATEGORY}) until after label_actions..."
DEFERRED_LABEL="ready-to-code"
if [[ "${TRIAGE_AUTO_CODE}" == "false" ]]; then
echo "TRIAGE_AUTO_CODE=false — applying triaged label instead of ready-to-code (${CATEGORY})"
add_label "triaged"
else
echo "Deferring ready-to-code label (${CATEGORY}) until after label_actions..."
DEFERRED_LABEL="ready-to-code"
fi
;;
documentation)
echo "Applying documentation label..."
add_label "documentation"
echo "Deferring ready-to-code label (${CATEGORY}) until after label_actions..."
DEFERRED_LABEL="ready-to-code"
if [[ "${TRIAGE_AUTO_CODE}" == "false" ]]; then
echo "TRIAGE_AUTO_CODE=false — applying triaged label instead of ready-to-code (${CATEGORY})"
add_label "triaged"
else
echo "Deferring ready-to-code label (${CATEGORY}) until after label_actions..."
DEFERRED_LABEL="ready-to-code"
fi
;;
performance)
echo "Deferring ready-to-code label (${CATEGORY}) until after label_actions..."
DEFERRED_LABEL="ready-to-code"
if [[ "${TRIAGE_AUTO_CODE}" == "false" ]]; then
echo "TRIAGE_AUTO_CODE=false — applying triaged label instead of ready-to-code (${CATEGORY})"
add_label "triaged"
else
echo "Deferring ready-to-code label (${CATEGORY}) until after label_actions..."
DEFERRED_LABEL="ready-to-code"
fi
;;
feature)
echo "Applying feature + triaged labels..."
Expand Down
Loading