eval scrub: add minimum-length guard on ::add-mask:: values

[waynesun09]

Problem

In scrub-eval-results.sh, the mask value filter (Pass 1) rejects "", "***", and "[REDACTED]" but accepts any other value regardless of length. If an eval log contains ::add-mask::a (a single character), the letter a is added to the secrets set and str.replace("a", "***") is applied to every file — catastrophically corrupting all text output.

Similarly, common short strings like ::add-mask::true or ::add-mask::1 would cause widespread over-redaction.

GitHub Actions has the same foot-gun (it masks single-character values), but this script has the opportunity to be smarter since it runs after the fact on artifact files.

Suggested Fix

Add a minimum length threshold:

MIN_MASK_LEN = 4

for match in ADD_MASK_RE.finditer(text):
    value = match.group(1).strip()
    if value in ("***", "[REDACTED]", "") or len(value) < MIN_MASK_LEN:
        continue
    secrets.add(value)

A threshold of 4 avoids masking trivially short strings while catching all real secrets (GitHub tokens are 20+ characters, passwords typically 8+).

Context

Found during review of #184. The scrub script was introduced in that PR and all prior review findings were addressed, but this gap was not caught by the automated reviewers.

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 6:16 PM UTC · Completed 6:20 PM UTC
Commit: 5cd495a · View workflow run →

[fullsend-ai-triage]

Triage Summary

Severity: High · Category: Bug

Problem

The scrub-eval-results.sh script (introduced in PR #184) collects ::add-mask:: values in Pass 1 but only skips empty strings, "***", and "[REDACTED]". Any other value — regardless of length — is added to the secrets set and globally replaced across all text files. A single-character mask value like ::add-mask::a would corrupt the entire eval output tree.

Root Cause

The filter condition in the secrets collection loop lacks a minimum length check. Short strings pass through and cause catastrophic over-redaction during the global str.replace() pass.

Recommended Fix

Add a MIN_MASK_LEN = 4 constant and extend the skip condition:

MIN_MASK_LEN = 4

for match in ADD_MASK_RE.finditer(text):
    value = match.group(1).strip()
    if value in ("***", "[REDACTED]", "") or len(value) < MIN_MASK_LEN:
        continue
    secrets.add(value)

Proposed Test Case

# Test: short mask values are ignored, long ones are redacted
# Setup: create temp dir with a text file containing:
#   ::add-mask::a
#   The quick brown fox jumps over a lazy dog
#   ::add-mask::realsecret123
#   Token is realsecret123
#
# After scrub:
#   - Letter 'a' must be preserved in normal text (not redacted)
#   - 'realsecret123' must be replaced with '***'
#   - Both ::add-mask:: lines become ::add-mask::***
#
# Edge cases:
#   - len=3 value (e.g., 'abc') -> rejected
#   - len=4 value (e.g., 'abcd') -> accepted and redacted
#   - Existing skip values ('***', '[REDACTED]', '') -> still skipped

Related

• Issue eval scrub: TEXT_SUFFIXES blind spot — upload path broader than scrub/verify path #185 tracks a separate scrub script gap (TEXT_SUFFIXES blind spot)

[fullsend-ai-coder]

🤖 Finished Code · ✅ Success · Started 6:21 PM UTC · Completed 6:31 PM UTC
Commit: 5cd495a · View workflow run →