[Evaluation] Fix RedTeam.scan() decoding encoded attack prompts in stored results

[huliang-microsoft]

Fixes #47228

Problem

RedTeam.scan() was storing the decoded plaintext objective in
evaluation_results.json / results.json for every converter-based attack
strategy (Base64, Flip, Morse, ROT13, Caesar, Leetspeak,
AsciiArt, AnsiAttack, Atbash, Binary, CharacterSpace, CharSwap,
Diacritic, StringJoin, SuffixAppend, UnicodeConfusable,
UnicodeSubstitution, Url, AsciiSmuggler, Tense). The target callback
actually received the encoded converted_value, but persisted
attack_details[].conversation[].content was the pre-converter
original_value. Customer impact (from #47228):

• Impossible to audit/verify the attack surface post-scan.
• Cannot debug why specific encoding variants succeeded / failed.
• Cannot correlate target responses to the exact encoded prompts received.
• attack_technique metadata correctly said e.g. "base64", but
  conversation content was plaintext — breaking reproducibility.

Root cause

In FoundryResultProcessor._build_messages_from_pieces, user turns
deliberately preferred original_value over converted_value, dropping the
on-wire payload from the persisted conversation.

Fix

• content now always reflects what was sent on the wire
  (converted_value, falling back to original_value only when
  converted_value is empty) for both user and assistant turns.
• The pre-converter adversarial objective is preserved as a new sibling
  original_value field on user messages only when it differs from
  content, so the audit trail of "what the attack meant to say" is not
  lost and baseline (non-encoded) strategies stay byte-identical to the
  old output.

Tests

• Updated test_build_messages_from_pieces to assert no original_value
  field appears when there is no encoding (no regression for baseline).
• Added test_build_messages_preserves_encoded_user_prompt (Base64
  example) — primary regression test for Azure AI Evaluation's RedTeam.scan() method decodes **all** encoded attack prompts when storing the result files #47228.
• Added test_build_messages_falls_back_to_original_when_converted_missing
  — covers converted_value is None path.

CHANGELOG

Added a 1.17.1 (Unreleased) entry under ### Bugs Fixed.

cc @singankit @w-javed

[github-actions]

Thank you for your contribution @huliang-microsoft! We will review the pull request and get back to you soon.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Fixes persisted red-team conversations so they reflect the actual encoded payload sent to the target (converter output), while preserving the pre-conversion prompt for auditability.

Changes:

• Update FoundryResultProcessor._build_messages_from_pieces() to store converted_value as content and add original_value only when it differs.
• Add unit tests covering encoded user prompts and fallback behavior when converted_value is missing.
• Document the bug fix and new persisted field in the changelog.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/red_team/_foundry/_foundry_result_processor.py	Changes message serialization to persist wire payload and optionally include original_value for auditing.
sdk/evaluation/azure-ai-evaluation/tests/unittests/test_redteam/test_foundry.py	Adds regression + behavior tests for encoded prompts and fallback behavior.
sdk/evaluation/azure-ai-evaluation/CHANGELOG.md	Documents the behavior change and new original_value field for persisted conversations.

[huliang-microsoft]

Addressed in 493577f — dropped the isinstance(str) guards around content selection so non-string converted_value / original_value (bytes, structured/multimodal payloads) pass through unchanged. The str check is kept only on the original_value audit-field emission, where comparing two non-text values for inequality would be meaningless.

[huliang-microsoft]

Added in 493577f — test_build_messages_preserves_non_string_payloads covers a list-of-parts user payload and a bytes assistant payload, asserting both survive on content without being coerced to "".

[huliang-microsoft]

Addressed in 493577f — wrapped the 1.17.1 entry across multiple lines, dropped the exhaustive strategy enumeration (now reads Base64, Flip, Morse, ROT13, etc.), and kept the key behavior change (content uses converted_value, new original_value audit field) plus the issue link.