FFL-2783: add tests for observeFullEvaluationData PII protection (gated as missing_feature)

[vjfridge]

Motivation

FFL-2783 — part of FFL-2780 (Protecting PII in flagevaluations track)

See informal RFC for more context

Summary

• Adds three test classes to existing tests/ffe/test_flag_eval_evp.py covering the new per-env UFC observeFullEvaluationData field
• Extends make_ufc_fixture with an observe_full_evaluation_data: bool | None = None kwarg. None omits the field entirely, so existing tests are unaffected
• All three classes are deactivated as missing_feature (FFL-2783) on Go — the only SDK where the file-level gate would otherwise activate them. Other SDK manifests already deactivate the whole file via missing_feature (FFL-2446) or irrelevant.

Truth-table coverage

UFC observeFullEvaluationData	Expected targeting_key	Expected context.evaluation
absent	sha256_-prefixed (71 chars, lowercase hex suffix)	omitted
false	sha256_-prefixed	omitted
true	raw customer value, no prefix	full object with all attributes

Canonical PII vector (shared across SDKs)

Every SDK's unit tests should assert against the same input/output. Input: "jane.doe@datadoghq.com" → "sha256_b4698f9b6d186781fa8dc59e533578fa2d8379a46b1cf6db85cda6aa9c99e51b". Canonical attributes cover string, integer, dotted-key, and email PII shapes.

Forward-compat for existing SDKs

Introducing a new field inside the UFC environment object could theoretically break SDKs if their UFC parser rejects unknown fields. That risk is already covered by tests/ffe/test_dynamic_evaluation.py::Test_FFE_Unknown_Fields_Tolerance, which injects random unknown fields at every UFC level (including inside environment) and asserts each SDK continues to evaluate flags correctly. That test runs on every SDK where test_dynamic_evaluation.py is active, so no per-fixture retrofit is needed here.

Out of scope

• Existing kill switch (DD_FLAGGING_EVALUATION_COUNTS_ENABLED) — per-SDK unit test territory, no system-tests scenario variant needed.

Test plan

• ./format.sh clean (mypy, ruff, node linters)
• Once Go SDK ships FFL-2790, flip Go's three missing_feature (FFL-2783) rows to v2.X.Y-dev and confirm all three cells pass

🤖 Generated with Claude Code

[github-actions]

CODEOWNERS have been resolved as:

manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
tests/ffe/test_flag_eval_evp.py                                         @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/ffe/utils/fixtures.py                                             @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
utils/__init__.py                                                       @DataDog/system-tests-core
utils/_decorators.py                                                    @DataDog/system-tests-core

[datadog-datadog-us1-prod]

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: e9a5752 | Docs | Datadog PR Page | Give us feedback!}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d1ad980b2f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Move fallible setup work into the test

These new setup_* methods perform remote-config application and the weblog request before the test body; if either path raises or times out in CI, pytest reports a setup error and the scenario can stop instead of recording a normal test failure. .cursor/rules/pr-review.mdc says setup methods must only trigger observability and failures/assertions belong in test_*, so keep setup minimal and move the fallible validation path into the matching test method.

Useful? React with 👍 / 👎.

[vjfridge]

Not addressing this one. The other eight test classes in this file (Test_FFE_EVP_Flagevaluation_Basic, _Count, _Context_Bounds, _Runtime_Default, _Load_Aggregation, _Burst_Aggregation, _High_Cardinality_Aggregation, _Degradation) all place rc...apply() and evaluate_flag(...) in setup_* and only run assert self.r.status_code == 200 in test_*. Diverging in only the three new classes would create an inconsistency reviewers would flag; if the setup-vs-test split needs enforcement here, it belongs in a dedicated refactor across the whole file.

[cbeauchesne]

From framework usage, just a small request change.

Cold you also get a review from someone familiar with the feature for the test logic ?

[cbeauchesne]

There are semantic decorators in utils that allow to achieve the same, and also declare why we want to skip them : @slow and @scenario_crash, could you use them ?

[vjfridge]

Thanks for raising this! You're right that using @pytest.mark.skip_if_xfail directly is too raw. We looked at @slow and @scenario_crash but neither quite fits — these tests aren't slow, and they don't crash the scenario. They're gated on SDK implementation status.

So we added a new alias @not_yet_implemented in utils/_decorators.py (same underlying skip_if_xfail mark, just with a clearer name) and used it consistently across all 11 classes in the file.

[nccatoni]

Why is it not enough to mark the tests as xfails in that case?

[leoromanovsky]

logic is good, before merging please:

• add a env var=false case
• adjust labels on existing test cases

[leoromanovsky]

hehe @cbeauchesne skipped these because they are not set up correctly; I needed to break system test convention to get them to work with PHP, which has its own sidecar that needs an additional await to count received EVP events in the backend. It's on my backlog to address, but a more correct skip decorator would be flaky or bug

[vjfridge]

Thanks for raising Leo! I looked into this more carefully. The existing @flaky/@bug decorators are all-or-nothing across SDKs. Per-SDK skipping lives in the manifests, so I think it would be best to wait until PHP gets enabled and then add class-level manifest entries in manifests/php.yml.

For now I've added a docstring note to Burst, HighCardinality, and Degradation pointing to FFL-2529 and the manifest pattern, so whenever we drop the file-level missing_feature for PHP we will know to add those entries at the same time.

I left @not_yet_implemented on all three for now since that's still the accurate for PHP and most SDKs. Let me know if this works for you or if you were looking for something different!

[leoromanovsky]

Please add a test for DD_FLAGGING_EVALUATION_COUNTS_ENABLED=false + observeFullEvaluationData=true - sorry I missed adding coverage for the false case.

[vjfridge]

Happy to add coverage for DD_FLAGGING_EVALUATION_COUNTS_ENABLED =false + observeFullEvaluationData=true, but I think it lives better at L1 unit tests/L2 SDK integration layers.

The assertion is "SDK-level killswitch overrides UFC opt-in" which is purely internal to the SDK and doesn't exercise anything system-level. Adding an L3 scenario here means a new weblog boot per SDK (~minutes of CI × N languages) to verify the boolean killswitch.

I propose we cover this in each SDK's unit or integration suite instead. Let me know if you have strong feelings otherwise!

[vjfridge]

Local system-tests validation — Java SDK ✅

Ran the three new observeFullEvaluationData PII-protection tests locally against dd-trace-java#12042 (java@1.65.0-SNAPSHOT+81af153ca7).

All three pass:

Test	Result
Test_FFE_EVP_Flagevaluation_ObserveFullData_Absent_Hashed	✅ PASS
Test_FFE_EVP_Flagevaluation_ObserveFullData_False_Hashed	✅ PASS
Test_FFE_EVP_Flagevaluation_ObserveFullData_True_Unhashed	✅ PASS

36 passed, 8 skipped, 2704 deselected, 1 xfailed, 3 xpassed in 301.86s

Run command:

./run.sh +l java FEATURE_FLAGGING_AND_EXPERIMENTATION \
  -F tests/ffe/test_flag_eval_evp.py::Test_FFE_EVP_Flagevaluation_ObserveFullData_Absent_Hashed \
  -F tests/ffe/test_flag_eval_evp.py::Test_FFE_EVP_Flagevaluation_ObserveFullData_False_Hashed \
  -F tests/ffe/test_flag_eval_evp.py::Test_FFE_EVP_Flagevaluation_ObserveFullData_True_Unhashed

Note: an earlier build (d65c79f266) failed True_Unhashed — the SDK was always hashing regardless of the flag. Root cause was FlagEvaluationWriterImpl.buildEventList() reading observeFullEvaluationData from CURRENT_CONFIG at flush time, which could be overwritten by a subsequent RC update before the 10s flush fired. Fixed in the dd-trace-java PR by capturing the value per-bucket at enqueue time with a privacy-preserving fold (any false in a bucket's lifetime sinks it to hashed).