Skip to content
Draft
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions docs/testing/TEST_COVERAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,8 @@
- Type conversion behavior
- Type precedence in operations

**Nested Structure Coverage**: For operators that collect or pass through document values (e.g. `$push`, `$addToSet`, `$mergeObjects`), test deeply nested arrays-of-objects with embedded arrays (e.g. `{"data": {"users": [{"profile": {"name": "Alice", "scores": [85, 90]}}]}}`) to verify the full structure is preserved without flattening or truncation. Also test array traversal via field paths (e.g. `$a.b` on `{"a": [{"b": 1}, {"b": 2}]}`) to verify correct collection of values from arrays-of-objects.

---

### 2. Arithmetic Operator Coverage
Expand Down Expand Up @@ -98,6 +100,7 @@ Note: Distinguish between fractional doubles (2.5) and whole-number doubles (3.0
- **Underflow**: `INT32_MIN` - 1 → long, `INT64_MIN` - 1 → double
- **Sign handling**: positive, negative, zero
- **Negative zero**: `DOUBLE_NEGATIVE_ZERO` → verify behavior (some operators normalize to `0.0`, others preserve `-0.0`); `DECIMAL128_NEGATIVE_ZERO` → verify; `NumberDecimal("-0E+N")` and `NumberDecimal("-0E-N")` → verify exponent preservation
- **Special float values mixed with non-numeric types**: For operators that collect or pass through values (e.g. `$push`, `$addToSet`), test special float values (NaN, Infinity, -Infinity, -0.0) alongside non-numeric types (string, boolean, null, object, array) in the same group to verify all values are preserved without coercion or loss.
- **Special values**: MinKey, MaxKey combinations
- **Two's complement asymmetry (single-input operators)**: `INT32_MIN` has no positive int counterpart → must promote to long; `INT64_MIN` has no positive long counterpart → verify overflow/error behavior
- **Double precision boundaries**: `DOUBLE_NEAR_MAX`, `DOUBLE_MIN_SUBNORMAL`, `DOUBLE_MIN_NEGATIVE_SUBNORMAL`, `DOUBLE_NEAR_MIN`, `DOUBLE_NEGATIVE_ZERO` → use `NUMERIC_DOUBLE` from `test_constants.py`
Expand Down Expand Up @@ -362,6 +365,7 @@ For each invalid_type in [string, object, array, ...]:
- Primary operation on basic input
- Empty input and non-existent collection both produce correct output without error
- Works as the sole pipeline stage
- For `$group`: test compound `_id` (grouping by multiple fields, e.g. `{"_id": {"region": "$region", "status": "$status"}}`), nested field path `_id` (e.g. `"_id": "$user.dept"`), and large-scale multi-group aggregation (e.g. 10,000 documents across 100+ groups) to verify group partitioning at scale beyond single-group `_id: null`

**Parameter Validation**:
- Test every BSON type against the parameter. Numeric stages (`$limit`, `$skip`, `$sample`) accept int32, int64, whole-number double, whole-number Decimal128. Document stages (`$match`, `$project`, `$group`, `$set`) reject non-documents. String stages (`$count`, `$unwind`) reject non-strings.
Expand All @@ -378,6 +382,7 @@ For each invalid_type in [string, object, array, ...]:
- Multi-stage interaction tests belong in the parent `stages/` directory, not in individual stage folders. Per `FOLDER_STRUCTURE.md`, interactions between same-level features go in the parent folder (e.g., `stages/test_stages_combination_sort.py`, `stages/test_stages_position_match.py`).
- Test interactions where ordering affects results or where adjacent stages compose non-obviously (e.g., optimization coalescence, count-modifying vs non-count-modifying intervening stages, additive vs min-taking consecutive stages)
- Cover common multi-stage usage patterns for the stage under test
- Test multiple `$group` stages in a single pipeline (e.g. `$group` → `$sort` → `$group`), where each stage performs actual grouping (not just `_id: null`). The second `$group` should re-aggregate the output of the first, verifying that accumulated results (arrays, counts, etc.) can be correctly consumed as input to another `$group`.

**Out of Scope**:
- Cross-cutting concerns (views, capped collections, timeseries) belong in their own directories
Expand Down Expand Up @@ -435,6 +440,10 @@ For each invalid_type in [string, object, array, ...]:
The expression-form of dual-form operators (`$max`, `$min`, `$sum`, `$avg`, `$first`, `$last`, etc.) is tested separately under `tests/core/operator/expressions/accumulator/$op/` and is out of scope for the
accumulator-form file.

**Actual Grouping Requirement**: Accumulator tests must not rely exclusively on `_id: null` (single-group). Each test category (null/missing, special values, core behavior, etc.) should include at least one test with actual multi-group grouping (e.g. `_id: "$cat"`) to verify accumulators reset correctly across group boundaries.

**Multiple Accumulators in Single $group**: Test multiple accumulators of the same type in a single `$group` (e.g. `{"$group": {"_id": "$cat", "xs": {"$push": "$x"}, "ys": {"$push": "$y"}}}`) with actual grouping, not just `_id: null`. Each accumulator should independently handle its own field's null/missing values without interfering with siblings. This is distinct from sibling-accumulator interaction tests (which test different accumulator types together, e.g. `$push` + `$sum`) — it verifies that multiple instances of the same accumulator independently collect from their respective fields without cross-contamination.

**Expression Error Propagation**:
Errors raised during sub-expression evaluation propagate through the accumulator without being caught:
- `{$op: {$divide: [1, "$v"]}}` → `CONVERSION_FAILURE_ERROR` (field violation)
Expand All @@ -449,6 +458,10 @@ For each invalid_type in [string, object, array, ...]:

- For order-dependent accumulators, tests asserting a specific result must include a preceding `$sort`. Tests without `$sort` are flaky. (e.g. $first)
- For order-independent accumulators, the result must be the same regardless of input order. Verify this by running the same input twice with different `$sort` directions and asserting identical results.
- Sort coverage must include compound sorts with mixed directions (e.g. `{"$sort": {"priority": 1, "status": -1, "timestamp": 1}}`) and sorts on nested field paths (e.g. `{"$sort": {"user.dept": 1}}`).

**Large-Scale Result Verification**:
When testing accumulators at scale (1000+ documents), verify the actual content of the result, not just its count. Prefer building the expected result with a loop (e.g. `expected=[{"_id": None, "result": list(range(10_000))}]`) over using server-side aggregates like `$size` or `$count` in a `$project`. A count-only check can pass even if values are duplicated, dropped, or corrupted. When a full element-by-element expected list is impractical (e.g. multi-group aggregation), use server-side content checks (`$sum`, `$min`, `$max` on the pushed array) as a secondary option.

**Tested in Other Folders** (in scope, but add under a different folder):
- **Host-stage compatibility** — when adding a new accumulator, add one smoke case for each host stage that supports it (`$group`, `$bucket`, `$bucketAuto`, `$setWindowFields`) under that stage's folder
Expand Down
Loading