Skip to content

Hierarchy/MDX Filter Support & Value Handling Modernization#445

Open
PPN-OL wants to merge 4 commits into
cubewise-code:masterfrom
PPN-OL:master
Open

Hierarchy/MDX Filter Support & Value Handling Modernization#445
PPN-OL wants to merge 4 commits into
cubewise-code:masterfrom
PPN-OL:master

Conversation

@PPN-OL
Copy link
Copy Markdown

@PPN-OL PPN-OL commented May 15, 2026

Pull Request: Bedrock v4.1 — Hierarchy/MDX Filter Support & Value Handling Modernization

What was changed?

This PR delivers two independent sets of changes across four Bedrock TI processes.


1. Hierarchy & MDX Filter Support

}bedrock.cube.view.create

  • Hierarchy syntax: pFilter now accepts DimName:HierName¦elem1+elem2. The dimension token is split on : to extract dimension and hierarchy name separately. All subsequent operations (HierarchySubsetCreate, HierarchySubsetDeleteAllElements, ViewSubsetAssign) use the correct hierarchy.
  • MDX syntax: pFilter now accepts DimName¦MDX:{<expression>}. The brace-depth counter (nBraceDepth) prevents all delimiter characters (+, &, ¦) inside {...} from being interpreted as delimiters. The MDX subset is created by delegating to }bedrock.hier.sub.create.bymdx.
  • Element validation: Replaced DIMIX with HIERARCHYELEMENTINDEX — works correctly for both default and non-default hierarchies.
  • Parsed filter output: sTargetFilter preserves Dim:Hier notation for downstream consumers.
  • Validation: Added checks for unmatched braces, invalid hierarchy names, and use of : as a delimiter character.

}bedrock.cube.data.copy.intercube

  • Filter parser extended: The existing character-by-character parser now:
    • Splits DimName:HierName tokens — extracts the bare dimension name for target-cube dimension matching while preserving the full token in the reconstructed target filter.
    • Handles MDX:{...} blocks as atomic units — brace-depth counter prevents delimiters inside the block from being interpreted as filter structure.
    • Validates element membership via HIERARCHYELEMENTINDEX (with sHierarchy = sDimension as fallback for the default hierarchy).
    • Raises an error for unmatched braces after parsing.
  • pFilterParallel validation: The conflict check for pFilterParallel vs. pFilter now also detects Dim:Hier¦... notation, preventing false negatives when the parallel dimension is specified with hierarchy syntax.

}bedrock.cube.data.copy

  • pFilterParallel validation: Extended the conflict check to detect Dim:Hier¦... notation in pFilter (added % Scan(Lower(sDimParallel) | ':', Lower(pFilter)) > 0).

}bedrock.cube.data.export

  • pFilterParallel validation: Identical fix as data.copy — one-line change at the same validation point.

}bedrock.cube.data.clear

  • pFilterParallel validation: Identical fix as data.copy and data.export — one-line change extending the conflict check to detect Dim:Hier¦... notation in pFilter.

2. Value Handling Modernization (data.copy and data.copy.intercube)

}bedrock.cube.data.copy and }bedrock.cube.data.copy.intercube — Prolog

  • CubeDimensionCountGet replaces the existing TabDim-based while-loop for counting cube dimensions. One API call instead of a loop.
  • Precomputed index strings: sDimCount and sDimCountP1 are computed once from nDimensionCount for use in Expand-based DataProcedure lookups.
  • sLastDimName: The name of the cube's last dimension is computed once in the Prolog via TabDim(pCube, nDimensionCount) (or TabDim(pTgtCube, ...) for intercube). Constant across all data rows — avoids a repeated lookup per row.

}bedrock.cube.data.copy — DataProcedure

  • Single-pass value preparation: A unified block before the nDimensionCount branch now handles all value and type logic once per data row:
    • sLastEle is read via Expand('%v' | sDimCount | '%').
    • sElType = DType(sLastDimName, sLastEle) replaces the per-block DType(sDimN, vN) call.
    • VIEW mode (pFile = 0): numeric value from NValue * nFactor; string value from SValue.
    • CSV mode (pFile > 0): value from Expand('%v' | sDimCountP1 | '%') with StringToNumberEx.
  • Simplified dimension blocks (n = 3..27): Each block now contains only a CellIsUpdateable check and a CellPutN/CellPutS/CellIncrementN call — all type detection and value preparation removed.
  • CellIncrementN replaces CellGetN + add + CellPutN for pCumulate = 1 — atomic operation, eliminates race conditions under parallel execution.
  • nDimensionCount = 2 block: Normal string/numeric cells modernized to use CellIncrementN and the pre-computed vString/nCbal. Special-case handling (}ElementSecurity_*, attribute cubes AA/AS/AN) is preserved unchanged.

}bedrock.cube.data.copy.intercube — DataProcedure

  • Same single-pass value preparation pattern, adapted for sV variables: sLastEle = Expand('%sV' | sDimCount | '%').
  • All 26 dimension blocks (n = 2..27) simplified to CellIsUpdateable + CellPutS(sVString) / CellIncrementN(nCbal).
  • nDimensionCount = 2 block: ElementSecurityPut and AttrPut* special cases updated to use pre-computed sVString/nCbal.

Why?

Hierarchy & MDX Filter Support

TM1/Planning Analytics supports non-default hierarchies and MDX set expressions in view definitions. Without these changes:

  • A pFilter value of Year:FiscalYear¦2024 would either fail silently or raise a spurious error because the parser did not understand the : separator.
  • An MDX expression in pFilter would be corrupted by the character-by-character parser treating +, &, and ¦ inside the MDX block as structural delimiters.
  • The pFilterParallel conflict check would produce a false negative when pFilter used Dim:Hier notation — the same dimension could be specified in both parameters without triggering a validation error, leading to silent double-filtering or incorrect parallel splits.

Value Handling Modernization

  • Correctness: Reading numeric values via StringToNumberEx in VIEW mode (pFile = 0) is unnecessary — the value is already available as a native number in NValue. Any mismatch between the server's actual decimal separator and pDecimalSeparator would cause silent wrong values.
  • Performance: DType(sDimN, vN) and StringToNumberEx(vN, ...) were repeated in every one of the 27 dimension blocks, even though the result is identical for all blocks at the same data row. Moving this to a single pre-computation eliminates 26 redundant lookups per row.
  • Concurrency safety: CellGetN + add + CellPutN is not atomic — under parallel execution with multiple threads writing to the same cell, intermediate reads can produce incorrect results. CellIncrementN is an atomic server-side operation.

Affected Processes

Process Changes
}bedrock.cube.view.create Hierarchy syntax, MDX syntax, HIERARCHYELEMENTINDEX, brace-depth parser, sTargetFilter output
}bedrock.cube.data.copy.intercube Filter parser extension (Hierarchy + MDX), pFilterParallel validation fix, NValue/CellIncrementN/single-pass value prep
}bedrock.cube.data.copy pFilterParallel validation fix, NValue/CellIncrementN/single-pass value prep
}bedrock.cube.data.export pFilterParallel validation fix only
}bedrock.cube.data.clear pFilterParallel validation fix only

Backward Compatibility

All changes are fully backward compatible.

Concern Status
Existing pFilter calls without hierarchy syntax ✅ Unchanged — the parser falls back to sHierarchy = sDimension (default hierarchy) when no : is present
Existing pFilter calls without MDX ✅ Unchanged — the brace-depth counter activates only when { is encountered
pDecimalSeparator / pThousandSeparator parameters ✅ Still used in CSV mode (pFile > 0) — no parameters removed or renamed
pCumulate semantics ✅ Identical — CellIncrementN produces the same result as the previous CellGetN + add + CellPutN sequence
DType result for nDimensionCount = 2 special cases sElType is still derived from DType — attribute cube detection (AA/AS/AN) and }ElementSecurity_* handling are unchanged
Calls using plain DimName¦elem syntax ✅ Identical behavior — no syntax changes required for existing callers
}bedrock.cube.data.clear callers ✅ No interface change — one-line internal validation fix only
}bedrock.cube.clone callers ✅ No interface change — clone is not modified
Process parameter interfaces ✅ No parameters added, removed, or renamed in any of the four processes

Deliverables

main_asJSON/

Contains all Bedrock processes as ready-to-import JSON files. Each file is a self-contained, patchable process definition that can be uploaded directly to a TM1/Planning Analytics server via REST API or TM1py.

The Attributes block ("Caption") has been stripped from every file in this folder to prevent accidental overwrite of server-side captions during import.

docs/wiki/

Contains Markdown documentation for each process modified in this PR:

File Process documented
bedrock.cube.view.create.md }bedrock.cube.view.create
bedrock.cube.data.copy.md }bedrock.cube.data.copy
bedrock.cube.data.copy.intercube.md }bedrock.cube.data.copy.intercube
bedrock.cube.data.export.md }bedrock.cube.data.export
bedrock.cube.data.clear.md }bedrock.cube.data.clear
process-dependencies.md Full transitive call graph for all }bedrock.* processes

PPN-OL added 3 commits May 15, 2026 16:08
Hierarchy/MDX Filter Support & Value Handling Modernization

## What was changed?

This PR delivers two independent sets of changes across four Bedrock TI processes.

---

### 1. Hierarchy & MDX Filter Support

#### `}bedrock.cube.view.create`

- **Hierarchy syntax:** `pFilter` now accepts `DimName:HierName¦elem1+elem2`. The dimension token is split on `:` to extract dimension and hierarchy name separately. All subsequent operations (`HierarchySubsetCreate`, `HierarchySubsetDeleteAllElements`, `ViewSubsetAssign`) use the correct hierarchy.
- **MDX syntax:** `pFilter` now accepts `DimName¦MDX:{<expression>}`. The brace-depth counter (`nBraceDepth`) prevents all delimiter characters (`+`, `&`, `¦`) inside `{...}` from being interpreted as delimiters. The MDX subset is created by delegating to `}bedrock.hier.sub.create.bymdx`.
- **Element validation:** Replaced `DIMIX` with `HIERARCHYELEMENTINDEX` — works correctly for both default and non-default hierarchies.
- **Parsed filter output:** `sTargetFilter` preserves `Dim:Hier` notation for downstream consumers.
- **Validation:** Added checks for unmatched braces, invalid hierarchy names, and use of `:` as a delimiter character.

#### `}bedrock.cube.data.copy.intercube`

- **Filter parser extended:** The existing character-by-character parser now:
  - Splits `DimName:HierName` tokens — extracts the bare dimension name for target-cube dimension matching while preserving the full token in the reconstructed target filter.
  - Handles `MDX:{...}` blocks as atomic units — brace-depth counter prevents delimiters inside the block from being interpreted as filter structure.
  - Validates element membership via `HIERARCHYELEMENTINDEX` (with `sHierarchy = sDimension` as fallback for the default hierarchy).
  - Raises an error for unmatched braces after parsing.
- **`pFilterParallel` validation:** The conflict check for `pFilterParallel` vs. `pFilter` now also detects `Dim:Hier¦...` notation, preventing false negatives when the parallel dimension is specified with hierarchy syntax.

#### `}bedrock.cube.data.copy`

- **`pFilterParallel` validation:** Extended the conflict check to detect `Dim:Hier¦...` notation in `pFilter` (added `% Scan(Lower(sDimParallel) | ':', Lower(pFilter)) > 0`).

#### `}bedrock.cube.data.export`

- **`pFilterParallel` validation:** Identical fix as `data.copy` — one-line change at the same validation point.

#### `}bedrock.cube.data.clear`

- **`pFilterParallel` validation:** Identical fix as `data.copy` and `data.export` — one-line change extending the conflict check to detect `Dim:Hier¦...` notation in `pFilter`.

---

### 2. Value Handling Modernization (`data.copy` and `data.copy.intercube`)

#### `}bedrock.cube.data.copy` and `}bedrock.cube.data.copy.intercube` — Prolog

- **`CubeDimensionCountGet`** replaces the existing `TabDim`-based while-loop for counting cube dimensions. One API call instead of a loop.
- **Precomputed index strings:** `sDimCount` and `sDimCountP1` are computed once from `nDimensionCount` for use in `Expand`-based DataProcedure lookups.
- **`sLastDimName`:** The name of the cube's last dimension is computed once in the Prolog via `TabDim(pCube, nDimensionCount)` (or `TabDim(pTgtCube, ...)` for intercube). Constant across all data rows — avoids a repeated lookup per row.

#### `}bedrock.cube.data.copy` — DataProcedure

- **Single-pass value preparation:** A unified block before the `nDimensionCount` branch now handles all value and type logic once per data row:
  - `sLastEle` is read via `Expand('%v' | sDimCount | '%')`.
  - `sElType = DType(sLastDimName, sLastEle)` replaces the per-block `DType(sDimN, vN)` call.
  - VIEW mode (`pFile = 0`): numeric value from `NValue * nFactor`; string value from `SValue`.
  - CSV mode (`pFile > 0`): value from `Expand('%v' | sDimCountP1 | '%')` with `StringToNumberEx`.
- **Simplified dimension blocks (n = 3..27):** Each block now contains only a `CellIsUpdateable` check and a `CellPutN`/`CellPutS`/`CellIncrementN` call — all type detection and value preparation removed.
- **`CellIncrementN`** replaces `CellGetN + add + CellPutN` for `pCumulate = 1` — atomic operation, eliminates race conditions under parallel execution.
- **`nDimensionCount = 2` block:** Normal string/numeric cells modernized to use `CellIncrementN` and the pre-computed `vString`/`nCbal`. Special-case handling (`}ElementSecurity_*`, attribute cubes `AA`/`AS`/`AN`) is preserved unchanged.

#### `}bedrock.cube.data.copy.intercube` — DataProcedure

- Same single-pass value preparation pattern, adapted for `sV` variables: `sLastEle = Expand('%sV' | sDimCount | '%')`.
- All 26 dimension blocks (n = 2..27) simplified to `CellIsUpdateable` + `CellPutS(sVString)` / `CellIncrementN(nCbal)`.
- `nDimensionCount = 2` block: `ElementSecurityPut` and `AttrPut*` special cases updated to use pre-computed `sVString`/`nCbal`.

---

## Why?

### Hierarchy & MDX Filter Support

TM1/Planning Analytics supports non-default hierarchies and MDX set expressions in view definitions. Without these changes:

- A `pFilter` value of `Year:FiscalYear¦2024` would either fail silently or raise a spurious error because the parser did not understand the `:` separator.
- An MDX expression in `pFilter` would be corrupted by the character-by-character parser treating `+`, `&`, and `¦` inside the MDX block as structural delimiters.
- The `pFilterParallel` conflict check would produce a **false negative** when `pFilter` used `Dim:Hier` notation — the same dimension could be specified in both parameters without triggering a validation error, leading to silent double-filtering or incorrect parallel splits.

### Value Handling Modernization

- **Correctness:** Reading numeric values via `StringToNumberEx` in VIEW mode (`pFile = 0`) is unnecessary — the value is already available as a native number in `NValue`. Any mismatch between the server's actual decimal separator and `pDecimalSeparator` would cause silent wrong values.
- **Performance:** `DType(sDimN, vN)` and `StringToNumberEx(vN, ...)` were repeated in every one of the 27 dimension blocks, even though the result is identical for all blocks at the same data row. Moving this to a single pre-computation eliminates 26 redundant lookups per row.
- **Concurrency safety:** `CellGetN + add + CellPutN` is not atomic — under parallel execution with multiple threads writing to the same cell, intermediate reads can produce incorrect results. `CellIncrementN` is an atomic server-side operation.

---

## Affected Processes

| Process | Changes |
|---|---|
| `}bedrock.cube.view.create` | Hierarchy syntax, MDX syntax, `HIERARCHYELEMENTINDEX`, brace-depth parser, `sTargetFilter` output |
| `}bedrock.cube.data.copy.intercube` | Filter parser extension (Hierarchy + MDX), `pFilterParallel` validation fix, `NValue`/`CellIncrementN`/single-pass value prep |
| `}bedrock.cube.data.copy` | `pFilterParallel` validation fix, `NValue`/`CellIncrementN`/single-pass value prep |
| `}bedrock.cube.data.export` | `pFilterParallel` validation fix only |
| `}bedrock.cube.data.clear` | `pFilterParallel` validation fix only |

`}bedrock.cube.clone`

---

## Backward Compatibility

All changes are fully backward compatible.

| Concern | Status |
|---|---|
| Existing `pFilter` calls without hierarchy syntax | ✅ Unchanged — the parser falls back to `sHierarchy = sDimension` (default hierarchy) when no `:` is present |
| Existing `pFilter` calls without MDX | ✅ Unchanged — the brace-depth counter activates only when `{` is encountered |
| `pDecimalSeparator` / `pThousandSeparator` parameters | ✅ Still used in CSV mode (`pFile > 0`) — no parameters removed or renamed |
| `pCumulate` semantics | ✅ Identical — `CellIncrementN` produces the same result as the previous `CellGetN + add + CellPutN` sequence |
| `DType` result for `nDimensionCount = 2` special cases | ✅ `sElType` is still derived from `DType` — attribute cube detection (`AA`/`AS`/`AN`) and `}ElementSecurity_*` handling are unchanged |
| Calls using plain `DimName¦elem` syntax | ✅ Identical behavior — no syntax changes required for existing callers |
| `}bedrock.cube.data.clear` callers | ✅ No interface change — one-line internal validation fix only |
| `}bedrock.cube.clone` callers | ✅ No interface change — `clone` is not modified |
| Process parameter interfaces | ✅ No parameters added, removed, or renamed in any of the four processes |
@wimgielis
Copy link
Copy Markdown
Contributor

Nice work!

In particular I do like the mdx keyword to provide an MDX. For completeness, would it be possible to add a subset keyword and provide a public subset within the hierarchy/dimension? That would be useful since without it, we either need additional Bedrock calls, or a SubsetGetMDX and then supply with the mdx keyword and maybe other implications.

But for now, thanks a lot!

@PPN-OL
Copy link
Copy Markdown
Author

PPN-OL commented May 18, 2026

Hi.
Thanks. I was thinking about adding subset in filter but came up with that MDX will cover this as well by TM1SubsetToSet([Dimension].[Hierarchy], "SubsetName")

@wimgielis
Copy link
Copy Markdown
Contributor

wimgielis commented May 18, 2026 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants