refactor: Entity hierarchy#6812
Open
tcaiger wants to merge 57 commits into
Open
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
tcaiger
force-pushed
the
epic-entity-hierarchy
branch
2 times, most recently
from
2a9c0f6 to
2ceb65b
Compare
…ble (#6738) * first commit * Update models.ts * Update schemas.ts * Update 20260428000000-createEntityPolygonAndMigrateGis-modifies-schema.js * updates * fixes * Update Entity.js * performance * tweaks * types * tests * generate ids as default * remove string literals
* specs * Update RN-1853-refinement.md * Update RN-1853-refinement.md * Update RN-1853-refinement.md * plan * first cut * Update 20260501000000-addProjectIdToEntityAndDuplicateSharedEntities-modifies-schema.js * fix tests * test * Update schemas.ts * breakup migrations * refactor * Update getDbMigrator.js * revert * fixes * Update 20260501000001-backfillProjectIdsAndDuplicateSharedEntities-modifies-data.js * tweaks
) * feat(database): TUP-3060: project-scoped entity access foundation Splits out of #6761 (consolidated TUP-3065 branch). First of three stacked PRs. Adds the foundation for project-scoped entity lookups post-RN-1853, plus the mechanical fixes to known bare `entity.findOne({ code })` callsites that silently break when codes are duplicated per project. ## Schema - New `project_country` table (`20260507000000-addProjectCountryTable`) — declarative project ↔ country mapping that replaces the implicit `entity_relation` join. Backfilled (`20260507000001-backfillProjectCountry`) from existing `entity_relation` rows where parent is a project entity and child is a country. - `project_country` registered in `initSyncComponents.js` so the table is part of the sync surface. ## Models / types plumbing - New `ProjectCountry` model + record. `id`, `project_id`, `country_id`, `updated_at_sync_tick`. `UNIQUE (project_id, country_id)`. - Type plumbing across `@tupaia/types` (schemas + interface), `tsmodels`, `server-boilerplate` re-export, `entity-server` + `sync-server` test registry fields. ## findOneByCodeInProject - `Entity.findOneByCodeInProject(code, projectId, otherCriteria)` — canonical helper. With `projectId` set, filters `(project_id IS NULL OR project_id = ?)` so structural (world/project/country) entities resolve correctly and sub-country lookups stay in the requested project. Without `projectId`, falls back to bare `findOne({ code })` — documented as such. ## Bare findOne audit fixes Five mechanical callsites where project context was already available: - `SurveyResponseVariablesExtractor.getVariablesByEntityCode` accepts `surveyId` and resolves `survey.project_id` for the lookup. `getParametersFromInput` threads it through. - `exportSurveyResponses` passes its existing `surveyId` query param. - `importSurveyResponses` × 3 callsites — `ANSWER_TRANSFORMERS[ENTITY]` signature gains a `projectId` param, the per-column entity lookup and `constructNewSurveyResponseDetails` both use `survey.project_id`. - `datatrak-web/useEntityByCode` reads `projectId` from `useCurrentUserContext`, threads it through `localContext` to the offline query function, uses `findOneByCodeInProject`. ## Test fixtures - `buildAndInsertProjectsAndHierarchies` rewritten — writes `entity.parent_id` directly + `project_country` for project ↔ country edges, mirrors RN-1853's per-project entity duplication so entity-server / sync-server tests see the same shape as production. - `clearTestData` deletes `project_country` before `entity` (the `country_id` FK is `ON DELETE RESTRICT`). - `CentralSyncManager.syncLookup.test.ts` seeds a `project_country` row to exercise the new bridge. ## Adjacent fixes picked up - vite.config: `react-router` added to dedupe for every package except psss (psss intentionally has v5 + v6 react-router via `react-router-dom-v6`). `@tanstack/react-query` + `@tanstack/react-query-devtools` added to dedupe for the same class of bug. - admin-panel entities table: new "Project" column (`source: 'project.code'`) — useful QA tool for spotting per-project duplicates in the UI. - Spec doc renamed `RN-1853-refinement.md` → `TUP-3056-refinement.md`, contains full audit table and release-path plan. ## What's NOT in this PR - Closure cache rebuild simplification (TUP-3068) — next PR in the stack. - entity_relation consumer retirement (TUP-3065) — third PR in the stack. Stacked PR plan: 1. This PR — TUP-3060 foundation 2. Next — TUP-3068 closure cache rebuild 3. Third — TUP-3065 entity_relation consumer retirement * cleanup
…ld (#6777) * feat(database): TUP-3068: simplify ancestor_descendant_relation rebuild Splits out of #6761. Second of three stacked PRs — stacks on #6776 (TUP-3060 foundation: project_country, ProjectCountry model, findOneByCodeInProject). The `ancestor_descendant_relation` closure cache is kept as the read source for hierarchy walks. What changes is how it's rebuilt — TUP-3068's brief was to simplify the rebuild algorithm now that hierarchy edges live on `entity.parent_id` + `project_country`. The legacy 3-class pipeline (EntityHierarchyCacher → EntityHierarchySubtreeRebuilder → EntityParentChildRelationBuilder) with conditional `entity_relation` vs `entity.parent_id` branching at each hierarchy level collapses into a single recursive CTE per project. ## New: AncestorDescendantCacheBuilder `AncestorDescendantCacheBuilder.rebuildForProject(projectId)` is the whole rebuild: - One recursive CTE walking `entity.parent_id` (sub-country edges) ∪ `project_country` (project ↔ country bridge), scoped to one project's hierarchy. - Wipe-and-rebuild per-project — DELETE for the hierarchy + INSERT ... SELECT from the CTE, inside one transaction. - Filters `child.type NOT IN ('project', 'country')` from the parent_id leg: project.parent_id and country.parent_id both point at the world entity, which is meta in the project-hierarchy model. Without this filter, world surfaces as `parent_code` of every project/country and the frontend walks up into 403s. ## Rewritten: EntityHierarchyCacher Change-handler translators now key on `project_id` (was: `hierarchyId + rootEntityId` pairs). Listens to `entity` / `projectCountry` / `entityHierarchy` changes; each translates to "rebuild these project_ids' caches". ## Deleted - `EntityHierarchySubtreeRebuilder.js` — its job is now the recursive CTE. - `EntityParentChildRelationBuilder.js` — its `entity_relation` / parent_id branching is gone. `entity_parent_child_relation` is no longer maintained (drop is TUP-3066). - Two old test fixture/test files for the deleted classes. ## Redirected - `buildEntityParentChildRelationIfEmpty` — function name preserved for the central-server bootstrap call site, but now checks `ancestor_descendant_relation` (the surviving cache) and invokes `AncestorDescendantCacheBuilder.rebuildAll`. ## Migration TRUNCATE The TUP-3068 data migration (`20260507000001-backfillProjectCountry`) ends with `TRUNCATE ancestor_descendant_relation`. This invalidates stale rows from the pre-3068 cacher (which reference entity ids that RN-1853 duplicated/replaced) so the bootstrap rebuild on next central-server boot sees an empty cache and runs `rebuildAll` against current state. Self-healing for clean prod deploys. The cleanup of `20201006211507-BuildAncestorDescendantRelationCache-modifies-data.js` reduces it to a no-op shell so db-migrate's directory scan still loads cleanly once the old builder it imported is gone. The `repointSurveyResponses` `ANALYZE` step added to the RN-1853 migration is an adjacent perf fix: post-entity-backfill stats are stale, the planner picks a bad plan for the four-table-join repoint UPDATEs (observed 15 min on 400k responses on tom-db; ANALYZE expected to drop runtime to a fraction). ## Entity.js rewrite `getEntitiesFromParentChildRelation` (and its `getAncestorsFromParentChildRelation` / `getDescendantsFromParentChildRelation` wrappers) used to read from `entity_parent_child_relation`. That table is no longer maintained — rewrite walks `entity.parent_id` directly with a recursive CTE, scoped by project via `(project_id IS NULL OR project_id = ?)`. Also drops `getChildrenViaHierarchy` — no live callers, the only reader was the legacy `entity_relation` table lookup. ## Test fixture rewrite (bundled here, not PR1) `buildAndInsertProjectsAndHierarchies` rewritten to write `entity.parent_id` directly + `project_country` for project ↔ country edges, mirroring RN-1853's per-project entity duplication. This belongs with the cacher rewrite (PR2) because the new cacher walks parent_id + project_country and would otherwise find no edges in tests. `clearTestData` orders `project_country` before `entity` (FK is `ON DELETE RESTRICT`). `CentralSyncManager.syncLookup.test.ts` seeds a `project_country` row to exercise the new bridge. ## Tests - New regression suite `Entity/getDescendantsAncestorsProjectScoping.test.js` (9 tests). Triggers `AncestorDescendantCacheBuilder.rebuildForProject` in `beforeEach`, asserts via the public `getDescendants` / `getAncestors` API. - Entity-server test setup explicitly invokes `AncestorDescendantCacheBuilder.rebuildAll()` in `beforeAll` — the change-handler-driven cacher doesn't run in test mode. - Datatrak-web `useSubmitSurveyResponse` — removed the offline shim that manually wrote `entity_parent_child_relation` rows when datatrak inserted new entities. With the cacher rebuilt from `entity.parent_id` directly, no shim needed. Stacked PR plan: 1. #6776 — TUP-3060 foundation (merged) 2. **This PR** — TUP-3068 closure cache rebuild + test fixture rewrite 3. #6778 — TUP-3065 entity_relation consumer retirement * rename * Update EntityHierarchyCacher.js * Update Entity.js * Update AncestorDescendantCacheBuilder.js * Update Entity.js * refactor
…n to project_country (#6778) * feat(database): TUP-3068: simplify ancestor_descendant_relation rebuild Splits out of #6761. Second of three stacked PRs — stacks on #6776 (TUP-3060 foundation: project_country, ProjectCountry model, findOneByCodeInProject). The `ancestor_descendant_relation` closure cache is kept as the read source for hierarchy walks. What changes is how it's rebuilt — TUP-3068's brief was to simplify the rebuild algorithm now that hierarchy edges live on `entity.parent_id` + `project_country`. The legacy 3-class pipeline (EntityHierarchyCacher → EntityHierarchySubtreeRebuilder → EntityParentChildRelationBuilder) with conditional `entity_relation` vs `entity.parent_id` branching at each hierarchy level collapses into a single recursive CTE per project. ## New: AncestorDescendantCacheBuilder `AncestorDescendantCacheBuilder.rebuildForProject(projectId)` is the whole rebuild: - One recursive CTE walking `entity.parent_id` (sub-country edges) ∪ `project_country` (project ↔ country bridge), scoped to one project's hierarchy. - Wipe-and-rebuild per-project — DELETE for the hierarchy + INSERT ... SELECT from the CTE, inside one transaction. - Filters `child.type NOT IN ('project', 'country')` from the parent_id leg: project.parent_id and country.parent_id both point at the world entity, which is meta in the project-hierarchy model. Without this filter, world surfaces as `parent_code` of every project/country and the frontend walks up into 403s. ## Rewritten: EntityHierarchyCacher Change-handler translators now key on `project_id` (was: `hierarchyId + rootEntityId` pairs). Listens to `entity` / `projectCountry` / `entityHierarchy` changes; each translates to "rebuild these project_ids' caches". ## Deleted - `EntityHierarchySubtreeRebuilder.js` — its job is now the recursive CTE. - `EntityParentChildRelationBuilder.js` — its `entity_relation` / parent_id branching is gone. `entity_parent_child_relation` is no longer maintained (drop is TUP-3066). - Two old test fixture/test files for the deleted classes. ## Redirected - `buildEntityParentChildRelationIfEmpty` — function name preserved for the central-server bootstrap call site, but now checks `ancestor_descendant_relation` (the surviving cache) and invokes `AncestorDescendantCacheBuilder.rebuildAll`. ## Migration TRUNCATE The TUP-3068 data migration (`20260507000001-backfillProjectCountry`) ends with `TRUNCATE ancestor_descendant_relation`. This invalidates stale rows from the pre-3068 cacher (which reference entity ids that RN-1853 duplicated/replaced) so the bootstrap rebuild on next central-server boot sees an empty cache and runs `rebuildAll` against current state. Self-healing for clean prod deploys. The cleanup of `20201006211507-BuildAncestorDescendantRelationCache-modifies-data.js` reduces it to a no-op shell so db-migrate's directory scan still loads cleanly once the old builder it imported is gone. The `repointSurveyResponses` `ANALYZE` step added to the RN-1853 migration is an adjacent perf fix: post-entity-backfill stats are stale, the planner picks a bad plan for the four-table-join repoint UPDATEs (observed 15 min on 400k responses on tom-db; ANALYZE expected to drop runtime to a fraction). ## Entity.js rewrite `getEntitiesFromParentChildRelation` (and its `getAncestorsFromParentChildRelation` / `getDescendantsFromParentChildRelation` wrappers) used to read from `entity_parent_child_relation`. That table is no longer maintained — rewrite walks `entity.parent_id` directly with a recursive CTE, scoped by project via `(project_id IS NULL OR project_id = ?)`. Also drops `getChildrenViaHierarchy` — no live callers, the only reader was the legacy `entity_relation` table lookup. ## Test fixture rewrite (bundled here, not PR1) `buildAndInsertProjectsAndHierarchies` rewritten to write `entity.parent_id` directly + `project_country` for project ↔ country edges, mirroring RN-1853's per-project entity duplication. This belongs with the cacher rewrite (PR2) because the new cacher walks parent_id + project_country and would otherwise find no edges in tests. `clearTestData` orders `project_country` before `entity` (FK is `ON DELETE RESTRICT`). `CentralSyncManager.syncLookup.test.ts` seeds a `project_country` row to exercise the new bridge. ## Tests - New regression suite `Entity/getDescendantsAncestorsProjectScoping.test.js` (9 tests). Triggers `AncestorDescendantCacheBuilder.rebuildForProject` in `beforeEach`, asserts via the public `getDescendants` / `getAncestors` API. - Entity-server test setup explicitly invokes `AncestorDescendantCacheBuilder.rebuildAll()` in `beforeAll` — the change-handler-driven cacher doesn't run in test mode. - Datatrak-web `useSubmitSurveyResponse` — removed the offline shim that manually wrote `entity_parent_child_relation` rows when datatrak inserted new entities. With the cacher rebuilt from `entity.parent_id` directly, no shim needed. Stacked PR plan: 1. #6776 — TUP-3060 foundation (merged) 2. **This PR** — TUP-3068 closure cache rebuild + test fixture rewrite 3. #6778 — TUP-3065 entity_relation consumer retirement * refactor(central-server): TUP-3065: switch consumers from entity_relation to project_country Splits out of #6761. Third of three stacked PRs — stacks on #6777 (TUP-3068 closure cache rebuild). #6776 (TUP-3060 foundation) already merged. Mechanical replacement of `entity_relation`-based project↔country lookups with `project_country` joins, plus retiring the now-orphaned `/entityRelations` admin route and pruning multi-hierarchy test fixtures. ## Consumer switches All four consumer paths joined `entity_relation` to discover a project's countries; they now join `project_country` directly. Country code comes from the country entity itself (`entity.code`), not from `entity.country_code`-on-a-child. - `Project.countries()` — single source of truth - `createDashboardRelationsDBFilter` — dashboards filtered by project access - `assertMapOverlaysPermissions` — map overlay access checks - `GETProjects.permissionsFilteredInternally` — project listing - `hasAccessToEntityForVisualisation` / `hasTupaiaAdminAccessToEntityForVisualisation` ## CreateProject `createProjectCountries` writes `project_country` rows on project creation (was: `entity_relation`). ## Retired - `/v1/entityRelations/:id` admin route + its assertion helper - `DeleteEntity` no longer fires `entityRelation.delete` cascade - Multi-hierarchy `Entity.test.js` cases ## Test fixture switch Central-server test fixtures + the database-level `buildAndInsertProjectsAndHierarchies` helper switched to seed `project_country` only (PR2 had it write both to bridge the consumer switch). ## Out of scope - Dropping the legacy tables — TUP-3066, gated on TUP-3067 (MediTrak compat) Stacked PR plan: 1. #6776 — TUP-3060 foundation (merged) 2. #6777 — TUP-3068 closure cache rebuild 3. **This PR** — TUP-3065 entity_relation consumer retirement * rename * Update EntityHierarchyCacher.js * Update Entity.js * Update AncestorDescendantCacheBuilder.js * Update Entity.js * refactor * plan * remove comments * Update Entity.test.js
* feat(database): TUP-3068: simplify ancestor_descendant_relation rebuild Splits out of #6761. Second of three stacked PRs — stacks on #6776 (TUP-3060 foundation: project_country, ProjectCountry model, findOneByCodeInProject). The `ancestor_descendant_relation` closure cache is kept as the read source for hierarchy walks. What changes is how it's rebuilt — TUP-3068's brief was to simplify the rebuild algorithm now that hierarchy edges live on `entity.parent_id` + `project_country`. The legacy 3-class pipeline (EntityHierarchyCacher → EntityHierarchySubtreeRebuilder → EntityParentChildRelationBuilder) with conditional `entity_relation` vs `entity.parent_id` branching at each hierarchy level collapses into a single recursive CTE per project. ## New: AncestorDescendantCacheBuilder `AncestorDescendantCacheBuilder.rebuildForProject(projectId)` is the whole rebuild: - One recursive CTE walking `entity.parent_id` (sub-country edges) ∪ `project_country` (project ↔ country bridge), scoped to one project's hierarchy. - Wipe-and-rebuild per-project — DELETE for the hierarchy + INSERT ... SELECT from the CTE, inside one transaction. - Filters `child.type NOT IN ('project', 'country')` from the parent_id leg: project.parent_id and country.parent_id both point at the world entity, which is meta in the project-hierarchy model. Without this filter, world surfaces as `parent_code` of every project/country and the frontend walks up into 403s. ## Rewritten: EntityHierarchyCacher Change-handler translators now key on `project_id` (was: `hierarchyId + rootEntityId` pairs). Listens to `entity` / `projectCountry` / `entityHierarchy` changes; each translates to "rebuild these project_ids' caches". ## Deleted - `EntityHierarchySubtreeRebuilder.js` — its job is now the recursive CTE. - `EntityParentChildRelationBuilder.js` — its `entity_relation` / parent_id branching is gone. `entity_parent_child_relation` is no longer maintained (drop is TUP-3066). - Two old test fixture/test files for the deleted classes. ## Redirected - `buildEntityParentChildRelationIfEmpty` — function name preserved for the central-server bootstrap call site, but now checks `ancestor_descendant_relation` (the surviving cache) and invokes `AncestorDescendantCacheBuilder.rebuildAll`. ## Migration TRUNCATE The TUP-3068 data migration (`20260507000001-backfillProjectCountry`) ends with `TRUNCATE ancestor_descendant_relation`. This invalidates stale rows from the pre-3068 cacher (which reference entity ids that RN-1853 duplicated/replaced) so the bootstrap rebuild on next central-server boot sees an empty cache and runs `rebuildAll` against current state. Self-healing for clean prod deploys. The cleanup of `20201006211507-BuildAncestorDescendantRelationCache-modifies-data.js` reduces it to a no-op shell so db-migrate's directory scan still loads cleanly once the old builder it imported is gone. The `repointSurveyResponses` `ANALYZE` step added to the RN-1853 migration is an adjacent perf fix: post-entity-backfill stats are stale, the planner picks a bad plan for the four-table-join repoint UPDATEs (observed 15 min on 400k responses on tom-db; ANALYZE expected to drop runtime to a fraction). ## Entity.js rewrite `getEntitiesFromParentChildRelation` (and its `getAncestorsFromParentChildRelation` / `getDescendantsFromParentChildRelation` wrappers) used to read from `entity_parent_child_relation`. That table is no longer maintained — rewrite walks `entity.parent_id` directly with a recursive CTE, scoped by project via `(project_id IS NULL OR project_id = ?)`. Also drops `getChildrenViaHierarchy` — no live callers, the only reader was the legacy `entity_relation` table lookup. ## Test fixture rewrite (bundled here, not PR1) `buildAndInsertProjectsAndHierarchies` rewritten to write `entity.parent_id` directly + `project_country` for project ↔ country edges, mirroring RN-1853's per-project entity duplication. This belongs with the cacher rewrite (PR2) because the new cacher walks parent_id + project_country and would otherwise find no edges in tests. `clearTestData` orders `project_country` before `entity` (FK is `ON DELETE RESTRICT`). `CentralSyncManager.syncLookup.test.ts` seeds a `project_country` row to exercise the new bridge. ## Tests - New regression suite `Entity/getDescendantsAncestorsProjectScoping.test.js` (9 tests). Triggers `AncestorDescendantCacheBuilder.rebuildForProject` in `beforeEach`, asserts via the public `getDescendants` / `getAncestors` API. - Entity-server test setup explicitly invokes `AncestorDescendantCacheBuilder.rebuildAll()` in `beforeAll` — the change-handler-driven cacher doesn't run in test mode. - Datatrak-web `useSubmitSurveyResponse` — removed the offline shim that manually wrote `entity_parent_child_relation` rows when datatrak inserted new entities. With the cacher rebuilt from `entity.parent_id` directly, no shim needed. Stacked PR plan: 1. #6776 — TUP-3060 foundation (merged) 2. **This PR** — TUP-3068 closure cache rebuild + test fixture rewrite 3. #6778 — TUP-3065 entity_relation consumer retirement * refactor(central-server): TUP-3065: switch consumers from entity_relation to project_country Splits out of #6761. Third of three stacked PRs — stacks on #6777 (TUP-3068 closure cache rebuild). #6776 (TUP-3060 foundation) already merged. Mechanical replacement of `entity_relation`-based project↔country lookups with `project_country` joins, plus retiring the now-orphaned `/entityRelations` admin route and pruning multi-hierarchy test fixtures. ## Consumer switches All four consumer paths joined `entity_relation` to discover a project's countries; they now join `project_country` directly. Country code comes from the country entity itself (`entity.code`), not from `entity.country_code`-on-a-child. - `Project.countries()` — single source of truth - `createDashboardRelationsDBFilter` — dashboards filtered by project access - `assertMapOverlaysPermissions` — map overlay access checks - `GETProjects.permissionsFilteredInternally` — project listing - `hasAccessToEntityForVisualisation` / `hasTupaiaAdminAccessToEntityForVisualisation` ## CreateProject `createProjectCountries` writes `project_country` rows on project creation (was: `entity_relation`). ## Retired - `/v1/entityRelations/:id` admin route + its assertion helper - `DeleteEntity` no longer fires `entityRelation.delete` cascade - Multi-hierarchy `Entity.test.js` cases ## Test fixture switch Central-server test fixtures + the database-level `buildAndInsertProjectsAndHierarchies` helper switched to seed `project_country` only (PR2 had it write both to bridge the consumer switch). ## Out of scope - Dropping the legacy tables — TUP-3066, gated on TUP-3067 (MediTrak compat) Stacked PR plan: 1. #6776 — TUP-3060 foundation (merged) 2. #6777 — TUP-3068 closure cache rebuild 3. **This PR** — TUP-3065 entity_relation consumer retirement * rename * Update EntityHierarchyCacher.js * Update Entity.js * Update AncestorDescendantCacheBuilder.js * Update Entity.js * refactor * plan * refactor: TUP-3066: rename hierarchyId → projectId in entity hierarchy code path Each project has exactly one hierarchy, so the indirection through `entity_hierarchy_id` is dead weight. Renames `ancestor_descendant_relation.entity_hierarchy_id` to `project_id` (schema migration + types regen) and ripples the parameter rename `hierarchyId` → `projectId` across: - `Entity.js` hierarchy walk methods (record + model) - `AncestorDescendantRelation.js` getImmediateRelations + caches - `AncestorDescendantCacheBuilder.js` rebuildForProject - entity-server hierarchy routes + middleware (CommonContext) - datatrak-web entity accessors - web-config-server `fetchHierarchyId` → `fetchProjectId` - `fetchDefaultEntityHierarchyIdPatiently` → `fetchDefaultProjectIdPatiently` (rewritten to query projects via ancestor_descendant_relation) Drops: - `getChildrenViaHierarchy` (all callers retired in PR #6778) - `findOneOrThrow({entity_hierarchy_id})` lookup inside `getEntitiesFromParentChildRelation` — closes review-hero comment #3217296820 on PR #6777 Also restores a typo introduced by 90a0ebb (`getEntitiesFromParentChildRelagetEntitiesFromParentChildRelationtion`). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066a: fetchDefaultProjectIdPatiently sort by project.code The project table has no `name` column (that lived on entity_hierarchy). Sort by `code` to mirror the original alphabetical-fallback behaviour. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066a: backport CI fixups (data-broker stub, EventBuilder fixture, types regen) Same drifts surfaced on this PR as on TUP-3066b; smaller subset because this branch only renames the column, doesn't drop the legacy tables: - data-broker DhisService stubs: swap entityHierarchy → project to match EventsPuller.ts's new lookup path - central-server EventBuilder.test.js: ancestor_descendant_relation now keyed by project_id (the column was renamed) - @tupaia/types models.ts + schemas.ts: - Analytics{,Create,Update}.type now QuestionType (analytics MV column is question_type on CI's fresh build) - EntityType enum order: document_group before document, tamanu_country between ird_village and srh_district (matches pg_enum.enumsortorder) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * remove comments * Update Entity.test.js * Update Entity.js * fix: TUP-3066a: address review-hero + bugbot feedback - surveyDataExport.fetchProjectId: return project.id (not entity_hierarchy_id) - migration: delete orphan ancestor_descendant_relation rows before SET NOT NULL - Entity.fetchDefaultProjectIdPatiently: dedupe project_id in SQL via DISTINCT - data-broker models: extract named Project type to match file convention Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…on, entity_hierarchy (#6787) * feat(database): TUP-3068: simplify ancestor_descendant_relation rebuild Splits out of #6761. Second of three stacked PRs — stacks on #6776 (TUP-3060 foundation: project_country, ProjectCountry model, findOneByCodeInProject). The `ancestor_descendant_relation` closure cache is kept as the read source for hierarchy walks. What changes is how it's rebuilt — TUP-3068's brief was to simplify the rebuild algorithm now that hierarchy edges live on `entity.parent_id` + `project_country`. The legacy 3-class pipeline (EntityHierarchyCacher → EntityHierarchySubtreeRebuilder → EntityParentChildRelationBuilder) with conditional `entity_relation` vs `entity.parent_id` branching at each hierarchy level collapses into a single recursive CTE per project. ## New: AncestorDescendantCacheBuilder `AncestorDescendantCacheBuilder.rebuildForProject(projectId)` is the whole rebuild: - One recursive CTE walking `entity.parent_id` (sub-country edges) ∪ `project_country` (project ↔ country bridge), scoped to one project's hierarchy. - Wipe-and-rebuild per-project — DELETE for the hierarchy + INSERT ... SELECT from the CTE, inside one transaction. - Filters `child.type NOT IN ('project', 'country')` from the parent_id leg: project.parent_id and country.parent_id both point at the world entity, which is meta in the project-hierarchy model. Without this filter, world surfaces as `parent_code` of every project/country and the frontend walks up into 403s. ## Rewritten: EntityHierarchyCacher Change-handler translators now key on `project_id` (was: `hierarchyId + rootEntityId` pairs). Listens to `entity` / `projectCountry` / `entityHierarchy` changes; each translates to "rebuild these project_ids' caches". ## Deleted - `EntityHierarchySubtreeRebuilder.js` — its job is now the recursive CTE. - `EntityParentChildRelationBuilder.js` — its `entity_relation` / parent_id branching is gone. `entity_parent_child_relation` is no longer maintained (drop is TUP-3066). - Two old test fixture/test files for the deleted classes. ## Redirected - `buildEntityParentChildRelationIfEmpty` — function name preserved for the central-server bootstrap call site, but now checks `ancestor_descendant_relation` (the surviving cache) and invokes `AncestorDescendantCacheBuilder.rebuildAll`. ## Migration TRUNCATE The TUP-3068 data migration (`20260507000001-backfillProjectCountry`) ends with `TRUNCATE ancestor_descendant_relation`. This invalidates stale rows from the pre-3068 cacher (which reference entity ids that RN-1853 duplicated/replaced) so the bootstrap rebuild on next central-server boot sees an empty cache and runs `rebuildAll` against current state. Self-healing for clean prod deploys. The cleanup of `20201006211507-BuildAncestorDescendantRelationCache-modifies-data.js` reduces it to a no-op shell so db-migrate's directory scan still loads cleanly once the old builder it imported is gone. The `repointSurveyResponses` `ANALYZE` step added to the RN-1853 migration is an adjacent perf fix: post-entity-backfill stats are stale, the planner picks a bad plan for the four-table-join repoint UPDATEs (observed 15 min on 400k responses on tom-db; ANALYZE expected to drop runtime to a fraction). ## Entity.js rewrite `getEntitiesFromParentChildRelation` (and its `getAncestorsFromParentChildRelation` / `getDescendantsFromParentChildRelation` wrappers) used to read from `entity_parent_child_relation`. That table is no longer maintained — rewrite walks `entity.parent_id` directly with a recursive CTE, scoped by project via `(project_id IS NULL OR project_id = ?)`. Also drops `getChildrenViaHierarchy` — no live callers, the only reader was the legacy `entity_relation` table lookup. ## Test fixture rewrite (bundled here, not PR1) `buildAndInsertProjectsAndHierarchies` rewritten to write `entity.parent_id` directly + `project_country` for project ↔ country edges, mirroring RN-1853's per-project entity duplication. This belongs with the cacher rewrite (PR2) because the new cacher walks parent_id + project_country and would otherwise find no edges in tests. `clearTestData` orders `project_country` before `entity` (FK is `ON DELETE RESTRICT`). `CentralSyncManager.syncLookup.test.ts` seeds a `project_country` row to exercise the new bridge. ## Tests - New regression suite `Entity/getDescendantsAncestorsProjectScoping.test.js` (9 tests). Triggers `AncestorDescendantCacheBuilder.rebuildForProject` in `beforeEach`, asserts via the public `getDescendants` / `getAncestors` API. - Entity-server test setup explicitly invokes `AncestorDescendantCacheBuilder.rebuildAll()` in `beforeAll` — the change-handler-driven cacher doesn't run in test mode. - Datatrak-web `useSubmitSurveyResponse` — removed the offline shim that manually wrote `entity_parent_child_relation` rows when datatrak inserted new entities. With the cacher rebuilt from `entity.parent_id` directly, no shim needed. Stacked PR plan: 1. #6776 — TUP-3060 foundation (merged) 2. **This PR** — TUP-3068 closure cache rebuild + test fixture rewrite 3. #6778 — TUP-3065 entity_relation consumer retirement * refactor(central-server): TUP-3065: switch consumers from entity_relation to project_country Splits out of #6761. Third of three stacked PRs — stacks on #6777 (TUP-3068 closure cache rebuild). #6776 (TUP-3060 foundation) already merged. Mechanical replacement of `entity_relation`-based project↔country lookups with `project_country` joins, plus retiring the now-orphaned `/entityRelations` admin route and pruning multi-hierarchy test fixtures. ## Consumer switches All four consumer paths joined `entity_relation` to discover a project's countries; they now join `project_country` directly. Country code comes from the country entity itself (`entity.code`), not from `entity.country_code`-on-a-child. - `Project.countries()` — single source of truth - `createDashboardRelationsDBFilter` — dashboards filtered by project access - `assertMapOverlaysPermissions` — map overlay access checks - `GETProjects.permissionsFilteredInternally` — project listing - `hasAccessToEntityForVisualisation` / `hasTupaiaAdminAccessToEntityForVisualisation` ## CreateProject `createProjectCountries` writes `project_country` rows on project creation (was: `entity_relation`). ## Retired - `/v1/entityRelations/:id` admin route + its assertion helper - `DeleteEntity` no longer fires `entityRelation.delete` cascade - Multi-hierarchy `Entity.test.js` cases ## Test fixture switch Central-server test fixtures + the database-level `buildAndInsertProjectsAndHierarchies` helper switched to seed `project_country` only (PR2 had it write both to bridge the consumer switch). ## Out of scope - Dropping the legacy tables — TUP-3066, gated on TUP-3067 (MediTrak compat) Stacked PR plan: 1. #6776 — TUP-3060 foundation (merged) 2. #6777 — TUP-3068 closure cache rebuild 3. **This PR** — TUP-3065 entity_relation consumer retirement * rename * Update EntityHierarchyCacher.js * Update Entity.js * Update AncestorDescendantCacheBuilder.js * Update Entity.js * refactor * plan * refactor: TUP-3066: rename hierarchyId → projectId in entity hierarchy code path Each project has exactly one hierarchy, so the indirection through `entity_hierarchy_id` is dead weight. Renames `ancestor_descendant_relation.entity_hierarchy_id` to `project_id` (schema migration + types regen) and ripples the parameter rename `hierarchyId` → `projectId` across: - `Entity.js` hierarchy walk methods (record + model) - `AncestorDescendantRelation.js` getImmediateRelations + caches - `AncestorDescendantCacheBuilder.js` rebuildForProject - entity-server hierarchy routes + middleware (CommonContext) - datatrak-web entity accessors - web-config-server `fetchHierarchyId` → `fetchProjectId` - `fetchDefaultEntityHierarchyIdPatiently` → `fetchDefaultProjectIdPatiently` (rewritten to query projects via ancestor_descendant_relation) Drops: - `getChildrenViaHierarchy` (all callers retired in PR #6778) - `findOneOrThrow({entity_hierarchy_id})` lookup inside `getEntitiesFromParentChildRelation` — closes review-hero comment #3217296820 on PR #6777 Also restores a typo introduced by 90a0ebb (`getEntitiesFromParentChildRelagetEntitiesFromParentChildRelationtion`). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: TUP-3066b: drop entity_relation, entity_parent_child_relation, entity_hierarchy Subtractive cleanup once TUP-3066a's rename is in: the closure cache is the only hierarchy reader still standing and it's keyed by project_id, so the three legacy tables can go. ⚠ GATED ON TUP-3067 (MediTrak compatibility). The schema migration must not run in production until mobile sync is no longer reading entity_parent_child_relation through the boilerplate sync lookup. Draft now to review the code shape; merge once 3067 ships. Schema migration drops `project.entity_hierarchy_id` then the three tables in FK-dependency order (down migration restores structure + minimal seed; the relation-row data is reproducible from entity.parent_id + project_country). Code: - Delete model classes: EntityRelation, EntityParentChildRelation, EntityHierarchy - Drop ENTITY_RELATION / ENTITY_HIERARCHY / ENTITY_PARENT_CHILD_RELATION records, syncing-tables list, post-migration trigger list, clearTestData list - Rewrite Entity.buildSyncLookupQueryDetails to compute project_ids without entity_parent_child_relation joins - Retire EntityHierarchyCacher.translateEntityHierarchyChange (the table is gone) - Drop CreateProject.createEntityHierarchy + project.entity_hierarchy_id field - Drop DeleteEntity entity_relation walk; now walks per-project via project_country - Drop test fixture's entity_hierarchy creation - Delete admin routes: - central-server: /entityHierarchy GET + PUT routes + tests - entity-server: /hierarchies route + integration tests - Drop project.entity_hierarchy_id projection in GETProjects + Project.find Types are NOT regenerated in this commit because the legacy tables still exist on the dev DB; the regen happens when the schema migration is applied alongside TUP-3067. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Update Entity.test.js * fix: TUP-3066b: drop type references to deleted entity-hierarchy models Followups noticed during local test setup: - tsmodels: delete EntityHierarchy.ts and EntityParentChildRelation.ts stubs + remove their re-exports from models/index.ts - server-boilerplate: drop EntityHierarchyModel/Record re-export - sync-server: drop deleted models from SyncServerModelRegistry + TestSyncServerModelRegistry shapes - datatrak-web: drop entityHierarchy/entityParentChildRelation fields from DatatrakWebModelRegistry Also: in fetchDefaultProjectIdPatiently, sort projects by `code` instead of `name` (project table has no `name` column — that column lived on entity_hierarchy which is gone). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066a: fetchDefaultProjectIdPatiently sort by project.code The project table has no `name` column (that lived on entity_hierarchy). Sort by `code` to mirror the original alphabetical-fallback behaviour. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066b: clear remaining references to dropped legacy tables in tests + types Catches the CI fallout from PR #6787: - Regenerate @tupaia/types (drops EntityHierarchy / EntityRelation / EntityParentChildRelation from models.ts + schemas.ts) - entity-server src/types.ts + __tests__/types.ts: drop EntityHierarchyModel / EntityRelationModel from registries - data-broker DhisService stubs: replace ENTITY_HIERARCHIES fixture with PROJECTS stub so EventsPuller's models.project.findOne resolves - Database tests: - Entity/getDescendantsAncestorsProjectScoping.test.js: stop creating entity_hierarchy + entity_hierarchy_id on project - Central-server tests: - apiV2/landingPages/utils.js: drop entity_hierarchy create - apiV2/projects/CreateProject.test.js: drop the "creates a valid entity hierarchy record" assertion + entityHierarchy cleanup - apiV2/projects/EditProject.test.js, GETProjects.test.js: same cleanup - apiV2/surveys/CreateAndEditSurveys.test.js: project fixture without entity_hierarchy_id - dhis/EventBuilder.test.js: ancestor_descendant_relation now keyed by project_id; create project instead of entity_hierarchy - Sync-server tests: - CentralSyncManager.pull.test.ts: drop entityHierarchy from prepareData - CentralSyncManager.syncLookup.test.ts: drop entity_parent_child_relation insert; entity2's parent linked via entity.parent_id instead Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066b: include analytics types + clear leftover test references CI surfaced three remaining issues after the first pass: - @tupaia/types Validate: regenerate against a DB that has the analytics materialized view, so `Analytics` / `AnalyticsCreate` / `AnalyticsUpdate` interfaces are present alongside the dropped legacy table cleanup - entity-server permissions.test.ts: remove the "filters hierarchies when requested for some with access" case (route `/v1/hierarchies` is gone) and the unused `getHierarchiesWithFields` import - sync-server: - CentralSyncManager.pull.test.ts: snapshotRecords expectation now 3 (country + userAccount + project) — entityHierarchy isn't seeded anymore - CentralSyncManager.syncLookup.test.ts: drop the now-unused `entity2` let-binding (the entity is still inserted, just not held for later use) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066b: match types regen against CI (analytics.type enum, entity_type order) CI generates types against a freshly-built DB; my local dev DB has older state and produced two drifts: - Analytics{,Create,Update}.type: my dev DB has the analytics MV with column type 'text'. CI's fresh MV has 'question_type', so the generated TS type is QuestionType. Match CI by changing 'string | null' → 'QuestionType | null' in the three Analytics interfaces. - EntityType enum order: pg_enum.enumsortorder differs between my dev DB and CI. Reorder document_group/document and tamanu_country to match CI's insertion order. schemas.ts uses alphabetical ordering and didn't need changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066b: match types schemas.ts against CI regen Three more drifts caught by the Validate types CI step: - ENTITY_TYPE_CHILDREN map (used by UserAccountPreferencesSchema): document_group before document, and tamanu_country between ird_village and srh_district — matches Postgres pg_enum.enumsortorder on CI - Analytics{,Create,Update}Schema.type now includes a `QuestionType` enum array (the analytics MV column is question_type on CI's fresh build). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066a: backport CI fixups (data-broker stub, EventBuilder fixture, types regen) Same drifts surfaced on this PR as on TUP-3066b; smaller subset because this branch only renames the column, doesn't drop the legacy tables: - data-broker DhisService stubs: swap entityHierarchy → project to match EventsPuller.ts's new lookup path - central-server EventBuilder.test.js: ancestor_descendant_relation now keyed by project_id (the column was renamed) - @tupaia/types models.ts + schemas.ts: - Analytics{,Create,Update}.type now QuestionType (analytics MV column is question_type on CI's fresh build) - EntityType enum order: document_group before document, tamanu_country between ird_village and srh_district (matches pg_enum.enumsortorder) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Update index.js * specs * fix: TUP-3066b: restore review-fix changes lost in merge conflict Merging epic-entity-hierarchy into tup-3066b-drop-legacy-tables dropped five fixes that were correctly squash-merged via PRs #6778 and #6786. Restored: - surveyDataExport.fetchProjectId: return project.id (not entity_hierarchy_id) - migration: delete orphan ancestor_descendant_relation rows before SET NOT NULL - Entity.fetchDefaultProjectIdPatiently: dedupe project_id in SQL via DISTINCT - Entity.findOneByCodeInProject: forward 4th options arg (was silently dropped) - data-broker models: extract named Project type to match file convention Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066b: restore central-server project changes lost in merge The recent merge of epic-entity-hierarchy reintroduced entity_hierarchy references in CreateProject.js and the project test rollback helpers. This was the same merge-conflict-lost-work pattern as commit ae6a9d8; restoring from 55ddae5 (last known-good CI state): - CreateProject.js: drop createEntityHierarchy method, its call site, and the entity_hierarchy_id field on project create - CreateProject.test.js: drop "creates a valid entity hierarchy record" test and entityHierarchy.delete from rollbackRecords - EditProject.test.js: drop entityHierarchy.delete from rollbackRecords Fixes the 5 failing central-server tests on CI — the after-each hook crash on the missing entityHierarchy model was cascading sinon stub leakage into the next two test files. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Delete TUP-3066-refinement.md * Update entity-hierarchy-improvements.md * refactor: TUP-3156: delete MS1 sync (#6789) Andrew confirmed in the TUP-3156 refinement that MS1 is no longer used: last response in 2023, prior in 2021. The sync is bare entity-code lookups outside any request context — post-3056 these silently return arbitrary copies of duplicated entities. Cleaner to remove than to project-scope a dead path. Deletes: - packages/central-server/src/ms1/ directory (sync code) - Ms1SyncLog + Ms1SyncQueue model wrappers (only callers were in /ms1/) - startSyncWithMs1 call in index.js Out of scope: - DB tables ms1_sync_queue / ms1_sync_log stay (separate cleanup) - types/EntityMetadata.ms1 field stays (no readers/writers, harmless) - clearTestData entries stay (tables still exist, no-op cleanup is fine) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Update index.js * clean up --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(database): TUP-3068: simplify ancestor_descendant_relation rebuild Splits out of #6761. Second of three stacked PRs — stacks on #6776 (TUP-3060 foundation: project_country, ProjectCountry model, findOneByCodeInProject). The `ancestor_descendant_relation` closure cache is kept as the read source for hierarchy walks. What changes is how it's rebuilt — TUP-3068's brief was to simplify the rebuild algorithm now that hierarchy edges live on `entity.parent_id` + `project_country`. The legacy 3-class pipeline (EntityHierarchyCacher → EntityHierarchySubtreeRebuilder → EntityParentChildRelationBuilder) with conditional `entity_relation` vs `entity.parent_id` branching at each hierarchy level collapses into a single recursive CTE per project. ## New: AncestorDescendantCacheBuilder `AncestorDescendantCacheBuilder.rebuildForProject(projectId)` is the whole rebuild: - One recursive CTE walking `entity.parent_id` (sub-country edges) ∪ `project_country` (project ↔ country bridge), scoped to one project's hierarchy. - Wipe-and-rebuild per-project — DELETE for the hierarchy + INSERT ... SELECT from the CTE, inside one transaction. - Filters `child.type NOT IN ('project', 'country')` from the parent_id leg: project.parent_id and country.parent_id both point at the world entity, which is meta in the project-hierarchy model. Without this filter, world surfaces as `parent_code` of every project/country and the frontend walks up into 403s. ## Rewritten: EntityHierarchyCacher Change-handler translators now key on `project_id` (was: `hierarchyId + rootEntityId` pairs). Listens to `entity` / `projectCountry` / `entityHierarchy` changes; each translates to "rebuild these project_ids' caches". ## Deleted - `EntityHierarchySubtreeRebuilder.js` — its job is now the recursive CTE. - `EntityParentChildRelationBuilder.js` — its `entity_relation` / parent_id branching is gone. `entity_parent_child_relation` is no longer maintained (drop is TUP-3066). - Two old test fixture/test files for the deleted classes. ## Redirected - `buildEntityParentChildRelationIfEmpty` — function name preserved for the central-server bootstrap call site, but now checks `ancestor_descendant_relation` (the surviving cache) and invokes `AncestorDescendantCacheBuilder.rebuildAll`. ## Migration TRUNCATE The TUP-3068 data migration (`20260507000001-backfillProjectCountry`) ends with `TRUNCATE ancestor_descendant_relation`. This invalidates stale rows from the pre-3068 cacher (which reference entity ids that RN-1853 duplicated/replaced) so the bootstrap rebuild on next central-server boot sees an empty cache and runs `rebuildAll` against current state. Self-healing for clean prod deploys. The cleanup of `20201006211507-BuildAncestorDescendantRelationCache-modifies-data.js` reduces it to a no-op shell so db-migrate's directory scan still loads cleanly once the old builder it imported is gone. The `repointSurveyResponses` `ANALYZE` step added to the RN-1853 migration is an adjacent perf fix: post-entity-backfill stats are stale, the planner picks a bad plan for the four-table-join repoint UPDATEs (observed 15 min on 400k responses on tom-db; ANALYZE expected to drop runtime to a fraction). ## Entity.js rewrite `getEntitiesFromParentChildRelation` (and its `getAncestorsFromParentChildRelation` / `getDescendantsFromParentChildRelation` wrappers) used to read from `entity_parent_child_relation`. That table is no longer maintained — rewrite walks `entity.parent_id` directly with a recursive CTE, scoped by project via `(project_id IS NULL OR project_id = ?)`. Also drops `getChildrenViaHierarchy` — no live callers, the only reader was the legacy `entity_relation` table lookup. ## Test fixture rewrite (bundled here, not PR1) `buildAndInsertProjectsAndHierarchies` rewritten to write `entity.parent_id` directly + `project_country` for project ↔ country edges, mirroring RN-1853's per-project entity duplication. This belongs with the cacher rewrite (PR2) because the new cacher walks parent_id + project_country and would otherwise find no edges in tests. `clearTestData` orders `project_country` before `entity` (FK is `ON DELETE RESTRICT`). `CentralSyncManager.syncLookup.test.ts` seeds a `project_country` row to exercise the new bridge. ## Tests - New regression suite `Entity/getDescendantsAncestorsProjectScoping.test.js` (9 tests). Triggers `AncestorDescendantCacheBuilder.rebuildForProject` in `beforeEach`, asserts via the public `getDescendants` / `getAncestors` API. - Entity-server test setup explicitly invokes `AncestorDescendantCacheBuilder.rebuildAll()` in `beforeAll` — the change-handler-driven cacher doesn't run in test mode. - Datatrak-web `useSubmitSurveyResponse` — removed the offline shim that manually wrote `entity_parent_child_relation` rows when datatrak inserted new entities. With the cacher rebuilt from `entity.parent_id` directly, no shim needed. Stacked PR plan: 1. #6776 — TUP-3060 foundation (merged) 2. **This PR** — TUP-3068 closure cache rebuild + test fixture rewrite 3. #6778 — TUP-3065 entity_relation consumer retirement * refactor(central-server): TUP-3065: switch consumers from entity_relation to project_country Splits out of #6761. Third of three stacked PRs — stacks on #6777 (TUP-3068 closure cache rebuild). #6776 (TUP-3060 foundation) already merged. Mechanical replacement of `entity_relation`-based project↔country lookups with `project_country` joins, plus retiring the now-orphaned `/entityRelations` admin route and pruning multi-hierarchy test fixtures. ## Consumer switches All four consumer paths joined `entity_relation` to discover a project's countries; they now join `project_country` directly. Country code comes from the country entity itself (`entity.code`), not from `entity.country_code`-on-a-child. - `Project.countries()` — single source of truth - `createDashboardRelationsDBFilter` — dashboards filtered by project access - `assertMapOverlaysPermissions` — map overlay access checks - `GETProjects.permissionsFilteredInternally` — project listing - `hasAccessToEntityForVisualisation` / `hasTupaiaAdminAccessToEntityForVisualisation` ## CreateProject `createProjectCountries` writes `project_country` rows on project creation (was: `entity_relation`). ## Retired - `/v1/entityRelations/:id` admin route + its assertion helper - `DeleteEntity` no longer fires `entityRelation.delete` cascade - Multi-hierarchy `Entity.test.js` cases ## Test fixture switch Central-server test fixtures + the database-level `buildAndInsertProjectsAndHierarchies` helper switched to seed `project_country` only (PR2 had it write both to bridge the consumer switch). ## Out of scope - Dropping the legacy tables — TUP-3066, gated on TUP-3067 (MediTrak compat) Stacked PR plan: 1. #6776 — TUP-3060 foundation (merged) 2. #6777 — TUP-3068 closure cache rebuild 3. **This PR** — TUP-3065 entity_relation consumer retirement * rename * Update EntityHierarchyCacher.js * Update Entity.js * Update AncestorDescendantCacheBuilder.js * Update Entity.js * refactor * plan * refactor: TUP-3066: rename hierarchyId → projectId in entity hierarchy code path Each project has exactly one hierarchy, so the indirection through `entity_hierarchy_id` is dead weight. Renames `ancestor_descendant_relation.entity_hierarchy_id` to `project_id` (schema migration + types regen) and ripples the parameter rename `hierarchyId` → `projectId` across: - `Entity.js` hierarchy walk methods (record + model) - `AncestorDescendantRelation.js` getImmediateRelations + caches - `AncestorDescendantCacheBuilder.js` rebuildForProject - entity-server hierarchy routes + middleware (CommonContext) - datatrak-web entity accessors - web-config-server `fetchHierarchyId` → `fetchProjectId` - `fetchDefaultEntityHierarchyIdPatiently` → `fetchDefaultProjectIdPatiently` (rewritten to query projects via ancestor_descendant_relation) Drops: - `getChildrenViaHierarchy` (all callers retired in PR #6778) - `findOneOrThrow({entity_hierarchy_id})` lookup inside `getEntitiesFromParentChildRelation` — closes review-hero comment #3217296820 on PR #6777 Also restores a typo introduced by 90a0ebb (`getEntitiesFromParentChildRelagetEntitiesFromParentChildRelationtion`). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: TUP-3066b: drop entity_relation, entity_parent_child_relation, entity_hierarchy Subtractive cleanup once TUP-3066a's rename is in: the closure cache is the only hierarchy reader still standing and it's keyed by project_id, so the three legacy tables can go. ⚠ GATED ON TUP-3067 (MediTrak compatibility). The schema migration must not run in production until mobile sync is no longer reading entity_parent_child_relation through the boilerplate sync lookup. Draft now to review the code shape; merge once 3067 ships. Schema migration drops `project.entity_hierarchy_id` then the three tables in FK-dependency order (down migration restores structure + minimal seed; the relation-row data is reproducible from entity.parent_id + project_country). Code: - Delete model classes: EntityRelation, EntityParentChildRelation, EntityHierarchy - Drop ENTITY_RELATION / ENTITY_HIERARCHY / ENTITY_PARENT_CHILD_RELATION records, syncing-tables list, post-migration trigger list, clearTestData list - Rewrite Entity.buildSyncLookupQueryDetails to compute project_ids without entity_parent_child_relation joins - Retire EntityHierarchyCacher.translateEntityHierarchyChange (the table is gone) - Drop CreateProject.createEntityHierarchy + project.entity_hierarchy_id field - Drop DeleteEntity entity_relation walk; now walks per-project via project_country - Drop test fixture's entity_hierarchy creation - Delete admin routes: - central-server: /entityHierarchy GET + PUT routes + tests - entity-server: /hierarchies route + integration tests - Drop project.entity_hierarchy_id projection in GETProjects + Project.find Types are NOT regenerated in this commit because the legacy tables still exist on the dev DB; the regen happens when the schema migration is applied alongside TUP-3067. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Update Entity.test.js * fix: TUP-3066b: drop type references to deleted entity-hierarchy models Followups noticed during local test setup: - tsmodels: delete EntityHierarchy.ts and EntityParentChildRelation.ts stubs + remove their re-exports from models/index.ts - server-boilerplate: drop EntityHierarchyModel/Record re-export - sync-server: drop deleted models from SyncServerModelRegistry + TestSyncServerModelRegistry shapes - datatrak-web: drop entityHierarchy/entityParentChildRelation fields from DatatrakWebModelRegistry Also: in fetchDefaultProjectIdPatiently, sort projects by `code` instead of `name` (project table has no `name` column — that column lived on entity_hierarchy which is gone). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066a: fetchDefaultProjectIdPatiently sort by project.code The project table has no `name` column (that lived on entity_hierarchy). Sort by `code` to mirror the original alphabetical-fallback behaviour. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066b: clear remaining references to dropped legacy tables in tests + types Catches the CI fallout from PR #6787: - Regenerate @tupaia/types (drops EntityHierarchy / EntityRelation / EntityParentChildRelation from models.ts + schemas.ts) - entity-server src/types.ts + __tests__/types.ts: drop EntityHierarchyModel / EntityRelationModel from registries - data-broker DhisService stubs: replace ENTITY_HIERARCHIES fixture with PROJECTS stub so EventsPuller's models.project.findOne resolves - Database tests: - Entity/getDescendantsAncestorsProjectScoping.test.js: stop creating entity_hierarchy + entity_hierarchy_id on project - Central-server tests: - apiV2/landingPages/utils.js: drop entity_hierarchy create - apiV2/projects/CreateProject.test.js: drop the "creates a valid entity hierarchy record" assertion + entityHierarchy cleanup - apiV2/projects/EditProject.test.js, GETProjects.test.js: same cleanup - apiV2/surveys/CreateAndEditSurveys.test.js: project fixture without entity_hierarchy_id - dhis/EventBuilder.test.js: ancestor_descendant_relation now keyed by project_id; create project instead of entity_hierarchy - Sync-server tests: - CentralSyncManager.pull.test.ts: drop entityHierarchy from prepareData - CentralSyncManager.syncLookup.test.ts: drop entity_parent_child_relation insert; entity2's parent linked via entity.parent_id instead Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066b: include analytics types + clear leftover test references CI surfaced three remaining issues after the first pass: - @tupaia/types Validate: regenerate against a DB that has the analytics materialized view, so `Analytics` / `AnalyticsCreate` / `AnalyticsUpdate` interfaces are present alongside the dropped legacy table cleanup - entity-server permissions.test.ts: remove the "filters hierarchies when requested for some with access" case (route `/v1/hierarchies` is gone) and the unused `getHierarchiesWithFields` import - sync-server: - CentralSyncManager.pull.test.ts: snapshotRecords expectation now 3 (country + userAccount + project) — entityHierarchy isn't seeded anymore - CentralSyncManager.syncLookup.test.ts: drop the now-unused `entity2` let-binding (the entity is still inserted, just not held for later use) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066b: match types regen against CI (analytics.type enum, entity_type order) CI generates types against a freshly-built DB; my local dev DB has older state and produced two drifts: - Analytics{,Create,Update}.type: my dev DB has the analytics MV with column type 'text'. CI's fresh MV has 'question_type', so the generated TS type is QuestionType. Match CI by changing 'string | null' → 'QuestionType | null' in the three Analytics interfaces. - EntityType enum order: pg_enum.enumsortorder differs between my dev DB and CI. Reorder document_group/document and tamanu_country to match CI's insertion order. schemas.ts uses alphabetical ordering and didn't need changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066b: match types schemas.ts against CI regen Three more drifts caught by the Validate types CI step: - ENTITY_TYPE_CHILDREN map (used by UserAccountPreferencesSchema): document_group before document, and tamanu_country between ird_village and srh_district — matches Postgres pg_enum.enumsortorder on CI - Analytics{,Create,Update}Schema.type now includes a `QuestionType` enum array (the analytics MV column is question_type on CI's fresh build). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066a: backport CI fixups (data-broker stub, EventBuilder fixture, types regen) Same drifts surfaced on this PR as on TUP-3066b; smaller subset because this branch only renames the column, doesn't drop the legacy tables: - data-broker DhisService stubs: swap entityHierarchy → project to match EventsPuller.ts's new lookup path - central-server EventBuilder.test.js: ancestor_descendant_relation now keyed by project_id (the column was renamed) - @tupaia/types models.ts + schemas.ts: - Analytics{,Create,Update}.type now QuestionType (analytics MV column is question_type on CI's fresh build) - EntityType enum order: document_group before document, tamanu_country between ird_village and srh_district (matches pg_enum.enumsortorder) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Update index.js * specs * fix: TUP-3066b: restore review-fix changes lost in merge conflict Merging epic-entity-hierarchy into tup-3066b-drop-legacy-tables dropped five fixes that were correctly squash-merged via PRs #6778 and #6786. Restored: - surveyDataExport.fetchProjectId: return project.id (not entity_hierarchy_id) - migration: delete orphan ancestor_descendant_relation rows before SET NOT NULL - Entity.fetchDefaultProjectIdPatiently: dedupe project_id in SQL via DISTINCT - Entity.findOneByCodeInProject: forward 4th options arg (was silently dropped) - data-broker models: extract named Project type to match file convention Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066b: restore central-server project changes lost in merge The recent merge of epic-entity-hierarchy reintroduced entity_hierarchy references in CreateProject.js and the project test rollback helpers. This was the same merge-conflict-lost-work pattern as commit ae6a9d8; restoring from 55ddae5 (last known-good CI state): - CreateProject.js: drop createEntityHierarchy method, its call site, and the entity_hierarchy_id field on project create - CreateProject.test.js: drop "creates a valid entity hierarchy record" test and entityHierarchy.delete from rollbackRecords - EditProject.test.js: drop entityHierarchy.delete from rollbackRecords Fixes the 5 failing central-server tests on CI — the after-each hook crash on the missing entityHierarchy model was cascading sinon stub leakage into the next two test files. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Delete TUP-3066-refinement.md * Update entity-hierarchy-improvements.md * refactor: TUP-3156: delete MS1 sync Andrew confirmed in the TUP-3156 refinement that MS1 is no longer used: last response in 2023, prior in 2021. The sync is bare entity-code lookups outside any request context — post-3056 these silently return arbitrary copies of duplicated entities. Cleaner to remove than to project-scope a dead path. Deletes: - packages/central-server/src/ms1/ directory (sync code) - Ms1SyncLog + Ms1SyncQueue model wrappers (only callers were in /ms1/) - startSyncWithMs1 call in index.js Out of scope: - DB tables ms1_sync_queue / ms1_sync_log stay (separate cleanup) - types/EntityMetadata.ms1 field stays (no readers/writers, harmless) - clearTestData entries stay (tables still exist, no-op cleanup is fine) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: TUP-3156: project-scope KoBo entity lookups + data-broker context Sub-country entity codes are duplicated per project after TUP-3056, so bare findOne({ code }) in the KoBo sync path silently returned an arbitrary copy. KoBo always syncs one project at a time, so we can thread the survey's project_id through. - central-server/kobo/startSyncWithKoBo.js: fetch the survey from the sync group's data_group_code, pass survey.project_id to dataBroker pullSyncGroupResults, and use entity.findOneByCodeInProject in writeKoboDataToTupaia. - data-broker KoBoService: accept projectId in pullSyncGroupResults options, forward to translator. - data-broker KoBoTranslator: accept projectId, use findOneByCodeInProject when present (falls back to bare findOne for safety). One cast at the call site because findOneByCodeInProject inherits its return type from BaseEntityModel and loses data-broker's Entity fields. - DataServiceResolver.getMappingByOrgUnitCode: left as bare findOne, but documented why it's safe — only orgUnit.country_code is consumed downstream, and that's identical across all duplicated copies of a sub-country entity. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: TUP-3156: delete MS1 sync (#6789) Andrew confirmed in the TUP-3156 refinement that MS1 is no longer used: last response in 2023, prior in 2021. The sync is bare entity-code lookups outside any request context — post-3056 these silently return arbitrary copies of duplicated entities. Cleaner to remove than to project-scope a dead path. Deletes: - packages/central-server/src/ms1/ directory (sync code) - Ms1SyncLog + Ms1SyncQueue model wrappers (only callers were in /ms1/) - startSyncWithMs1 call in index.js Out of scope: - DB tables ms1_sync_queue / ms1_sync_log stay (separate cleanup) - types/EntityMetadata.ms1 field stays (no readers/writers, harmless) - clearTestData entries stay (tables still exist, no-op cleanup is fine) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * clean up --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ups (#6791) * feat(database): TUP-3068: simplify ancestor_descendant_relation rebuild Splits out of #6761. Second of three stacked PRs — stacks on #6776 (TUP-3060 foundation: project_country, ProjectCountry model, findOneByCodeInProject). The `ancestor_descendant_relation` closure cache is kept as the read source for hierarchy walks. What changes is how it's rebuilt — TUP-3068's brief was to simplify the rebuild algorithm now that hierarchy edges live on `entity.parent_id` + `project_country`. The legacy 3-class pipeline (EntityHierarchyCacher → EntityHierarchySubtreeRebuilder → EntityParentChildRelationBuilder) with conditional `entity_relation` vs `entity.parent_id` branching at each hierarchy level collapses into a single recursive CTE per project. ## New: AncestorDescendantCacheBuilder `AncestorDescendantCacheBuilder.rebuildForProject(projectId)` is the whole rebuild: - One recursive CTE walking `entity.parent_id` (sub-country edges) ∪ `project_country` (project ↔ country bridge), scoped to one project's hierarchy. - Wipe-and-rebuild per-project — DELETE for the hierarchy + INSERT ... SELECT from the CTE, inside one transaction. - Filters `child.type NOT IN ('project', 'country')` from the parent_id leg: project.parent_id and country.parent_id both point at the world entity, which is meta in the project-hierarchy model. Without this filter, world surfaces as `parent_code` of every project/country and the frontend walks up into 403s. ## Rewritten: EntityHierarchyCacher Change-handler translators now key on `project_id` (was: `hierarchyId + rootEntityId` pairs). Listens to `entity` / `projectCountry` / `entityHierarchy` changes; each translates to "rebuild these project_ids' caches". ## Deleted - `EntityHierarchySubtreeRebuilder.js` — its job is now the recursive CTE. - `EntityParentChildRelationBuilder.js` — its `entity_relation` / parent_id branching is gone. `entity_parent_child_relation` is no longer maintained (drop is TUP-3066). - Two old test fixture/test files for the deleted classes. ## Redirected - `buildEntityParentChildRelationIfEmpty` — function name preserved for the central-server bootstrap call site, but now checks `ancestor_descendant_relation` (the surviving cache) and invokes `AncestorDescendantCacheBuilder.rebuildAll`. ## Migration TRUNCATE The TUP-3068 data migration (`20260507000001-backfillProjectCountry`) ends with `TRUNCATE ancestor_descendant_relation`. This invalidates stale rows from the pre-3068 cacher (which reference entity ids that RN-1853 duplicated/replaced) so the bootstrap rebuild on next central-server boot sees an empty cache and runs `rebuildAll` against current state. Self-healing for clean prod deploys. The cleanup of `20201006211507-BuildAncestorDescendantRelationCache-modifies-data.js` reduces it to a no-op shell so db-migrate's directory scan still loads cleanly once the old builder it imported is gone. The `repointSurveyResponses` `ANALYZE` step added to the RN-1853 migration is an adjacent perf fix: post-entity-backfill stats are stale, the planner picks a bad plan for the four-table-join repoint UPDATEs (observed 15 min on 400k responses on tom-db; ANALYZE expected to drop runtime to a fraction). ## Entity.js rewrite `getEntitiesFromParentChildRelation` (and its `getAncestorsFromParentChildRelation` / `getDescendantsFromParentChildRelation` wrappers) used to read from `entity_parent_child_relation`. That table is no longer maintained — rewrite walks `entity.parent_id` directly with a recursive CTE, scoped by project via `(project_id IS NULL OR project_id = ?)`. Also drops `getChildrenViaHierarchy` — no live callers, the only reader was the legacy `entity_relation` table lookup. ## Test fixture rewrite (bundled here, not PR1) `buildAndInsertProjectsAndHierarchies` rewritten to write `entity.parent_id` directly + `project_country` for project ↔ country edges, mirroring RN-1853's per-project entity duplication. This belongs with the cacher rewrite (PR2) because the new cacher walks parent_id + project_country and would otherwise find no edges in tests. `clearTestData` orders `project_country` before `entity` (FK is `ON DELETE RESTRICT`). `CentralSyncManager.syncLookup.test.ts` seeds a `project_country` row to exercise the new bridge. ## Tests - New regression suite `Entity/getDescendantsAncestorsProjectScoping.test.js` (9 tests). Triggers `AncestorDescendantCacheBuilder.rebuildForProject` in `beforeEach`, asserts via the public `getDescendants` / `getAncestors` API. - Entity-server test setup explicitly invokes `AncestorDescendantCacheBuilder.rebuildAll()` in `beforeAll` — the change-handler-driven cacher doesn't run in test mode. - Datatrak-web `useSubmitSurveyResponse` — removed the offline shim that manually wrote `entity_parent_child_relation` rows when datatrak inserted new entities. With the cacher rebuilt from `entity.parent_id` directly, no shim needed. Stacked PR plan: 1. #6776 — TUP-3060 foundation (merged) 2. **This PR** — TUP-3068 closure cache rebuild + test fixture rewrite 3. #6778 — TUP-3065 entity_relation consumer retirement * refactor(central-server): TUP-3065: switch consumers from entity_relation to project_country Splits out of #6761. Third of three stacked PRs — stacks on #6777 (TUP-3068 closure cache rebuild). #6776 (TUP-3060 foundation) already merged. Mechanical replacement of `entity_relation`-based project↔country lookups with `project_country` joins, plus retiring the now-orphaned `/entityRelations` admin route and pruning multi-hierarchy test fixtures. ## Consumer switches All four consumer paths joined `entity_relation` to discover a project's countries; they now join `project_country` directly. Country code comes from the country entity itself (`entity.code`), not from `entity.country_code`-on-a-child. - `Project.countries()` — single source of truth - `createDashboardRelationsDBFilter` — dashboards filtered by project access - `assertMapOverlaysPermissions` — map overlay access checks - `GETProjects.permissionsFilteredInternally` — project listing - `hasAccessToEntityForVisualisation` / `hasTupaiaAdminAccessToEntityForVisualisation` ## CreateProject `createProjectCountries` writes `project_country` rows on project creation (was: `entity_relation`). ## Retired - `/v1/entityRelations/:id` admin route + its assertion helper - `DeleteEntity` no longer fires `entityRelation.delete` cascade - Multi-hierarchy `Entity.test.js` cases ## Test fixture switch Central-server test fixtures + the database-level `buildAndInsertProjectsAndHierarchies` helper switched to seed `project_country` only (PR2 had it write both to bridge the consumer switch). ## Out of scope - Dropping the legacy tables — TUP-3066, gated on TUP-3067 (MediTrak compat) Stacked PR plan: 1. #6776 — TUP-3060 foundation (merged) 2. #6777 — TUP-3068 closure cache rebuild 3. **This PR** — TUP-3065 entity_relation consumer retirement * rename * Update EntityHierarchyCacher.js * Update Entity.js * Update AncestorDescendantCacheBuilder.js * Update Entity.js * refactor * plan * refactor: TUP-3066: rename hierarchyId → projectId in entity hierarchy code path Each project has exactly one hierarchy, so the indirection through `entity_hierarchy_id` is dead weight. Renames `ancestor_descendant_relation.entity_hierarchy_id` to `project_id` (schema migration + types regen) and ripples the parameter rename `hierarchyId` → `projectId` across: - `Entity.js` hierarchy walk methods (record + model) - `AncestorDescendantRelation.js` getImmediateRelations + caches - `AncestorDescendantCacheBuilder.js` rebuildForProject - entity-server hierarchy routes + middleware (CommonContext) - datatrak-web entity accessors - web-config-server `fetchHierarchyId` → `fetchProjectId` - `fetchDefaultEntityHierarchyIdPatiently` → `fetchDefaultProjectIdPatiently` (rewritten to query projects via ancestor_descendant_relation) Drops: - `getChildrenViaHierarchy` (all callers retired in PR #6778) - `findOneOrThrow({entity_hierarchy_id})` lookup inside `getEntitiesFromParentChildRelation` — closes review-hero comment #3217296820 on PR #6777 Also restores a typo introduced by 90a0ebb (`getEntitiesFromParentChildRelagetEntitiesFromParentChildRelationtion`). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: TUP-3066b: drop entity_relation, entity_parent_child_relation, entity_hierarchy Subtractive cleanup once TUP-3066a's rename is in: the closure cache is the only hierarchy reader still standing and it's keyed by project_id, so the three legacy tables can go. ⚠ GATED ON TUP-3067 (MediTrak compatibility). The schema migration must not run in production until mobile sync is no longer reading entity_parent_child_relation through the boilerplate sync lookup. Draft now to review the code shape; merge once 3067 ships. Schema migration drops `project.entity_hierarchy_id` then the three tables in FK-dependency order (down migration restores structure + minimal seed; the relation-row data is reproducible from entity.parent_id + project_country). Code: - Delete model classes: EntityRelation, EntityParentChildRelation, EntityHierarchy - Drop ENTITY_RELATION / ENTITY_HIERARCHY / ENTITY_PARENT_CHILD_RELATION records, syncing-tables list, post-migration trigger list, clearTestData list - Rewrite Entity.buildSyncLookupQueryDetails to compute project_ids without entity_parent_child_relation joins - Retire EntityHierarchyCacher.translateEntityHierarchyChange (the table is gone) - Drop CreateProject.createEntityHierarchy + project.entity_hierarchy_id field - Drop DeleteEntity entity_relation walk; now walks per-project via project_country - Drop test fixture's entity_hierarchy creation - Delete admin routes: - central-server: /entityHierarchy GET + PUT routes + tests - entity-server: /hierarchies route + integration tests - Drop project.entity_hierarchy_id projection in GETProjects + Project.find Types are NOT regenerated in this commit because the legacy tables still exist on the dev DB; the regen happens when the schema migration is applied alongside TUP-3067. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Update Entity.test.js * fix: TUP-3066b: drop type references to deleted entity-hierarchy models Followups noticed during local test setup: - tsmodels: delete EntityHierarchy.ts and EntityParentChildRelation.ts stubs + remove their re-exports from models/index.ts - server-boilerplate: drop EntityHierarchyModel/Record re-export - sync-server: drop deleted models from SyncServerModelRegistry + TestSyncServerModelRegistry shapes - datatrak-web: drop entityHierarchy/entityParentChildRelation fields from DatatrakWebModelRegistry Also: in fetchDefaultProjectIdPatiently, sort projects by `code` instead of `name` (project table has no `name` column — that column lived on entity_hierarchy which is gone). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066a: fetchDefaultProjectIdPatiently sort by project.code The project table has no `name` column (that lived on entity_hierarchy). Sort by `code` to mirror the original alphabetical-fallback behaviour. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066b: clear remaining references to dropped legacy tables in tests + types Catches the CI fallout from PR #6787: - Regenerate @tupaia/types (drops EntityHierarchy / EntityRelation / EntityParentChildRelation from models.ts + schemas.ts) - entity-server src/types.ts + __tests__/types.ts: drop EntityHierarchyModel / EntityRelationModel from registries - data-broker DhisService stubs: replace ENTITY_HIERARCHIES fixture with PROJECTS stub so EventsPuller's models.project.findOne resolves - Database tests: - Entity/getDescendantsAncestorsProjectScoping.test.js: stop creating entity_hierarchy + entity_hierarchy_id on project - Central-server tests: - apiV2/landingPages/utils.js: drop entity_hierarchy create - apiV2/projects/CreateProject.test.js: drop the "creates a valid entity hierarchy record" assertion + entityHierarchy cleanup - apiV2/projects/EditProject.test.js, GETProjects.test.js: same cleanup - apiV2/surveys/CreateAndEditSurveys.test.js: project fixture without entity_hierarchy_id - dhis/EventBuilder.test.js: ancestor_descendant_relation now keyed by project_id; create project instead of entity_hierarchy - Sync-server tests: - CentralSyncManager.pull.test.ts: drop entityHierarchy from prepareData - CentralSyncManager.syncLookup.test.ts: drop entity_parent_child_relation insert; entity2's parent linked via entity.parent_id instead Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066b: include analytics types + clear leftover test references CI surfaced three remaining issues after the first pass: - @tupaia/types Validate: regenerate against a DB that has the analytics materialized view, so `Analytics` / `AnalyticsCreate` / `AnalyticsUpdate` interfaces are present alongside the dropped legacy table cleanup - entity-server permissions.test.ts: remove the "filters hierarchies when requested for some with access" case (route `/v1/hierarchies` is gone) and the unused `getHierarchiesWithFields` import - sync-server: - CentralSyncManager.pull.test.ts: snapshotRecords expectation now 3 (country + userAccount + project) — entityHierarchy isn't seeded anymore - CentralSyncManager.syncLookup.test.ts: drop the now-unused `entity2` let-binding (the entity is still inserted, just not held for later use) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066b: match types regen against CI (analytics.type enum, entity_type order) CI generates types against a freshly-built DB; my local dev DB has older state and produced two drifts: - Analytics{,Create,Update}.type: my dev DB has the analytics MV with column type 'text'. CI's fresh MV has 'question_type', so the generated TS type is QuestionType. Match CI by changing 'string | null' → 'QuestionType | null' in the three Analytics interfaces. - EntityType enum order: pg_enum.enumsortorder differs between my dev DB and CI. Reorder document_group/document and tamanu_country to match CI's insertion order. schemas.ts uses alphabetical ordering and didn't need changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066b: match types schemas.ts against CI regen Three more drifts caught by the Validate types CI step: - ENTITY_TYPE_CHILDREN map (used by UserAccountPreferencesSchema): document_group before document, and tamanu_country between ird_village and srh_district — matches Postgres pg_enum.enumsortorder on CI - Analytics{,Create,Update}Schema.type now includes a `QuestionType` enum array (the analytics MV column is question_type on CI's fresh build). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066a: backport CI fixups (data-broker stub, EventBuilder fixture, types regen) Same drifts surfaced on this PR as on TUP-3066b; smaller subset because this branch only renames the column, doesn't drop the legacy tables: - data-broker DhisService stubs: swap entityHierarchy → project to match EventsPuller.ts's new lookup path - central-server EventBuilder.test.js: ancestor_descendant_relation now keyed by project_id (the column was renamed) - @tupaia/types models.ts + schemas.ts: - Analytics{,Create,Update}.type now QuestionType (analytics MV column is question_type on CI's fresh build) - EntityType enum order: document_group before document, tamanu_country between ird_village and srh_district (matches pg_enum.enumsortorder) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Update index.js * specs * fix: TUP-3066b: restore review-fix changes lost in merge conflict Merging epic-entity-hierarchy into tup-3066b-drop-legacy-tables dropped five fixes that were correctly squash-merged via PRs #6778 and #6786. Restored: - surveyDataExport.fetchProjectId: return project.id (not entity_hierarchy_id) - migration: delete orphan ancestor_descendant_relation rows before SET NOT NULL - Entity.fetchDefaultProjectIdPatiently: dedupe project_id in SQL via DISTINCT - Entity.findOneByCodeInProject: forward 4th options arg (was silently dropped) - data-broker models: extract named Project type to match file convention Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066b: restore central-server project changes lost in merge The recent merge of epic-entity-hierarchy reintroduced entity_hierarchy references in CreateProject.js and the project test rollback helpers. This was the same merge-conflict-lost-work pattern as commit ae6a9d8; restoring from 55ddae5 (last known-good CI state): - CreateProject.js: drop createEntityHierarchy method, its call site, and the entity_hierarchy_id field on project create - CreateProject.test.js: drop "creates a valid entity hierarchy record" test and entityHierarchy.delete from rollbackRecords - EditProject.test.js: drop entityHierarchy.delete from rollbackRecords Fixes the 5 failing central-server tests on CI — the after-each hook crash on the missing entityHierarchy model was cascading sinon stub leakage into the next two test files. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Delete TUP-3066-refinement.md * Update entity-hierarchy-improvements.md * refactor: TUP-3156: delete MS1 sync Andrew confirmed in the TUP-3156 refinement that MS1 is no longer used: last response in 2023, prior in 2021. The sync is bare entity-code lookups outside any request context — post-3056 these silently return arbitrary copies of duplicated entities. Cleaner to remove than to project-scope a dead path. Deletes: - packages/central-server/src/ms1/ directory (sync code) - Ms1SyncLog + Ms1SyncQueue model wrappers (only callers were in /ms1/) - startSyncWithMs1 call in index.js Out of scope: - DB tables ms1_sync_queue / ms1_sync_log stay (separate cleanup) - types/EntityMetadata.ms1 field stays (no readers/writers, harmless) - clearTestData entries stay (tables still exist, no-op cleanup is fine) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: TUP-3156: project-scope KoBo entity lookups + data-broker context Sub-country entity codes are duplicated per project after TUP-3056, so bare findOne({ code }) in the KoBo sync path silently returned an arbitrary copy. KoBo always syncs one project at a time, so we can thread the survey's project_id through. - central-server/kobo/startSyncWithKoBo.js: fetch the survey from the sync group's data_group_code, pass survey.project_id to dataBroker pullSyncGroupResults, and use entity.findOneByCodeInProject in writeKoboDataToTupaia. - data-broker KoBoService: accept projectId in pullSyncGroupResults options, forward to translator. - data-broker KoBoTranslator: accept projectId, use findOneByCodeInProject when present (falls back to bare findOne for safety). One cast at the call site because findOneByCodeInProject inherits its return type from BaseEntityModel and loses data-broker's Entity fields. - DataServiceResolver.getMappingByOrgUnitCode: left as bare findOne, but documented why it's safe — only orgUnit.country_code is consumed downstream, and that's identical across all duplicated copies of a sub-country entity. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: TUP-3156: project-scope web-config-server apiV1 entity lookups The legacy apiV1 RouteHandler base and its data-aggregation subclass both did bare entity.findOne({ code }), which after TUP-3056 silently returns an arbitrary copy of a duplicated-per-project sub-country entity. Same pattern was repeated in seven downstream helpers. Fixed where project context is available: - RouteHandler.handleRequest: fetch project before entity, scope entity lookup. Reorders existing work — project was already needed later for permission checks. - DataAggregatingRouteHandler.fetchAndFilterDataSourceEntitiesOfType: uses this.project (populated by RouteHandler). - measureData.js: uses this.project (subclass of DataAggregatingRouteHandler). - dataBuilders/helpers/groupEvents.js: projectId already destructured from params. - dataBuilders/helpers/calculateOperationForAnalytics.js: projectId already in config. Documented as safe (bare lookup OK because only stable-across-duplicates fields are consumed): - measureBuilders/helpers.js mapMeasureDataToCountries: consumes only country_code. - dataBuilders/helpers/mapAnalyticsToCountries.js: consumes only country_code. - fetchIndicatorValues/fetchAggregatedAnalyticsByDhisIds.js: consumes only .code (= input). Skipped (no project context plumbed to that call path): - composePercentagesPerPeriodByOrgUnit.js Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * clean * refactor: TUP-3156: apply PR #6791 review feedback - RouteHandler: extract `get projectId()` getter to DRY `this.project?.id ?? null` across RouteHandler, DataAggregatingRouteHandler, measureData - RouteHandler: document the null-projectId fallback as a deliberate TUP-3054 interim — every route needs projectCode before we can remove it - groupEvents: null-guard parentOrgUnit before .getDescendantsOfType (the project-scoped lookup is narrower and can now return null where the bare findOne wouldn't have) - fetchAggregatedAnalyticsByDhisIds: upgrade from comment-only to actual scoping — projectId was already in the function signature Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: TUP-3156: update groupEvents mock for findOneByCodeInProject The groupEvents helper switched from models.entity.findOne to findOneByCodeInProject in PR #6791. The unit test mock still only defined findOne, so all five tests in this suite blew up on CI with "models.entity.findOneByCodeInProject is not a function". Extracted the lookup switch into a helper and mocked both methods against it so future refactors don't have to duplicate the data. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Update entity-hierarchy-improvements.md --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ec rollup (#6792) * feat(database): TUP-3068: simplify ancestor_descendant_relation rebuild Splits out of #6761. Second of three stacked PRs — stacks on #6776 (TUP-3060 foundation: project_country, ProjectCountry model, findOneByCodeInProject). The `ancestor_descendant_relation` closure cache is kept as the read source for hierarchy walks. What changes is how it's rebuilt — TUP-3068's brief was to simplify the rebuild algorithm now that hierarchy edges live on `entity.parent_id` + `project_country`. The legacy 3-class pipeline (EntityHierarchyCacher → EntityHierarchySubtreeRebuilder → EntityParentChildRelationBuilder) with conditional `entity_relation` vs `entity.parent_id` branching at each hierarchy level collapses into a single recursive CTE per project. ## New: AncestorDescendantCacheBuilder `AncestorDescendantCacheBuilder.rebuildForProject(projectId)` is the whole rebuild: - One recursive CTE walking `entity.parent_id` (sub-country edges) ∪ `project_country` (project ↔ country bridge), scoped to one project's hierarchy. - Wipe-and-rebuild per-project — DELETE for the hierarchy + INSERT ... SELECT from the CTE, inside one transaction. - Filters `child.type NOT IN ('project', 'country')` from the parent_id leg: project.parent_id and country.parent_id both point at the world entity, which is meta in the project-hierarchy model. Without this filter, world surfaces as `parent_code` of every project/country and the frontend walks up into 403s. ## Rewritten: EntityHierarchyCacher Change-handler translators now key on `project_id` (was: `hierarchyId + rootEntityId` pairs). Listens to `entity` / `projectCountry` / `entityHierarchy` changes; each translates to "rebuild these project_ids' caches". ## Deleted - `EntityHierarchySubtreeRebuilder.js` — its job is now the recursive CTE. - `EntityParentChildRelationBuilder.js` — its `entity_relation` / parent_id branching is gone. `entity_parent_child_relation` is no longer maintained (drop is TUP-3066). - Two old test fixture/test files for the deleted classes. ## Redirected - `buildEntityParentChildRelationIfEmpty` — function name preserved for the central-server bootstrap call site, but now checks `ancestor_descendant_relation` (the surviving cache) and invokes `AncestorDescendantCacheBuilder.rebuildAll`. ## Migration TRUNCATE The TUP-3068 data migration (`20260507000001-backfillProjectCountry`) ends with `TRUNCATE ancestor_descendant_relation`. This invalidates stale rows from the pre-3068 cacher (which reference entity ids that RN-1853 duplicated/replaced) so the bootstrap rebuild on next central-server boot sees an empty cache and runs `rebuildAll` against current state. Self-healing for clean prod deploys. The cleanup of `20201006211507-BuildAncestorDescendantRelationCache-modifies-data.js` reduces it to a no-op shell so db-migrate's directory scan still loads cleanly once the old builder it imported is gone. The `repointSurveyResponses` `ANALYZE` step added to the RN-1853 migration is an adjacent perf fix: post-entity-backfill stats are stale, the planner picks a bad plan for the four-table-join repoint UPDATEs (observed 15 min on 400k responses on tom-db; ANALYZE expected to drop runtime to a fraction). ## Entity.js rewrite `getEntitiesFromParentChildRelation` (and its `getAncestorsFromParentChildRelation` / `getDescendantsFromParentChildRelation` wrappers) used to read from `entity_parent_child_relation`. That table is no longer maintained — rewrite walks `entity.parent_id` directly with a recursive CTE, scoped by project via `(project_id IS NULL OR project_id = ?)`. Also drops `getChildrenViaHierarchy` — no live callers, the only reader was the legacy `entity_relation` table lookup. ## Test fixture rewrite (bundled here, not PR1) `buildAndInsertProjectsAndHierarchies` rewritten to write `entity.parent_id` directly + `project_country` for project ↔ country edges, mirroring RN-1853's per-project entity duplication. This belongs with the cacher rewrite (PR2) because the new cacher walks parent_id + project_country and would otherwise find no edges in tests. `clearTestData` orders `project_country` before `entity` (FK is `ON DELETE RESTRICT`). `CentralSyncManager.syncLookup.test.ts` seeds a `project_country` row to exercise the new bridge. ## Tests - New regression suite `Entity/getDescendantsAncestorsProjectScoping.test.js` (9 tests). Triggers `AncestorDescendantCacheBuilder.rebuildForProject` in `beforeEach`, asserts via the public `getDescendants` / `getAncestors` API. - Entity-server test setup explicitly invokes `AncestorDescendantCacheBuilder.rebuildAll()` in `beforeAll` — the change-handler-driven cacher doesn't run in test mode. - Datatrak-web `useSubmitSurveyResponse` — removed the offline shim that manually wrote `entity_parent_child_relation` rows when datatrak inserted new entities. With the cacher rebuilt from `entity.parent_id` directly, no shim needed. Stacked PR plan: 1. #6776 — TUP-3060 foundation (merged) 2. **This PR** — TUP-3068 closure cache rebuild + test fixture rewrite 3. #6778 — TUP-3065 entity_relation consumer retirement * refactor(central-server): TUP-3065: switch consumers from entity_relation to project_country Splits out of #6761. Third of three stacked PRs — stacks on #6777 (TUP-3068 closure cache rebuild). #6776 (TUP-3060 foundation) already merged. Mechanical replacement of `entity_relation`-based project↔country lookups with `project_country` joins, plus retiring the now-orphaned `/entityRelations` admin route and pruning multi-hierarchy test fixtures. ## Consumer switches All four consumer paths joined `entity_relation` to discover a project's countries; they now join `project_country` directly. Country code comes from the country entity itself (`entity.code`), not from `entity.country_code`-on-a-child. - `Project.countries()` — single source of truth - `createDashboardRelationsDBFilter` — dashboards filtered by project access - `assertMapOverlaysPermissions` — map overlay access checks - `GETProjects.permissionsFilteredInternally` — project listing - `hasAccessToEntityForVisualisation` / `hasTupaiaAdminAccessToEntityForVisualisation` ## CreateProject `createProjectCountries` writes `project_country` rows on project creation (was: `entity_relation`). ## Retired - `/v1/entityRelations/:id` admin route + its assertion helper - `DeleteEntity` no longer fires `entityRelation.delete` cascade - Multi-hierarchy `Entity.test.js` cases ## Test fixture switch Central-server test fixtures + the database-level `buildAndInsertProjectsAndHierarchies` helper switched to seed `project_country` only (PR2 had it write both to bridge the consumer switch). ## Out of scope - Dropping the legacy tables — TUP-3066, gated on TUP-3067 (MediTrak compat) Stacked PR plan: 1. #6776 — TUP-3060 foundation (merged) 2. #6777 — TUP-3068 closure cache rebuild 3. **This PR** — TUP-3065 entity_relation consumer retirement * rename * Update EntityHierarchyCacher.js * Update Entity.js * Update AncestorDescendantCacheBuilder.js * Update Entity.js * refactor * plan * refactor: TUP-3066: rename hierarchyId → projectId in entity hierarchy code path Each project has exactly one hierarchy, so the indirection through `entity_hierarchy_id` is dead weight. Renames `ancestor_descendant_relation.entity_hierarchy_id` to `project_id` (schema migration + types regen) and ripples the parameter rename `hierarchyId` → `projectId` across: - `Entity.js` hierarchy walk methods (record + model) - `AncestorDescendantRelation.js` getImmediateRelations + caches - `AncestorDescendantCacheBuilder.js` rebuildForProject - entity-server hierarchy routes + middleware (CommonContext) - datatrak-web entity accessors - web-config-server `fetchHierarchyId` → `fetchProjectId` - `fetchDefaultEntityHierarchyIdPatiently` → `fetchDefaultProjectIdPatiently` (rewritten to query projects via ancestor_descendant_relation) Drops: - `getChildrenViaHierarchy` (all callers retired in PR #6778) - `findOneOrThrow({entity_hierarchy_id})` lookup inside `getEntitiesFromParentChildRelation` — closes review-hero comment #3217296820 on PR #6777 Also restores a typo introduced by 90a0ebb (`getEntitiesFromParentChildRelagetEntitiesFromParentChildRelationtion`). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: TUP-3066b: drop entity_relation, entity_parent_child_relation, entity_hierarchy Subtractive cleanup once TUP-3066a's rename is in: the closure cache is the only hierarchy reader still standing and it's keyed by project_id, so the three legacy tables can go. ⚠ GATED ON TUP-3067 (MediTrak compatibility). The schema migration must not run in production until mobile sync is no longer reading entity_parent_child_relation through the boilerplate sync lookup. Draft now to review the code shape; merge once 3067 ships. Schema migration drops `project.entity_hierarchy_id` then the three tables in FK-dependency order (down migration restores structure + minimal seed; the relation-row data is reproducible from entity.parent_id + project_country). Code: - Delete model classes: EntityRelation, EntityParentChildRelation, EntityHierarchy - Drop ENTITY_RELATION / ENTITY_HIERARCHY / ENTITY_PARENT_CHILD_RELATION records, syncing-tables list, post-migration trigger list, clearTestData list - Rewrite Entity.buildSyncLookupQueryDetails to compute project_ids without entity_parent_child_relation joins - Retire EntityHierarchyCacher.translateEntityHierarchyChange (the table is gone) - Drop CreateProject.createEntityHierarchy + project.entity_hierarchy_id field - Drop DeleteEntity entity_relation walk; now walks per-project via project_country - Drop test fixture's entity_hierarchy creation - Delete admin routes: - central-server: /entityHierarchy GET + PUT routes + tests - entity-server: /hierarchies route + integration tests - Drop project.entity_hierarchy_id projection in GETProjects + Project.find Types are NOT regenerated in this commit because the legacy tables still exist on the dev DB; the regen happens when the schema migration is applied alongside TUP-3067. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Update Entity.test.js * fix: TUP-3066b: drop type references to deleted entity-hierarchy models Followups noticed during local test setup: - tsmodels: delete EntityHierarchy.ts and EntityParentChildRelation.ts stubs + remove their re-exports from models/index.ts - server-boilerplate: drop EntityHierarchyModel/Record re-export - sync-server: drop deleted models from SyncServerModelRegistry + TestSyncServerModelRegistry shapes - datatrak-web: drop entityHierarchy/entityParentChildRelation fields from DatatrakWebModelRegistry Also: in fetchDefaultProjectIdPatiently, sort projects by `code` instead of `name` (project table has no `name` column — that column lived on entity_hierarchy which is gone). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066a: fetchDefaultProjectIdPatiently sort by project.code The project table has no `name` column (that lived on entity_hierarchy). Sort by `code` to mirror the original alphabetical-fallback behaviour. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066b: clear remaining references to dropped legacy tables in tests + types Catches the CI fallout from PR #6787: - Regenerate @tupaia/types (drops EntityHierarchy / EntityRelation / EntityParentChildRelation from models.ts + schemas.ts) - entity-server src/types.ts + __tests__/types.ts: drop EntityHierarchyModel / EntityRelationModel from registries - data-broker DhisService stubs: replace ENTITY_HIERARCHIES fixture with PROJECTS stub so EventsPuller's models.project.findOne resolves - Database tests: - Entity/getDescendantsAncestorsProjectScoping.test.js: stop creating entity_hierarchy + entity_hierarchy_id on project - Central-server tests: - apiV2/landingPages/utils.js: drop entity_hierarchy create - apiV2/projects/CreateProject.test.js: drop the "creates a valid entity hierarchy record" assertion + entityHierarchy cleanup - apiV2/projects/EditProject.test.js, GETProjects.test.js: same cleanup - apiV2/surveys/CreateAndEditSurveys.test.js: project fixture without entity_hierarchy_id - dhis/EventBuilder.test.js: ancestor_descendant_relation now keyed by project_id; create project instead of entity_hierarchy - Sync-server tests: - CentralSyncManager.pull.test.ts: drop entityHierarchy from prepareData - CentralSyncManager.syncLookup.test.ts: drop entity_parent_child_relation insert; entity2's parent linked via entity.parent_id instead Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066b: include analytics types + clear leftover test references CI surfaced three remaining issues after the first pass: - @tupaia/types Validate: regenerate against a DB that has the analytics materialized view, so `Analytics` / `AnalyticsCreate` / `AnalyticsUpdate` interfaces are present alongside the dropped legacy table cleanup - entity-server permissions.test.ts: remove the "filters hierarchies when requested for some with access" case (route `/v1/hierarchies` is gone) and the unused `getHierarchiesWithFields` import - sync-server: - CentralSyncManager.pull.test.ts: snapshotRecords expectation now 3 (country + userAccount + project) — entityHierarchy isn't seeded anymore - CentralSyncManager.syncLookup.test.ts: drop the now-unused `entity2` let-binding (the entity is still inserted, just not held for later use) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066b: match types regen against CI (analytics.type enum, entity_type order) CI generates types against a freshly-built DB; my local dev DB has older state and produced two drifts: - Analytics{,Create,Update}.type: my dev DB has the analytics MV with column type 'text'. CI's fresh MV has 'question_type', so the generated TS type is QuestionType. Match CI by changing 'string | null' → 'QuestionType | null' in the three Analytics interfaces. - EntityType enum order: pg_enum.enumsortorder differs between my dev DB and CI. Reorder document_group/document and tamanu_country to match CI's insertion order. schemas.ts uses alphabetical ordering and didn't need changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066b: match types schemas.ts against CI regen Three more drifts caught by the Validate types CI step: - ENTITY_TYPE_CHILDREN map (used by UserAccountPreferencesSchema): document_group before document, and tamanu_country between ird_village and srh_district — matches Postgres pg_enum.enumsortorder on CI - Analytics{,Create,Update}Schema.type now includes a `QuestionType` enum array (the analytics MV column is question_type on CI's fresh build). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066a: backport CI fixups (data-broker stub, EventBuilder fixture, types regen) Same drifts surfaced on this PR as on TUP-3066b; smaller subset because this branch only renames the column, doesn't drop the legacy tables: - data-broker DhisService stubs: swap entityHierarchy → project to match EventsPuller.ts's new lookup path - central-server EventBuilder.test.js: ancestor_descendant_relation now keyed by project_id (the column was renamed) - @tupaia/types models.ts + schemas.ts: - Analytics{,Create,Update}.type now QuestionType (analytics MV column is question_type on CI's fresh build) - EntityType enum order: document_group before document, tamanu_country between ird_village and srh_district (matches pg_enum.enumsortorder) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Update index.js * specs * fix: TUP-3066b: restore review-fix changes lost in merge conflict Merging epic-entity-hierarchy into tup-3066b-drop-legacy-tables dropped five fixes that were correctly squash-merged via PRs #6778 and #6786. Restored: - surveyDataExport.fetchProjectId: return project.id (not entity_hierarchy_id) - migration: delete orphan ancestor_descendant_relation rows before SET NOT NULL - Entity.fetchDefaultProjectIdPatiently: dedupe project_id in SQL via DISTINCT - Entity.findOneByCodeInProject: forward 4th options arg (was silently dropped) - data-broker models: extract named Project type to match file convention Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066b: restore central-server project changes lost in merge The recent merge of epic-entity-hierarchy reintroduced entity_hierarchy references in CreateProject.js and the project test rollback helpers. This was the same merge-conflict-lost-work pattern as commit ae6a9d8; restoring from 55ddae5 (last known-good CI state): - CreateProject.js: drop createEntityHierarchy method, its call site, and the entity_hierarchy_id field on project create - CreateProject.test.js: drop "creates a valid entity hierarchy record" test and entityHierarchy.delete from rollbackRecords - EditProject.test.js: drop entityHierarchy.delete from rollbackRecords Fixes the 5 failing central-server tests on CI — the after-each hook crash on the missing entityHierarchy model was cascading sinon stub leakage into the next two test files. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Delete TUP-3066-refinement.md * Update entity-hierarchy-improvements.md * refactor: TUP-3156: delete MS1 sync Andrew confirmed in the TUP-3156 refinement that MS1 is no longer used: last response in 2023, prior in 2021. The sync is bare entity-code lookups outside any request context — post-3056 these silently return arbitrary copies of duplicated entities. Cleaner to remove than to project-scope a dead path. Deletes: - packages/central-server/src/ms1/ directory (sync code) - Ms1SyncLog + Ms1SyncQueue model wrappers (only callers were in /ms1/) - startSyncWithMs1 call in index.js Out of scope: - DB tables ms1_sync_queue / ms1_sync_log stay (separate cleanup) - types/EntityMetadata.ms1 field stays (no readers/writers, harmless) - clearTestData entries stay (tables still exist, no-op cleanup is fine) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: TUP-3156: project-scope KoBo entity lookups + data-broker context Sub-country entity codes are duplicated per project after TUP-3056, so bare findOne({ code }) in the KoBo sync path silently returned an arbitrary copy. KoBo always syncs one project at a time, so we can thread the survey's project_id through. - central-server/kobo/startSyncWithKoBo.js: fetch the survey from the sync group's data_group_code, pass survey.project_id to dataBroker pullSyncGroupResults, and use entity.findOneByCodeInProject in writeKoboDataToTupaia. - data-broker KoBoService: accept projectId in pullSyncGroupResults options, forward to translator. - data-broker KoBoTranslator: accept projectId, use findOneByCodeInProject when present (falls back to bare findOne for safety). One cast at the call site because findOneByCodeInProject inherits its return type from BaseEntityModel and loses data-broker's Entity fields. - DataServiceResolver.getMappingByOrgUnitCode: left as bare findOne, but documented why it's safe — only orgUnit.country_code is consumed downstream, and that's identical across all duplicated copies of a sub-country entity. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: TUP-3156: project-scope web-config-server apiV1 entity lookups The legacy apiV1 RouteHandler base and its data-aggregation subclass both did bare entity.findOne({ code }), which after TUP-3056 silently returns an arbitrary copy of a duplicated-per-project sub-country entity. Same pattern was repeated in seven downstream helpers. Fixed where project context is available: - RouteHandler.handleRequest: fetch project before entity, scope entity lookup. Reorders existing work — project was already needed later for permission checks. - DataAggregatingRouteHandler.fetchAndFilterDataSourceEntitiesOfType: uses this.project (populated by RouteHandler). - measureData.js: uses this.project (subclass of DataAggregatingRouteHandler). - dataBuilders/helpers/groupEvents.js: projectId already destructured from params. - dataBuilders/helpers/calculateOperationForAnalytics.js: projectId already in config. Documented as safe (bare lookup OK because only stable-across-duplicates fields are consumed): - measureBuilders/helpers.js mapMeasureDataToCountries: consumes only country_code. - dataBuilders/helpers/mapAnalyticsToCountries.js: consumes only country_code. - fetchIndicatorValues/fetchAggregatedAnalyticsByDhisIds.js: consumes only .code (= input). Skipped (no project context plumbed to that call path): - composePercentagesPerPeriodByOrgUnit.js Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: TUP-3156: temp project-scope fallback in entity import + spec rollup Three import callsites flagged in the audit are blocked on TUP-3054 plumbing projectCode through admin-panel import requests. This PR wires the plumbing now as a no-op: - importEntities POST endpoint reads req.query.projectCode (currently always unset), resolves to projectId (currently null), passes through importEntity → updateCountryEntities → getOrCreateParentEntity → getEntityMetadata. - Leaf helpers use entity.findOneByCodeInProject(code, projectId), which falls back to bare findOne when projectId is null — same behaviour as before. When TUP-3054 lands and starts passing projectCode in the request, these flows auto-scope without another round of changes. importUserPermissions.js: the validator restricts entity_code to type='country' and country codes aren't duplicated, so the bare lookup is safe. Documented inline. Also rolls up the cumulative spec doc update for the TUP-3156 stack: mark MS1 deleted, KoBo done, Superset audited, web-config-server done, import flows fallback-ready; DHIS2 and the three "Tom/Juliana to follow up" integrations marked blocked. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * clean * refactor: TUP-3156: apply PR #6791 review feedback - RouteHandler: extract `get projectId()` getter to DRY `this.project?.id ?? null` across RouteHandler, DataAggregatingRouteHandler, measureData - RouteHandler: document the null-projectId fallback as a deliberate TUP-3054 interim — every route needs projectCode before we can remove it - groupEvents: null-guard parentOrgUnit before .getDescendantsOfType (the project-scoped lookup is narrower and can now return null where the bare findOne wouldn't have) - fetchAggregatedAnalyticsByDhisIds: upgrade from comment-only to actual scoping — projectId was already in the function signature Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: TUP-3156: apply PR #6792 review feedback - updateCountryEntities: thread projectId into the per-entity-loop getEntityMetadata call. The country-level call was already scoped, but the per-entity lookup (the one that actually matters for duplicated sub-country codes) was missing it — would silently remain unscoped once TUP-3054 starts plumbing projectCode through. - importEntities: throw ValidationError when projectCode is supplied but no matching project exists. Prevents a stale/typo'd projectCode from silently falling through to bare lookups once TUP-3054 ships. Composite (code, project_id) index is already in place via the UNIQUE constraint added in 20260501000000-addProjectIdToEntity — no migration needed for the performance concern raised in review. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * comments --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…t_id (#6793) * feat(database): TUP-3068: simplify ancestor_descendant_relation rebuild Splits out of #6761. Second of three stacked PRs — stacks on #6776 (TUP-3060 foundation: project_country, ProjectCountry model, findOneByCodeInProject). The `ancestor_descendant_relation` closure cache is kept as the read source for hierarchy walks. What changes is how it's rebuilt — TUP-3068's brief was to simplify the rebuild algorithm now that hierarchy edges live on `entity.parent_id` + `project_country`. The legacy 3-class pipeline (EntityHierarchyCacher → EntityHierarchySubtreeRebuilder → EntityParentChildRelationBuilder) with conditional `entity_relation` vs `entity.parent_id` branching at each hierarchy level collapses into a single recursive CTE per project. ## New: AncestorDescendantCacheBuilder `AncestorDescendantCacheBuilder.rebuildForProject(projectId)` is the whole rebuild: - One recursive CTE walking `entity.parent_id` (sub-country edges) ∪ `project_country` (project ↔ country bridge), scoped to one project's hierarchy. - Wipe-and-rebuild per-project — DELETE for the hierarchy + INSERT ... SELECT from the CTE, inside one transaction. - Filters `child.type NOT IN ('project', 'country')` from the parent_id leg: project.parent_id and country.parent_id both point at the world entity, which is meta in the project-hierarchy model. Without this filter, world surfaces as `parent_code` of every project/country and the frontend walks up into 403s. ## Rewritten: EntityHierarchyCacher Change-handler translators now key on `project_id` (was: `hierarchyId + rootEntityId` pairs). Listens to `entity` / `projectCountry` / `entityHierarchy` changes; each translates to "rebuild these project_ids' caches". ## Deleted - `EntityHierarchySubtreeRebuilder.js` — its job is now the recursive CTE. - `EntityParentChildRelationBuilder.js` — its `entity_relation` / parent_id branching is gone. `entity_parent_child_relation` is no longer maintained (drop is TUP-3066). - Two old test fixture/test files for the deleted classes. ## Redirected - `buildEntityParentChildRelationIfEmpty` — function name preserved for the central-server bootstrap call site, but now checks `ancestor_descendant_relation` (the surviving cache) and invokes `AncestorDescendantCacheBuilder.rebuildAll`. ## Migration TRUNCATE The TUP-3068 data migration (`20260507000001-backfillProjectCountry`) ends with `TRUNCATE ancestor_descendant_relation`. This invalidates stale rows from the pre-3068 cacher (which reference entity ids that RN-1853 duplicated/replaced) so the bootstrap rebuild on next central-server boot sees an empty cache and runs `rebuildAll` against current state. Self-healing for clean prod deploys. The cleanup of `20201006211507-BuildAncestorDescendantRelationCache-modifies-data.js` reduces it to a no-op shell so db-migrate's directory scan still loads cleanly once the old builder it imported is gone. The `repointSurveyResponses` `ANALYZE` step added to the RN-1853 migration is an adjacent perf fix: post-entity-backfill stats are stale, the planner picks a bad plan for the four-table-join repoint UPDATEs (observed 15 min on 400k responses on tom-db; ANALYZE expected to drop runtime to a fraction). ## Entity.js rewrite `getEntitiesFromParentChildRelation` (and its `getAncestorsFromParentChildRelation` / `getDescendantsFromParentChildRelation` wrappers) used to read from `entity_parent_child_relation`. That table is no longer maintained — rewrite walks `entity.parent_id` directly with a recursive CTE, scoped by project via `(project_id IS NULL OR project_id = ?)`. Also drops `getChildrenViaHierarchy` — no live callers, the only reader was the legacy `entity_relation` table lookup. ## Test fixture rewrite (bundled here, not PR1) `buildAndInsertProjectsAndHierarchies` rewritten to write `entity.parent_id` directly + `project_country` for project ↔ country edges, mirroring RN-1853's per-project entity duplication. This belongs with the cacher rewrite (PR2) because the new cacher walks parent_id + project_country and would otherwise find no edges in tests. `clearTestData` orders `project_country` before `entity` (FK is `ON DELETE RESTRICT`). `CentralSyncManager.syncLookup.test.ts` seeds a `project_country` row to exercise the new bridge. ## Tests - New regression suite `Entity/getDescendantsAncestorsProjectScoping.test.js` (9 tests). Triggers `AncestorDescendantCacheBuilder.rebuildForProject` in `beforeEach`, asserts via the public `getDescendants` / `getAncestors` API. - Entity-server test setup explicitly invokes `AncestorDescendantCacheBuilder.rebuildAll()` in `beforeAll` — the change-handler-driven cacher doesn't run in test mode. - Datatrak-web `useSubmitSurveyResponse` — removed the offline shim that manually wrote `entity_parent_child_relation` rows when datatrak inserted new entities. With the cacher rebuilt from `entity.parent_id` directly, no shim needed. Stacked PR plan: 1. #6776 — TUP-3060 foundation (merged) 2. **This PR** — TUP-3068 closure cache rebuild + test fixture rewrite 3. #6778 — TUP-3065 entity_relation consumer retirement * refactor(central-server): TUP-3065: switch consumers from entity_relation to project_country Splits out of #6761. Third of three stacked PRs — stacks on #6777 (TUP-3068 closure cache rebuild). #6776 (TUP-3060 foundation) already merged. Mechanical replacement of `entity_relation`-based project↔country lookups with `project_country` joins, plus retiring the now-orphaned `/entityRelations` admin route and pruning multi-hierarchy test fixtures. ## Consumer switches All four consumer paths joined `entity_relation` to discover a project's countries; they now join `project_country` directly. Country code comes from the country entity itself (`entity.code`), not from `entity.country_code`-on-a-child. - `Project.countries()` — single source of truth - `createDashboardRelationsDBFilter` — dashboards filtered by project access - `assertMapOverlaysPermissions` — map overlay access checks - `GETProjects.permissionsFilteredInternally` — project listing - `hasAccessToEntityForVisualisation` / `hasTupaiaAdminAccessToEntityForVisualisation` ## CreateProject `createProjectCountries` writes `project_country` rows on project creation (was: `entity_relation`). ## Retired - `/v1/entityRelations/:id` admin route + its assertion helper - `DeleteEntity` no longer fires `entityRelation.delete` cascade - Multi-hierarchy `Entity.test.js` cases ## Test fixture switch Central-server test fixtures + the database-level `buildAndInsertProjectsAndHierarchies` helper switched to seed `project_country` only (PR2 had it write both to bridge the consumer switch). ## Out of scope - Dropping the legacy tables — TUP-3066, gated on TUP-3067 (MediTrak compat) Stacked PR plan: 1. #6776 — TUP-3060 foundation (merged) 2. #6777 — TUP-3068 closure cache rebuild 3. **This PR** — TUP-3065 entity_relation consumer retirement * rename * Update EntityHierarchyCacher.js * Update Entity.js * Update AncestorDescendantCacheBuilder.js * Update Entity.js * refactor * plan * refactor: TUP-3066: rename hierarchyId → projectId in entity hierarchy code path Each project has exactly one hierarchy, so the indirection through `entity_hierarchy_id` is dead weight. Renames `ancestor_descendant_relation.entity_hierarchy_id` to `project_id` (schema migration + types regen) and ripples the parameter rename `hierarchyId` → `projectId` across: - `Entity.js` hierarchy walk methods (record + model) - `AncestorDescendantRelation.js` getImmediateRelations + caches - `AncestorDescendantCacheBuilder.js` rebuildForProject - entity-server hierarchy routes + middleware (CommonContext) - datatrak-web entity accessors - web-config-server `fetchHierarchyId` → `fetchProjectId` - `fetchDefaultEntityHierarchyIdPatiently` → `fetchDefaultProjectIdPatiently` (rewritten to query projects via ancestor_descendant_relation) Drops: - `getChildrenViaHierarchy` (all callers retired in PR #6778) - `findOneOrThrow({entity_hierarchy_id})` lookup inside `getEntitiesFromParentChildRelation` — closes review-hero comment #3217296820 on PR #6777 Also restores a typo introduced by 90a0ebb (`getEntitiesFromParentChildRelagetEntitiesFromParentChildRelationtion`). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: TUP-3066b: drop entity_relation, entity_parent_child_relation, entity_hierarchy Subtractive cleanup once TUP-3066a's rename is in: the closure cache is the only hierarchy reader still standing and it's keyed by project_id, so the three legacy tables can go. ⚠ GATED ON TUP-3067 (MediTrak compatibility). The schema migration must not run in production until mobile sync is no longer reading entity_parent_child_relation through the boilerplate sync lookup. Draft now to review the code shape; merge once 3067 ships. Schema migration drops `project.entity_hierarchy_id` then the three tables in FK-dependency order (down migration restores structure + minimal seed; the relation-row data is reproducible from entity.parent_id + project_country). Code: - Delete model classes: EntityRelation, EntityParentChildRelation, EntityHierarchy - Drop ENTITY_RELATION / ENTITY_HIERARCHY / ENTITY_PARENT_CHILD_RELATION records, syncing-tables list, post-migration trigger list, clearTestData list - Rewrite Entity.buildSyncLookupQueryDetails to compute project_ids without entity_parent_child_relation joins - Retire EntityHierarchyCacher.translateEntityHierarchyChange (the table is gone) - Drop CreateProject.createEntityHierarchy + project.entity_hierarchy_id field - Drop DeleteEntity entity_relation walk; now walks per-project via project_country - Drop test fixture's entity_hierarchy creation - Delete admin routes: - central-server: /entityHierarchy GET + PUT routes + tests - entity-server: /hierarchies route + integration tests - Drop project.entity_hierarchy_id projection in GETProjects + Project.find Types are NOT regenerated in this commit because the legacy tables still exist on the dev DB; the regen happens when the schema migration is applied alongside TUP-3067. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Update Entity.test.js * fix: TUP-3066b: drop type references to deleted entity-hierarchy models Followups noticed during local test setup: - tsmodels: delete EntityHierarchy.ts and EntityParentChildRelation.ts stubs + remove their re-exports from models/index.ts - server-boilerplate: drop EntityHierarchyModel/Record re-export - sync-server: drop deleted models from SyncServerModelRegistry + TestSyncServerModelRegistry shapes - datatrak-web: drop entityHierarchy/entityParentChildRelation fields from DatatrakWebModelRegistry Also: in fetchDefaultProjectIdPatiently, sort projects by `code` instead of `name` (project table has no `name` column — that column lived on entity_hierarchy which is gone). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066a: fetchDefaultProjectIdPatiently sort by project.code The project table has no `name` column (that lived on entity_hierarchy). Sort by `code` to mirror the original alphabetical-fallback behaviour. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066b: clear remaining references to dropped legacy tables in tests + types Catches the CI fallout from PR #6787: - Regenerate @tupaia/types (drops EntityHierarchy / EntityRelation / EntityParentChildRelation from models.ts + schemas.ts) - entity-server src/types.ts + __tests__/types.ts: drop EntityHierarchyModel / EntityRelationModel from registries - data-broker DhisService stubs: replace ENTITY_HIERARCHIES fixture with PROJECTS stub so EventsPuller's models.project.findOne resolves - Database tests: - Entity/getDescendantsAncestorsProjectScoping.test.js: stop creating entity_hierarchy + entity_hierarchy_id on project - Central-server tests: - apiV2/landingPages/utils.js: drop entity_hierarchy create - apiV2/projects/CreateProject.test.js: drop the "creates a valid entity hierarchy record" assertion + entityHierarchy cleanup - apiV2/projects/EditProject.test.js, GETProjects.test.js: same cleanup - apiV2/surveys/CreateAndEditSurveys.test.js: project fixture without entity_hierarchy_id - dhis/EventBuilder.test.js: ancestor_descendant_relation now keyed by project_id; create project instead of entity_hierarchy - Sync-server tests: - CentralSyncManager.pull.test.ts: drop entityHierarchy from prepareData - CentralSyncManager.syncLookup.test.ts: drop entity_parent_child_relation insert; entity2's parent linked via entity.parent_id instead Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066b: include analytics types + clear leftover test references CI surfaced three remaining issues after the first pass: - @tupaia/types Validate: regenerate against a DB that has the analytics materialized view, so `Analytics` / `AnalyticsCreate` / `AnalyticsUpdate` interfaces are present alongside the dropped legacy table cleanup - entity-server permissions.test.ts: remove the "filters hierarchies when requested for some with access" case (route `/v1/hierarchies` is gone) and the unused `getHierarchiesWithFields` import - sync-server: - CentralSyncManager.pull.test.ts: snapshotRecords expectation now 3 (country + userAccount + project) — entityHierarchy isn't seeded anymore - CentralSyncManager.syncLookup.test.ts: drop the now-unused `entity2` let-binding (the entity is still inserted, just not held for later use) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066b: match types regen against CI (analytics.type enum, entity_type order) CI generates types against a freshly-built DB; my local dev DB has older state and produced two drifts: - Analytics{,Create,Update}.type: my dev DB has the analytics MV with column type 'text'. CI's fresh MV has 'question_type', so the generated TS type is QuestionType. Match CI by changing 'string | null' → 'QuestionType | null' in the three Analytics interfaces. - EntityType enum order: pg_enum.enumsortorder differs between my dev DB and CI. Reorder document_group/document and tamanu_country to match CI's insertion order. schemas.ts uses alphabetical ordering and didn't need changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066b: match types schemas.ts against CI regen Three more drifts caught by the Validate types CI step: - ENTITY_TYPE_CHILDREN map (used by UserAccountPreferencesSchema): document_group before document, and tamanu_country between ird_village and srh_district — matches Postgres pg_enum.enumsortorder on CI - Analytics{,Create,Update}Schema.type now includes a `QuestionType` enum array (the analytics MV column is question_type on CI's fresh build). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066a: backport CI fixups (data-broker stub, EventBuilder fixture, types regen) Same drifts surfaced on this PR as on TUP-3066b; smaller subset because this branch only renames the column, doesn't drop the legacy tables: - data-broker DhisService stubs: swap entityHierarchy → project to match EventsPuller.ts's new lookup path - central-server EventBuilder.test.js: ancestor_descendant_relation now keyed by project_id (the column was renamed) - @tupaia/types models.ts + schemas.ts: - Analytics{,Create,Update}.type now QuestionType (analytics MV column is question_type on CI's fresh build) - EntityType enum order: document_group before document, tamanu_country between ird_village and srh_district (matches pg_enum.enumsortorder) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Update index.js * specs * fix: TUP-3066b: restore review-fix changes lost in merge conflict Merging epic-entity-hierarchy into tup-3066b-drop-legacy-tables dropped five fixes that were correctly squash-merged via PRs #6778 and #6786. Restored: - surveyDataExport.fetchProjectId: return project.id (not entity_hierarchy_id) - migration: delete orphan ancestor_descendant_relation rows before SET NOT NULL - Entity.fetchDefaultProjectIdPatiently: dedupe project_id in SQL via DISTINCT - Entity.findOneByCodeInProject: forward 4th options arg (was silently dropped) - data-broker models: extract named Project type to match file convention Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: TUP-3066b: restore central-server project changes lost in merge The recent merge of epic-entity-hierarchy reintroduced entity_hierarchy references in CreateProject.js and the project test rollback helpers. This was the same merge-conflict-lost-work pattern as commit ae6a9d8; restoring from 55ddae5 (last known-good CI state): - CreateProject.js: drop createEntityHierarchy method, its call site, and the entity_hierarchy_id field on project create - CreateProject.test.js: drop "creates a valid entity hierarchy record" test and entityHierarchy.delete from rollbackRecords - EditProject.test.js: drop entityHierarchy.delete from rollbackRecords Fixes the 5 failing central-server tests on CI — the after-each hook crash on the missing entityHierarchy model was cascading sinon stub leakage into the next two test files. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Delete TUP-3066-refinement.md * Update entity-hierarchy-improvements.md * refactor: TUP-3156: delete MS1 sync Andrew confirmed in the TUP-3156 refinement that MS1 is no longer used: last response in 2023, prior in 2021. The sync is bare entity-code lookups outside any request context — post-3056 these silently return arbitrary copies of duplicated entities. Cleaner to remove than to project-scope a dead path. Deletes: - packages/central-server/src/ms1/ directory (sync code) - Ms1SyncLog + Ms1SyncQueue model wrappers (only callers were in /ms1/) - startSyncWithMs1 call in index.js Out of scope: - DB tables ms1_sync_queue / ms1_sync_log stay (separate cleanup) - types/EntityMetadata.ms1 field stays (no readers/writers, harmless) - clearTestData entries stay (tables still exist, no-op cleanup is fine) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: TUP-3156: project-scope KoBo entity lookups + data-broker context Sub-country entity codes are duplicated per project after TUP-3056, so bare findOne({ code }) in the KoBo sync path silently returned an arbitrary copy. KoBo always syncs one project at a time, so we can thread the survey's project_id through. - central-server/kobo/startSyncWithKoBo.js: fetch the survey from the sync group's data_group_code, pass survey.project_id to dataBroker pullSyncGroupResults, and use entity.findOneByCodeInProject in writeKoboDataToTupaia. - data-broker KoBoService: accept projectId in pullSyncGroupResults options, forward to translator. - data-broker KoBoTranslator: accept projectId, use findOneByCodeInProject when present (falls back to bare findOne for safety). One cast at the call site because findOneByCodeInProject inherits its return type from BaseEntityModel and loses data-broker's Entity fields. - DataServiceResolver.getMappingByOrgUnitCode: left as bare findOne, but documented why it's safe — only orgUnit.country_code is consumed downstream, and that's identical across all duplicated copies of a sub-country entity. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: TUP-3156: project-scope web-config-server apiV1 entity lookups The legacy apiV1 RouteHandler base and its data-aggregation subclass both did bare entity.findOne({ code }), which after TUP-3056 silently returns an arbitrary copy of a duplicated-per-project sub-country entity. Same pattern was repeated in seven downstream helpers. Fixed where project context is available: - RouteHandler.handleRequest: fetch project before entity, scope entity lookup. Reorders existing work — project was already needed later for permission checks. - DataAggregatingRouteHandler.fetchAndFilterDataSourceEntitiesOfType: uses this.project (populated by RouteHandler). - measureData.js: uses this.project (subclass of DataAggregatingRouteHandler). - dataBuilders/helpers/groupEvents.js: projectId already destructured from params. - dataBuilders/helpers/calculateOperationForAnalytics.js: projectId already in config. Documented as safe (bare lookup OK because only stable-across-duplicates fields are consumed): - measureBuilders/helpers.js mapMeasureDataToCountries: consumes only country_code. - dataBuilders/helpers/mapAnalyticsToCountries.js: consumes only country_code. - fetchIndicatorValues/fetchAggregatedAnalyticsByDhisIds.js: consumes only .code (= input). Skipped (no project context plumbed to that call path): - composePercentagesPerPeriodByOrgUnit.js Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: TUP-3156: temp project-scope fallback in entity import + spec rollup Three import callsites flagged in the audit are blocked on TUP-3054 plumbing projectCode through admin-panel import requests. This PR wires the plumbing now as a no-op: - importEntities POST endpoint reads req.query.projectCode (currently always unset), resolves to projectId (currently null), passes through importEntity → updateCountryEntities → getOrCreateParentEntity → getEntityMetadata. - Leaf helpers use entity.findOneByCodeInProject(code, projectId), which falls back to bare findOne when projectId is null — same behaviour as before. When TUP-3054 lands and starts passing projectCode in the request, these flows auto-scope without another round of changes. importUserPermissions.js: the validator restricts entity_code to type='country' and country codes aren't duplicated, so the bare lookup is safe. Documented inline. Also rolls up the cumulative spec doc update for the TUP-3156 stack: mark MS1 deleted, KoBo done, Superset audited, web-config-server done, import flows fallback-ready; DHIS2 and the three "Tom/Juliana to follow up" integrations marked blocked. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: TUP-3156: DHIS2 push project scoping + dhis_instance.project_id Two related-but-independent pieces in one PR: Schema (scaffolding): - New migration adds nullable project_id to dhis_instance. The ticket refinement says each DHIS2 instance is assigned to one project; this column lets admins record that association. Future PR can wire it into pusher routing and enforce entity.project_id == instance.project_id. Code (fix audit-flagged bare lookups in the pushers): - OrganisationUnitPusher.updateFacilityTypeGroup: country code lookup, documented safe (country codes not duplicated per project). - OrganisationUnitPusher.updateOrganisationUnitGroupsForFacility: ancestor lookup now scoped by the facility's own entity.project_id. - AggregateDataPusher.fetchOrganisationUnit (delete branch): documented residual risk — dhis_sync_log lacks project_id and the source survey response is gone post-delete. Fix requires schema change to dhis_sync_log (separate ticket). - AggregateDataPusher.addMatchingRecordsToSyncQueue: same residual risk, but only triggers on legacy sync log entries pre-dating entityId. - getEntityIdFromClinicId: now accepts projectId; central-server caller in meditrak sync passes survey.project_id (facility codes are sub-country and duplicated). The meditrak-app-server parallel implementation is gated by TUP-3067 (out of scope here). Spec doc: updated audit table reflecting all of the above. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * clean * refactor: TUP-3156: apply PR #6791 review feedback - RouteHandler: extract `get projectId()` getter to DRY `this.project?.id ?? null` across RouteHandler, DataAggregatingRouteHandler, measureData - RouteHandler: document the null-projectId fallback as a deliberate TUP-3054 interim — every route needs projectCode before we can remove it - groupEvents: null-guard parentOrgUnit before .getDescendantsOfType (the project-scoped lookup is narrower and can now return null where the bare findOne wouldn't have) - fetchAggregatedAnalyticsByDhisIds: upgrade from comment-only to actual scoping — projectId was already in the function signature Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * db: TUP-3156: drop unused dhis_instance.project_id migration The column was added as scaffolding for the original DHIS2-push plan (route entity lookups via dhis_instance.project_id), but the actual fix in this PR sidesteps that by scoping via entity.project_id instead. Nothing reads the column. Drop the speculative schema rather than ship dead plumbing — if the follow-up (admin-panel UI to set it, AggregateDataPusher routing) ever happens it can re-add the migration with concrete requirements. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: TUP-3165: rewrite getParentFieldsByChildId against closure cache The method's SQL still queried entity_parent_child_relation — dropped by TUP-3066b (PR #6787). Surfaced as a latent runtime bug during smoke testing prep: the build passed after the FormatContext hierarchyId → projectId rename, but the underlying SQL would throw against the missing table when datatrak-web's formatEntity batched parent_name / parent_code lookups. Rewrite uses ancestor_descendant_relation (the closure cache, project-scoped via project_id, with generational_distance = 1 for immediate parents). Matches the pattern entity-server's format.ts uses via ancestorDescendantRelation.getChildCodeToParentCode(projectId). Param renamed hierarchyId → projectId in the JS impl and tsmodels declaration (the underlying TUP-3066a rename was missed on this signature). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * buildAncestorDescendantRelationIfEmpty --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: TUP-3165: rewrite getParentFieldsByChildId against closure cache The method's SQL still queried entity_parent_child_relation — dropped by TUP-3066b (PR #6787). Surfaced as a latent runtime bug during smoke testing prep: the build passed after the FormatContext hierarchyId → projectId rename, but the underlying SQL would throw against the missing table when datatrak-web's formatEntity batched parent_name / parent_code lookups. Rewrite uses ancestor_descendant_relation (the closure cache, project-scoped via project_id, with generational_distance = 1 for immediate parents). Matches the pattern entity-server's format.ts uses via ancestorDescendantRelation.getChildCodeToParentCode(projectId). Param renamed hierarchyId → projectId in the JS impl and tsmodels declaration (the underlying TUP-3066a rename was missed on this signature). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * buildAncestorDescendantRelationIfEmpty * fix(db): TUP-3165: backfill missing ancestor copies in per-project entity duplication Smoke testing revealed ~26,500 entities ended up with cross-project parent_id references after the RN-1853 migration, breaking per-project closure cache walks. Worst affected: ~25.6k explore entities (schools, villages), 222 unfpa facilities (Fiji), 339 covid_samoa villages. Root cause: the entity-to-project mapping was sourced only from direct entity_parent_child_relation membership. Some pre-migration hierarchies shortcut levels (e.g. country → facility directly, skipping sub_district), so when a project's hierarchy referenced a facility but not its sub_district ancestor, the ancestor wasn't duplicated into that project. fixParentIdChains then couldn't re-point the facility's parent_id to a same-project sub_district copy because none existed, leaving a cross-project edge that breaks the project-scoped recursive CTE walk in the closure cache builder. Fix: build a temp table `entity_in_project` that starts from EPCR membership then recursively expands by walking entity.parent_id upward, adding every sub-country ancestor to the same project's set. Drive single-project assignment and multi-project duplication off this expanded set so every required ancestor copy exists before fixParentIdChains runs. Added assertNoCrossProjectParents as a post-fix sanity check — fail loudly if any cross-project parent_id remains, rather than producing silently broken closure caches. No changes to the down path: the existing flag-based teardown reverses all duplicates regardless of how many were created. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * tweaks --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…-devtools (#6800) - Use react-router-dom imports consistently (was importing from react-router in three files) - Remove unused @tanstack/react-query-devtools dependency - Minor UserProfileInfo styling tweaks Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(admin-panel): tidy react-router imports and drop react-query-devtools - Use react-router-dom imports consistently (was importing from react-router in three files) - Remove unused @tanstack/react-query-devtools dependency - Minor UserProfileInfo styling tweaks Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(admin-panel-server): RN-1855: scope admin requests by project Adds applyProjectScope middleware that intercepts admin-panel-server requests and applies a project filter to scoped resources (surveys, entities, surveyResponses) based on a ?projectCode= query param. - Forward unchanged when projectCode absent (all-data view). - Resolve projectCode to a project row and apply rule.filter to GETs. - For PUT/DELETE on a resource id, verify the record belongs to the active project. - For POST, verify body references the active project. GETSurveyResponses gains a survey join so the surveyResponse scope filter can reach survey.project_id. Express Request typings are extended with the project, survey and surveyResponse models the middleware reads from. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(admin-panel-server): RN-1855: scope admin requests by project Adds applyProjectScope middleware that intercepts admin-panel-server requests and applies a project filter to scoped resources (surveys, entities, surveyResponses) based on a ?projectCode= query param. - Forward unchanged when projectCode absent (all-data view). - Resolve projectCode to a project row and apply rule.filter to GETs. - For PUT/DELETE on a resource id, verify the record belongs to the active project. - For POST, verify body references the active project. GETSurveyResponses gains a survey join so the surveyResponse scope filter can reach survey.project_id. Express Request typings are extended with the project, survey and surveyResponse models the middleware reads from. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
#6803) refactor(admin-panel): deduplicate QueryClientProvider in StoreProvider main.jsx already wraps the app in QueryClientProvider; StoreProvider was creating a second QueryClient and provider underneath. Drop the inner wrapper (and the unused ReactQueryDevtools import that came with it — the devtools package itself was removed in #6800). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(admin-panel): tidy react-router imports and drop react-query-devtools - Use react-router-dom imports consistently (was importing from react-router in three files) - Remove unused @tanstack/react-query-devtools dependency - Minor UserProfileInfo styling tweaks Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(admin-panel-server): RN-1855: scope admin requests by project Adds applyProjectScope middleware that intercepts admin-panel-server requests and applies a project filter to scoped resources (surveys, entities, surveyResponses) based on a ?projectCode= query param. - Forward unchanged when projectCode absent (all-data view). - Resolve projectCode to a project row and apply rule.filter to GETs. - For PUT/DELETE on a resource id, verify the record belongs to the active project. - For POST, verify body references the active project. GETSurveyResponses gains a survey join so the surveyResponse scope filter can reach survey.project_id. Express Request typings are extended with the project, survey and surveyResponse models the middleware reads from. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(admin-panel-server): RN-1855: scope admin requests by project Adds applyProjectScope middleware that intercepts admin-panel-server requests and applies a project filter to scoped resources (surveys, entities, surveyResponses) based on a ?projectCode= query param. - Forward unchanged when projectCode absent (all-data view). - Resolve projectCode to a project row and apply rule.filter to GETs. - For PUT/DELETE on a resource id, verify the record belongs to the active project. - For POST, verify body references the active project. GETSurveyResponses gains a survey join so the surveyResponse scope filter can reach survey.project_id. Express Request typings are extended with the project, survey and surveyResponse models the middleware reads from. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(admin-panel): RN-1855: project filter via URL Adds a global project selector to the admin panel. The selected project lives in the URL (/:projectCode/...) — no Redux, no localStorage. The axios interceptor parses window.location.pathname and forwards ?projectCode= to the admin-panel-server so the applyProjectScope middleware can filter scoped resources. - New projects/ module with useSelectedProject/useSelectedProjectCode hooks and readSelectedProjectCode (sync, for interceptors). - ProjectSelector dropdown wired to react-router navigate. - DefaultRedirect lands on the first project's first single-project tab. - App.jsx splits routes into single-project (under :projectCode) and all-data (top-level) scopes via routes/scopes.js. - ResourcePage rekeys DataFetchingTable on project change so the Redux-driven table refetches. - useItemDetails includes projectCode in its query key for project-scoped cache separation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(admin-panel): RN-1855: address review feedback on project filter - getFlattenedChildViews: restore basePath as second positional arg (new pathPrefix moves to third) so existing callers like lesmis's AdminPanelApp.jsx that pass `getFlattenedChildViews(route, adminUrl)` continue to get adminUrl as basePath. Updates the single internal caller that wanted pathPrefix to pass an explicit empty basePath. - DefaultRedirect: wait for useProjects to resolve before deciding the redirect target. The previous early all-data fallback caused the component to unmount before single-project landing could fire on a cold load, so the "land on first project" intent was dead code. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(admin-panel): RN-1855: prefer user's saved project on landing DefaultRedirect now reads user_account.preferences.project_id from the /me response and lands on that project when it exists in the accessible project list. Falls back to the first project alphabetically (useProjects already sorts by entity.name ASC) when no preference is set or the saved project isn't accessible anymore. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(admin-panel-server): RN-1855: restore originalUrl mutation for proxy http-proxy-middleware prefers req.originalUrl over req.url (see source, http-proxy-middleware.js lines 76 + 90 — uses originalUrl first and resets req.url = originalUrl before forwarding). The previous review-suggested change to only mutate req.url meant the merged project filter and stripped projectCode param never reached central-server: requests to e.g. /entities?projectCode=X were forwarded verbatim, and central just returned unscoped data. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * cleanup * fix(adminPanel): RN-1855: populate sidebar on All Data routes via preferences fallback (#6806) * fix(admin-panel): RN-1855: populate sidebar on All Data routes via preferences fallback Single-Project nav was empty on All Data routes (no project in URL → useSelectedProjectCode returns null → navItems.single = []). Fall back to user_account.preferences.project_id, then first project alphabetically, so the sidebar stays usable. Request scoping is unchanged — the axios interceptor still reads the URL only, so All Data routes remain unscoped. ProjectSelector saves the chosen project to preferences on change. On single-project URLs it still navigates; on All Data URLs it stays put (sidebar updates from the new preference). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(admin-panel): RN-1855: dedup preferred-project resolution in DefaultRedirect DefaultRedirect was duplicating the "preferences → first alphabetical" fallback that useSidebarProjectCode now owns. Reuse the hook so the chain lives in one place. URL check inside useSidebarProjectCode is a no-op on `/` so behaviour is unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
tcaiger
force-pushed
the
epic-entity-hierarchy
branch
from
4420a96 to
f777250
Compare
…/export bug fixes (#6813) * fix(entityHierarchy): TUP-3181: parse entity-import attributes as JSON The exporter writes the `attributes` / `data_service_entity` columns with JSON.stringify, but the importer parsed them with convertCellToJson (the survey-importer's key:value-per-line helper), so a round-trip corrupted the data — `{"andrewtest":"true"}` became `{'{"andrewtest"':'"true"}'}`. Parse these cells with JSON.parse to match the exporter, rejecting non-object/invalid JSON with a clear error. Adds unit tests for the round-trip. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(entityHierarchy): TUP-3180: filter Projects tab to the selected project The admin-panel-server project-scope middleware had no rule for the `projects` resource, so the single-project Projects tab listed every project instead of the one editable row for the active project. Add a `projects` rule: filter the list to the selected project and restrict edits/deletes to its row, while still allowing project creation (a global action) by passing the active project into bodyOwnership. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(entityHierarchy): TUP-3180: scope edit/return navigation to the active project Scoped resources (Surveys, etc.) render under a /:projectCode prefix, but several links were built without it, so they didn't match a route and fell through to the all-data default tab (Data Elements): - EditButton prepends the active projectCode to edit links (fixes "edit survey" landing on Data Elements, for all scoped resources). - EditSurveyPage returns to the scoped surveys list. - DataTrak's resubmit-success "Return to admin panel" uses the survey's project to return to the scoped survey-responses tab. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(entityHierarchy): TUP-3181: drop deprecated facility_type/type_name/category_code from entity import/export These columns only wrote to / read from the legacy `clinic` table, which is deprecated (visuals now read facility_type from entity.attributes). Per Juliana's confirmation, remove them end to end: - exporter no longer emits the 3 columns or joins `clinic`. - importer no longer validates facility_type or upserts a clinic row (removed attemptFacilityUpsert + default-type machinery). Updates the export-header and facility import tests accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(central-server): avoid 'it only' false-positive in exclusive-test check The validateTests CI check greps for 'it.only' where '.' is a regex wildcard, so a comment containing 'it only' matched and failed the build. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(entityHierarchy): TUP-3180: keep projectCode out of the item-details filter The scoped edit route (/:projectCode/surveys/:id/edit) exposes projectCode as a route param, and useItemDetails serialised every param into the server filter — so the survey detail fetch became `where survey.projectCode = ...` and 500'd ("column survey.projectCode does not exist"). Strip projectCode (it's the scope, not a record column) before building the filter. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…sions + latitude-first (#6822) * fix(entityHierarchy): TUP-3181: scope entity export to user's country permissions + latitude-first columns - Export now matches baseline permissions: any user with Tupaia Admin Panel access can export (no BES Admin required), scoped to the countries they have that access for (getAdminPanelAllowedCountryCodes → country_code = ANY(...)). - Export columns ordered latitude before longitude (conventional). Importer reads by header name, so round-trip is unaffected. Tests updated for the new permission model (non-BES-Admin allowed + scoped; no-admin-panel rejected). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(entityHierarchy): TUP-3181: include the project's country entities in export - Export now UNIONs the project's shared country entities (via project_country) so a multi-country project's export is complete (#2). Importer skips type=country rows so re-import doesn't create project-scoped country copies. - Fixes a param-binding bug introduced in the previous commit: the country-scope `= ANY(?)` placeholder had no matching param. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(entityHierarchy): TUP-3181: scope entity export for BES admins by the project's countries getAdminPanelAllowedCountryCodes throws when a user has no explicit Tupaia Admin Panel entries, which is the case for BES admins (superusers) — so the export 500'd for them. Resolve the project before checking permissions (so an unknown project still 400s) and, for BES admins, scope to every country in the project rather than the empty admin-panel set. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(entityHierarchy): TUP-3181: address export review feedback - Call req.assertPermissions so the permission check is flagged for the ensurePermissionCheck sentinel (the download response would otherwise bypass res.send); unauthorised users now get a PermissionsError rather than a raw throw. Assert before the project lookup so auth precedes existence. - Drop the redundant fetchCountryParentLookup round-trip: the UNION already returns the project's country entities, so the parent_id→code map built from the result set covers every parent a sub-national row can reference. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…red country entities (#6823) * fix(entityHierarchy): TUP-3180: show the project's country entities in the scoped Entities tab The scoped Entities list filtered on project_id alone, which hid the shared country entities (project_id IS NULL) the project spans. Match those by code via the project's project_country set as well, while keeping world/project entities hidden. The condition is wrapped in _and_ so merging with a caller's filter ANDs it in as a hard scope boundary rather than letting the OR widen the result set. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(entityHierarchy): TUP-3181: create the country entity + project link when adding a country Adding a country from the Countries tab now creates the matching shared country entity (type country, parented to World, project_id NULL) and — when done from within a project scope — a project_country row linking the new country to the active project, so it's immediately usable for that project's entity import/export and visualisation. The admin-panel already appends the selected projectCode to every request; the project-scope middleware now forwards it for the countries endpoint instead of stripping it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(entityHierarchy): TUP-3180: drop redundant project re-fetch in the entities scope filter The middleware already holds the full project record, so pass it to the rule's filter instead of re-fetching it via models.project.findById on every entities request. This also lets the filter interface drop the models parameter it only added for that lookup. Adds an integration test asserting the scope filter resolves to (project_id = X OR (type='country' AND code IN ...)) — the project's own entities plus its country entities, excluding other projects. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(entityHierarchy): TUP-3181: harden country creation - Throw a clear error if the World entity is missing rather than dereferencing null into an opaque 500. - When a projectCode is explicitly supplied but no such project exists, fail (rolling back) instead of silently creating an unlinked country. - Drop the redundant duplicate create-data object in the projectCountry findOrCreate (the search criteria are merged into the created row). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(entityHierarchy): TUP-3180: make the scoped Entities tab query work The admin-panel Entities tab 500'd ("column entity._and_ does not exist") once the projectScope filter injected a conjunction: - processColumnSelectorKeys qualified every filter key, turning _and_/_or_ into entity._and_ so addWhereClause never saw the conjunction. Keep conjunction keys intact and recurse into them to qualify the nested column selectors. - GETEntities joined related tables (e.g. project for a project.code column) with an inner join, dropping shared entities whose project_id is null (countries, world). Use a LEFT join like the other list handlers so the project's country entities survive the join. Adds an endpoint-level test exercising the real query path (column join + conjunction filter), which a model-level find() doesn't cover. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * tweak(admin-panel): remove the unused Canonical types field from the Add project modal CreateProject destructures entityTypes but never uses it, and its validator is optional — the field had no effect, so drop it from the create form. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(admin-panel): drop the EntityHierarchiesPage barrel export Completes the dead Entity Hierarchies page removal — the page file is gone, so the re-export would be a broken import. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…m country creation (#6824) The Countries tab is an all-projects view, so no projectCode reaches the endpoint — the project_country auto-link never fired. Remove it (and the countries projectCode passthrough). Creating a country now just creates the country row and its shared country entity; associating a country with a project is handled separately via project_country. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e edit modal (#6825) * feat(entityHierarchy): TUP-3181: manage a project's countries from the edit modal Adding a country to an existing project had no admin-panel path — project_country was only written at project creation. Now the project edit modal exposes a Countries checkbox list (mirroring the Add project modal): - GETProjects attaches each project's countries as an array of country table ids (for the checkbox pre-fill) plus countryCodes (for the list display), mapping project_country's country entity ids back through the shared entity code. - EditProject reconciles project_country against the submitted selection — adding links for newly selected countries, removing deselected ones — inside a transaction with the record update. - Extracts the country-id → country-entity-id helper shared by create and edit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(entityHierarchy): TUP-3181: stop selecting the virtual countryCodes column on the projects list countries/countryCodes are attached post-query by attachCountries, not real project columns, so requesting them as columns made the list query select project.countryCodes and 500. Strip them from the SQL select in getProcessedColumns (mirroring GETEntities' parent_code handling), and mark the list column non-filterable / non-sortable since it can't be queried at SQL level. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(entityHierarchy): TUP-3181: pre-fill the project edit countries picker The editor's processRecordData keeps only each field's `source`, so with source=countryCodes the `countries` (id) array the checkbox list pre-fills from was dropped and nothing showed selected. Make `source` the countries id array (so it survives and pre-fills/submits), and render the codes in the list cell from the record's countryCodes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(entityHierarchy): TUP-3181: attach countries on the single-project fetch by id The edit modal fetches GET /projects/:id, whose columns don't include id, so attachCountries (keyed on record.id) early-returned and the response carried no countries — leaving the edit picker unpopulated. Key the single-record lookup off the known projectId instead, via a shared fetchCountriesByProject helper. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…_relation, entity_hierarchy (#6787)
Importing a row without a parent_code (and no district/sub_district) created an orphan or, on update, wiped the existing entity's parent. Reject such rows with a clear row-level error instead, matching baseline. Country rows are skipped before this check, so it only applies to sub-country entities (which always have a parent). Updates the existing facility/country-skip tests to carry a parent_code and adds a rejection test.
…ines (#6827) * fix(entityHierarchy): TUP-3181: round-trip attributes as key: value lines Issue 8 was fixed by aligning import + export on JSON, but that diverged from the documented reference-data format. Since attributes and data_service_entity are flat scalar objects, keep the human-friendly format instead: export writes newline-separated `key: value` lines and the importer parses them back. `true`/ `false` round-trip to booleans; everything else stays a string so ids (e.g. a kobo_id) aren't coerced to numbers. Empty objects export as a blank cell. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(entityHierarchy): TUP-3181: parse attribute cells with convertCellToJson (match dev) Reuse the same convertCellToJson helper the rest of the importer (and dev) use for newline-separated key: value cells, instead of a bespoke parser, so entity attribute import is byte-for-byte consistent with the reference-data format. All values import as strings (no boolean coercion). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(entityHierarchy): TUP-3181: update extract tests for the key: value format The extractEntitiesFromUpload tests still asserted the JSON parser; update them to the convertCellToJson key: value format (string values, no throw on non-JSON) so they match the round-trip behaviour. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…n code Match the baseline hierarchy export's ordering: countries first, then each level down the tree (generational distance from the country), with an alphabetical-by- code secondary sort within a level. Depth is computed by walking the parent_id chain within the result set (which always contains every ancestor up to the country); the SQL ORDER BY is dropped since the JS ordering is authoritative.
…ic merge - exportEntities ordering test: pass explicit relations so the villages sit under their countries (the helper otherwise parents every entity to the project entity, leaving the villages without an in-export parent and collapsing the generation sort). - importEntities Attribute cells test: add parent_code, since requiring a parent on import (merged separately) now rejects parentless rows.
* feat(entity): RN-1851: extract polygons into shared entity_polygon table (#6738)
* first commit
* Update models.ts
* Update schemas.ts
* Update 20260428000000-createEntityPolygonAndMigrateGis-modifies-schema.js
* updates
* fixes
* Update Entity.js
* performance
* tweaks
* types
* tests
* generate ids as default
* remove string literals
* feat(entityHierarchy): RN-1853: Migrate entities to projects (#6749)
* specs
* Update RN-1853-refinement.md
* Update RN-1853-refinement.md
* Update RN-1853-refinement.md
* plan
* first cut
* Update 20260501000000-addProjectIdToEntityAndDuplicateSharedEntities-modifies-schema.js
* fix tests
* test
* Update schemas.ts
* breakup migrations
* refactor
* Update getDbMigrator.js
* revert
* fixes
* Update 20260501000001-backfillProjectIdsAndDuplicateSharedEntities-modifies-data.js
* tweaks
* feat(database): TUP-3060: project-scoped entity access foundation (#6776)
* feat(database): TUP-3060: project-scoped entity access foundation
Splits out of #6761 (consolidated TUP-3065 branch). First of three stacked PRs.
Adds the foundation for project-scoped entity lookups post-RN-1853, plus the
mechanical fixes to known bare `entity.findOne({ code })` callsites that
silently break when codes are duplicated per project.
## Schema
- New `project_country` table (`20260507000000-addProjectCountryTable`) —
declarative project ↔ country mapping that replaces the implicit
`entity_relation` join. Backfilled (`20260507000001-backfillProjectCountry`)
from existing `entity_relation` rows where parent is a project entity and
child is a country.
- `project_country` registered in `initSyncComponents.js` so the table is part
of the sync surface.
## Models / types plumbing
- New `ProjectCountry` model + record. `id`, `project_id`, `country_id`,
`updated_at_sync_tick`. `UNIQUE (project_id, country_id)`.
- Type plumbing across `@tupaia/types` (schemas + interface), `tsmodels`,
`server-boilerplate` re-export, `entity-server` + `sync-server` test
registry fields.
## findOneByCodeInProject
- `Entity.findOneByCodeInProject(code, projectId, otherCriteria)` — canonical
helper. With `projectId` set, filters `(project_id IS NULL OR project_id = ?)`
so structural (world/project/country) entities resolve correctly and
sub-country lookups stay in the requested project. Without `projectId`,
falls back to bare `findOne({ code })` — documented as such.
## Bare findOne audit fixes
Five mechanical callsites where project context was already available:
- `SurveyResponseVariablesExtractor.getVariablesByEntityCode` accepts
`surveyId` and resolves `survey.project_id` for the lookup.
`getParametersFromInput` threads it through.
- `exportSurveyResponses` passes its existing `surveyId` query param.
- `importSurveyResponses` × 3 callsites — `ANSWER_TRANSFORMERS[ENTITY]`
signature gains a `projectId` param, the per-column entity lookup and
`constructNewSurveyResponseDetails` both use `survey.project_id`.
- `datatrak-web/useEntityByCode` reads `projectId` from
`useCurrentUserContext`, threads it through `localContext` to the offline
query function, uses `findOneByCodeInProject`.
## Test fixtures
- `buildAndInsertProjectsAndHierarchies` rewritten — writes `entity.parent_id`
directly + `project_country` for project ↔ country edges, mirrors RN-1853's
per-project entity duplication so entity-server / sync-server tests see the
same shape as production.
- `clearTestData` deletes `project_country` before `entity` (the
`country_id` FK is `ON DELETE RESTRICT`).
- `CentralSyncManager.syncLookup.test.ts` seeds a `project_country` row to
exercise the new bridge.
## Adjacent fixes picked up
- vite.config: `react-router` added to dedupe for every package except psss
(psss intentionally has v5 + v6 react-router via `react-router-dom-v6`).
`@tanstack/react-query` + `@tanstack/react-query-devtools` added to dedupe
for the same class of bug.
- admin-panel entities table: new "Project" column (`source: 'project.code'`)
— useful QA tool for spotting per-project duplicates in the UI.
- Spec doc renamed `RN-1853-refinement.md` → `TUP-3056-refinement.md`,
contains full audit table and release-path plan.
## What's NOT in this PR
- Closure cache rebuild simplification (TUP-3068) — next PR in the stack.
- entity_relation consumer retirement (TUP-3065) — third PR in the stack.
Stacked PR plan:
1. This PR — TUP-3060 foundation
2. Next — TUP-3068 closure cache rebuild
3. Third — TUP-3065 entity_relation consumer retirement
* cleanup
* Update 20260501000001-backfillProjectIdsAndDuplicateSharedEntities-modifies-data.js
* feat(database): TUP-3068: simplify ancestor_descendant_relation rebuild (#6777)
* feat(database): TUP-3068: simplify ancestor_descendant_relation rebuild
Splits out of #6761. Second of three stacked PRs — stacks on #6776 (TUP-3060
foundation: project_country, ProjectCountry model, findOneByCodeInProject).
The `ancestor_descendant_relation` closure cache is kept as the read source
for hierarchy walks. What changes is how it's rebuilt — TUP-3068's brief was
to simplify the rebuild algorithm now that hierarchy edges live on
`entity.parent_id` + `project_country`.
The legacy 3-class pipeline (EntityHierarchyCacher → EntityHierarchySubtreeRebuilder
→ EntityParentChildRelationBuilder) with conditional `entity_relation` vs
`entity.parent_id` branching at each hierarchy level collapses into a single
recursive CTE per project.
## New: AncestorDescendantCacheBuilder
`AncestorDescendantCacheBuilder.rebuildForProject(projectId)` is the whole rebuild:
- One recursive CTE walking `entity.parent_id` (sub-country edges) ∪
`project_country` (project ↔ country bridge), scoped to one project's hierarchy.
- Wipe-and-rebuild per-project — DELETE for the hierarchy + INSERT ... SELECT
from the CTE, inside one transaction.
- Filters `child.type NOT IN ('project', 'country')` from the parent_id leg:
project.parent_id and country.parent_id both point at the world entity,
which is meta in the project-hierarchy model. Without this filter, world
surfaces as `parent_code` of every project/country and the frontend walks
up into 403s.
## Rewritten: EntityHierarchyCacher
Change-handler translators now key on `project_id` (was: `hierarchyId +
rootEntityId` pairs). Listens to `entity` / `projectCountry` / `entityHierarchy`
changes; each translates to "rebuild these project_ids' caches".
## Deleted
- `EntityHierarchySubtreeRebuilder.js` — its job is now the recursive CTE.
- `EntityParentChildRelationBuilder.js` — its `entity_relation` / parent_id
branching is gone. `entity_parent_child_relation` is no longer maintained
(drop is TUP-3066).
- Two old test fixture/test files for the deleted classes.
## Redirected
- `buildEntityParentChildRelationIfEmpty` — function name preserved for the
central-server bootstrap call site, but now checks
`ancestor_descendant_relation` (the surviving cache) and invokes
`AncestorDescendantCacheBuilder.rebuildAll`.
## Migration TRUNCATE
The TUP-3068 data migration (`20260507000001-backfillProjectCountry`) ends
with `TRUNCATE ancestor_descendant_relation`. This invalidates stale rows
from the pre-3068 cacher (which reference entity ids that RN-1853
duplicated/replaced) so the bootstrap rebuild on next central-server boot
sees an empty cache and runs `rebuildAll` against current state. Self-healing
for clean prod deploys.
The cleanup of `20201006211507-BuildAncestorDescendantRelationCache-modifies-data.js`
reduces it to a no-op shell so db-migrate's directory scan still loads cleanly
once the old builder it imported is gone.
The `repointSurveyResponses` `ANALYZE` step added to the RN-1853 migration is
an adjacent perf fix: post-entity-backfill stats are stale, the planner picks
a bad plan for the four-table-join repoint UPDATEs (observed 15 min on 400k
responses on tom-db; ANALYZE expected to drop runtime to a fraction).
## Entity.js rewrite
`getEntitiesFromParentChildRelation` (and its `getAncestorsFromParentChildRelation`
/ `getDescendantsFromParentChildRelation` wrappers) used to read from
`entity_parent_child_relation`. That table is no longer maintained — rewrite
walks `entity.parent_id` directly with a recursive CTE, scoped by project via
`(project_id IS NULL OR project_id = ?)`.
Also drops `getChildrenViaHierarchy` — no live callers, the only reader was
the legacy `entity_relation` table lookup.
## Test fixture rewrite (bundled here, not PR1)
`buildAndInsertProjectsAndHierarchies` rewritten to write `entity.parent_id`
directly + `project_country` for project ↔ country edges, mirroring RN-1853's
per-project entity duplication. This belongs with the cacher rewrite (PR2)
because the new cacher walks parent_id + project_country and would otherwise
find no edges in tests.
`clearTestData` orders `project_country` before `entity` (FK is
`ON DELETE RESTRICT`).
`CentralSyncManager.syncLookup.test.ts` seeds a `project_country` row to
exercise the new bridge.
## Tests
- New regression suite `Entity/getDescendantsAncestorsProjectScoping.test.js`
(9 tests). Triggers `AncestorDescendantCacheBuilder.rebuildForProject` in
`beforeEach`, asserts via the public `getDescendants` / `getAncestors` API.
- Entity-server test setup explicitly invokes
`AncestorDescendantCacheBuilder.rebuildAll()` in `beforeAll` — the
change-handler-driven cacher doesn't run in test mode.
- Datatrak-web `useSubmitSurveyResponse` — removed the offline shim that
manually wrote `entity_parent_child_relation` rows when datatrak inserted
new entities. With the cacher rebuilt from `entity.parent_id` directly, no
shim needed.
Stacked PR plan:
1. #6776 — TUP-3060 foundation (merged)
2. **This PR** — TUP-3068 closure cache rebuild + test fixture rewrite
3. #6778 — TUP-3065 entity_relation consumer retirement
* rename
* Update EntityHierarchyCacher.js
* Update Entity.js
* Update AncestorDescendantCacheBuilder.js
* Update Entity.js
* refactor
* feat(entityHierarchy): TUP-3065: switch consumers from entity_relation to project_country (#6778)
* feat(database): TUP-3068: simplify ancestor_descendant_relation rebuild
Splits out of #6761. Second of three stacked PRs — stacks on #6776 (TUP-3060
foundation: project_country, ProjectCountry model, findOneByCodeInProject).
The `ancestor_descendant_relation` closure cache is kept as the read source
for hierarchy walks. What changes is how it's rebuilt — TUP-3068's brief was
to simplify the rebuild algorithm now that hierarchy edges live on
`entity.parent_id` + `project_country`.
The legacy 3-class pipeline (EntityHierarchyCacher → EntityHierarchySubtreeRebuilder
→ EntityParentChildRelationBuilder) with conditional `entity_relation` vs
`entity.parent_id` branching at each hierarchy level collapses into a single
recursive CTE per project.
## New: AncestorDescendantCacheBuilder
`AncestorDescendantCacheBuilder.rebuildForProject(projectId)` is the whole rebuild:
- One recursive CTE walking `entity.parent_id` (sub-country edges) ∪
`project_country` (project ↔ country bridge), scoped to one project's hierarchy.
- Wipe-and-rebuild per-project — DELETE for the hierarchy + INSERT ... SELECT
from the CTE, inside one transaction.
- Filters `child.type NOT IN ('project', 'country')` from the parent_id leg:
project.parent_id and country.parent_id both point at the world entity,
which is meta in the project-hierarchy model. Without this filter, world
surfaces as `parent_code` of every project/country and the frontend walks
up into 403s.
## Rewritten: EntityHierarchyCacher
Change-handler translators now key on `project_id` (was: `hierarchyId +
rootEntityId` pairs). Listens to `entity` / `projectCountry` / `entityHierarchy`
changes; each translates to "rebuild these project_ids' caches".
## Deleted
- `EntityHierarchySubtreeRebuilder.js` — its job is now the recursive CTE.
- `EntityParentChildRelationBuilder.js` — its `entity_relation` / parent_id
branching is gone. `entity_parent_child_relation` is no longer maintained
(drop is TUP-3066).
- Two old test fixture/test files for the deleted classes.
## Redirected
- `buildEntityParentChildRelationIfEmpty` — function name preserved for the
central-server bootstrap call site, but now checks
`ancestor_descendant_relation` (the surviving cache) and invokes
`AncestorDescendantCacheBuilder.rebuildAll`.
## Migration TRUNCATE
The TUP-3068 data migration (`20260507000001-backfillProjectCountry`) ends
with `TRUNCATE ancestor_descendant_relation`. This invalidates stale rows
from the pre-3068 cacher (which reference entity ids that RN-1853
duplicated/replaced) so the bootstrap rebuild on next central-server boot
sees an empty cache and runs `rebuildAll` against current state. Self-healing
for clean prod deploys.
The cleanup of `20201006211507-BuildAncestorDescendantRelationCache-modifies-data.js`
reduces it to a no-op shell so db-migrate's directory scan still loads cleanly
once the old builder it imported is gone.
The `repointSurveyResponses` `ANALYZE` step added to the RN-1853 migration is
an adjacent perf fix: post-entity-backfill stats are stale, the planner picks
a bad plan for the four-table-join repoint UPDATEs (observed 15 min on 400k
responses on tom-db; ANALYZE expected to drop runtime to a fraction).
## Entity.js rewrite
`getEntitiesFromParentChildRelation` (and its `getAncestorsFromParentChildRelation`
/ `getDescendantsFromParentChildRelation` wrappers) used to read from
`entity_parent_child_relation`. That table is no longer maintained — rewrite
walks `entity.parent_id` directly with a recursive CTE, scoped by project via
`(project_id IS NULL OR project_id = ?)`.
Also drops `getChildrenViaHierarchy` — no live callers, the only reader was
the legacy `entity_relation` table lookup.
## Test fixture rewrite (bundled here, not PR1)
`buildAndInsertProjectsAndHierarchies` rewritten to write `entity.parent_id`
directly + `project_country` for project ↔ country edges, mirroring RN-1853's
per-project entity duplication. This belongs with the cacher rewrite (PR2)
because the new cacher walks parent_id + project_country and would otherwise
find no edges in tests.
`clearTestData` orders `project_country` before `entity` (FK is
`ON DELETE RESTRICT`).
`CentralSyncManager.syncLookup.test.ts` seeds a `project_country` row to
exercise the new bridge.
## Tests
- New regression suite `Entity/getDescendantsAncestorsProjectScoping.test.js`
(9 tests). Triggers `AncestorDescendantCacheBuilder.rebuildForProject` in
`beforeEach`, asserts via the public `getDescendants` / `getAncestors` API.
- Entity-server test setup explicitly invokes
`AncestorDescendantCacheBuilder.rebuildAll()` in `beforeAll` — the
change-handler-driven cacher doesn't run in test mode.
- Datatrak-web `useSubmitSurveyResponse` — removed the offline shim that
manually wrote `entity_parent_child_relation` rows when datatrak inserted
new entities. With the cacher rebuilt from `entity.parent_id` directly, no
shim needed.
Stacked PR plan:
1. #6776 — TUP-3060 foundation (merged)
2. **This PR** — TUP-3068 closure cache rebuild + test fixture rewrite
3. #6778 — TUP-3065 entity_relation consumer retirement
* refactor(central-server): TUP-3065: switch consumers from entity_relation to project_country
Splits out of #6761. Third of three stacked PRs — stacks on #6777 (TUP-3068
closure cache rebuild). #6776 (TUP-3060 foundation) already merged.
Mechanical replacement of `entity_relation`-based project↔country lookups
with `project_country` joins, plus retiring the now-orphaned `/entityRelations`
admin route and pruning multi-hierarchy test fixtures.
## Consumer switches
All four consumer paths joined `entity_relation` to discover a project's
countries; they now join `project_country` directly. Country code comes from
the country entity itself (`entity.code`), not from `entity.country_code`-on-a-child.
- `Project.countries()` — single source of truth
- `createDashboardRelationsDBFilter` — dashboards filtered by project access
- `assertMapOverlaysPermissions` — map overlay access checks
- `GETProjects.permissionsFilteredInternally` — project listing
- `hasAccessToEntityForVisualisation` / `hasTupaiaAdminAccessToEntityForVisualisation`
## CreateProject
`createProjectCountries` writes `project_country` rows on project creation
(was: `entity_relation`).
## Retired
- `/v1/entityRelations/:id` admin route + its assertion helper
- `DeleteEntity` no longer fires `entityRelation.delete` cascade
- Multi-hierarchy `Entity.test.js` cases
## Test fixture switch
Central-server test fixtures + the database-level
`buildAndInsertProjectsAndHierarchies` helper switched to seed
`project_country` only (PR2 had it write both to bridge the consumer switch).
## Out of scope
- Dropping the legacy tables — TUP-3066, gated on TUP-3067 (MediTrak compat)
Stacked PR plan:
1. #6776 — TUP-3060 foundation (merged)
2. #6777 — TUP-3068 closure cache rebuild
3. **This PR** — TUP-3065 entity_relation consumer retirement
* rename
* Update EntityHierarchyCacher.js
* Update Entity.js
* Update AncestorDescendantCacheBuilder.js
* Update Entity.js
* refactor
* plan
* remove comments
* Update Entity.test.js
* refactor: TUP-3066a: rename hierarchyId → projectId (#6786)
* feat(database): TUP-3068: simplify ancestor_descendant_relation rebuild
Splits out of #6761. Second of three stacked PRs — stacks on #6776 (TUP-3060
foundation: project_country, ProjectCountry model, findOneByCodeInProject).
The `ancestor_descendant_relation` closure cache is kept as the read source
for hierarchy walks. What changes is how it's rebuilt — TUP-3068's brief was
to simplify the rebuild algorithm now that hierarchy edges live on
`entity.parent_id` + `project_country`.
The legacy 3-class pipeline (EntityHierarchyCacher → EntityHierarchySubtreeRebuilder
→ EntityParentChildRelationBuilder) with conditional `entity_relation` vs
`entity.parent_id` branching at each hierarchy level collapses into a single
recursive CTE per project.
## New: AncestorDescendantCacheBuilder
`AncestorDescendantCacheBuilder.rebuildForProject(projectId)` is the whole rebuild:
- One recursive CTE walking `entity.parent_id` (sub-country edges) ∪
`project_country` (project ↔ country bridge), scoped to one project's hierarchy.
- Wipe-and-rebuild per-project — DELETE for the hierarchy + INSERT ... SELECT
from the CTE, inside one transaction.
- Filters `child.type NOT IN ('project', 'country')` from the parent_id leg:
project.parent_id and country.parent_id both point at the world entity,
which is meta in the project-hierarchy model. Without this filter, world
surfaces as `parent_code` of every project/country and the frontend walks
up into 403s.
## Rewritten: EntityHierarchyCacher
Change-handler translators now key on `project_id` (was: `hierarchyId +
rootEntityId` pairs). Listens to `entity` / `projectCountry` / `entityHierarchy`
changes; each translates to "rebuild these project_ids' caches".
## Deleted
- `EntityHierarchySubtreeRebuilder.js` — its job is now the recursive CTE.
- `EntityParentChildRelationBuilder.js` — its `entity_relation` / parent_id
branching is gone. `entity_parent_child_relation` is no longer maintained
(drop is TUP-3066).
- Two old test fixture/test files for the deleted classes.
## Redirected
- `buildEntityParentChildRelationIfEmpty` — function name preserved for the
central-server bootstrap call site, but now checks
`ancestor_descendant_relation` (the surviving cache) and invokes
`AncestorDescendantCacheBuilder.rebuildAll`.
## Migration TRUNCATE
The TUP-3068 data migration (`20260507000001-backfillProjectCountry`) ends
with `TRUNCATE ancestor_descendant_relation`. This invalidates stale rows
from the pre-3068 cacher (which reference entity ids that RN-1853
duplicated/replaced) so the bootstrap rebuild on next central-server boot
sees an empty cache and runs `rebuildAll` against current state. Self-healing
for clean prod deploys.
The cleanup of `20201006211507-BuildAncestorDescendantRelationCache-modifies-data.js`
reduces it to a no-op shell so db-migrate's directory scan still loads cleanly
once the old builder it imported is gone.
The `repointSurveyResponses` `ANALYZE` step added to the RN-1853 migration is
an adjacent perf fix: post-entity-backfill stats are stale, the planner picks
a bad plan for the four-table-join repoint UPDATEs (observed 15 min on 400k
responses on tom-db; ANALYZE expected to drop runtime to a fraction).
## Entity.js rewrite
`getEntitiesFromParentChildRelation` (and its `getAncestorsFromParentChildRelation`
/ `getDescendantsFromParentChildRelation` wrappers) used to read from
`entity_parent_child_relation`. That table is no longer maintained — rewrite
walks `entity.parent_id` directly with a recursive CTE, scoped by project via
`(project_id IS NULL OR project_id = ?)`.
Also drops `getChildrenViaHierarchy` — no live callers, the only reader was
the legacy `entity_relation` table lookup.
## Test fixture rewrite (bundled here, not PR1)
`buildAndInsertProjectsAndHierarchies` rewritten to write `entity.parent_id`
directly + `project_country` for project ↔ country edges, mirroring RN-1853's
per-project entity duplication. This belongs with the cacher rewrite (PR2)
because the new cacher walks parent_id + project_country and would otherwise
find no edges in tests.
`clearTestData` orders `project_country` before `entity` (FK is
`ON DELETE RESTRICT`).
`CentralSyncManager.syncLookup.test.ts` seeds a `project_country` row to
exercise the new bridge.
## Tests
- New regression suite `Entity/getDescendantsAncestorsProjectScoping.test.js`
(9 tests). Triggers `AncestorDescendantCacheBuilder.rebuildForProject` in
`beforeEach`, asserts via the public `getDescendants` / `getAncestors` API.
- Entity-server test setup explicitly invokes
`AncestorDescendantCacheBuilder.rebuildAll()` in `beforeAll` — the
change-handler-driven cacher doesn't run in test mode.
- Datatrak-web `useSubmitSurveyResponse` — removed the offline shim that
manually wrote `entity_parent_child_relation` rows when datatrak inserted
new entities. With the cacher rebuilt from `entity.parent_id` directly, no
shim needed.
Stacked PR plan:
1. #6776 — TUP-3060 foundation (merged)
2. **This PR** — TUP-3068 closure cache rebuild + test fixture rewrite
3. #6778 — TUP-3065 entity_relation consumer retirement
* refactor(central-server): TUP-3065: switch consumers from entity_relation to project_country
Splits out of #6761. Third of three stacked PRs — stacks on #6777 (TUP-3068
closure cache rebuild). #6776 (TUP-3060 foundation) already merged.
Mechanical replacement of `entity_relation`-based project↔country lookups
with `project_country` joins, plus retiring the now-orphaned `/entityRelations`
admin route and pruning multi-hierarchy test fixtures.
## Consumer switches
All four consumer paths joined `entity_relation` to discover a project's
countries; they now join `project_country` directly. Country code comes from
the country entity itself (`entity.code`), not from `entity.country_code`-on-a-child.
- `Project.countries()` — single source of truth
- `createDashboardRelationsDBFilter` — dashboards filtered by project access
- `assertMapOverlaysPermissions` — map overlay access checks
- `GETProjects.permissionsFilteredInternally` — project listing
- `hasAccessToEntityForVisualisation` / `hasTupaiaAdminAccessToEntityForVisualisation`
## CreateProject
`createProjectCountries` writes `project_country` rows on project creation
(was: `entity_relation`).
## Retired
- `/v1/entityRelations/:id` admin route + its assertion helper
- `DeleteEntity` no longer fires `entityRelation.delete` cascade
- Multi-hierarchy `Entity.test.js` cases
## Test fixture switch
Central-server test fixtures + the database-level
`buildAndInsertProjectsAndHierarchies` helper switched to seed
`project_country` only (PR2 had it write both to bridge the consumer switch).
## Out of scope
- Dropping the legacy tables — TUP-3066, gated on TUP-3067 (MediTrak compat)
Stacked PR plan:
1. #6776 — TUP-3060 foundation (merged)
2. #6777 — TUP-3068 closure cache rebuild
3. **This PR** — TUP-3065 entity_relation consumer retirement
* rename
* Update EntityHierarchyCacher.js
* Update Entity.js
* Update AncestorDescendantCacheBuilder.js
* Update Entity.js
* refactor
* plan
* refactor: TUP-3066: rename hierarchyId → projectId in entity hierarchy code path
Each project has exactly one hierarchy, so the indirection through
`entity_hierarchy_id` is dead weight. Renames `ancestor_descendant_relation.entity_hierarchy_id`
to `project_id` (schema migration + types regen) and ripples the parameter
rename `hierarchyId` → `projectId` across:
- `Entity.js` hierarchy walk methods (record + model)
- `AncestorDescendantRelation.js` getImmediateRelations + caches
- `AncestorDescendantCacheBuilder.js` rebuildForProject
- entity-server hierarchy routes + middleware (CommonContext)
- datatrak-web entity accessors
- web-config-server `fetchHierarchyId` → `fetchProjectId`
- `fetchDefaultEntityHierarchyIdPatiently` → `fetchDefaultProjectIdPatiently`
(rewritten to query projects via ancestor_descendant_relation)
Drops:
- `getChildrenViaHierarchy` (all callers retired in PR #6778)
- `findOneOrThrow({entity_hierarchy_id})` lookup inside
`getEntitiesFromParentChildRelation` — closes review-hero comment
#3217296820 on PR #6777
Also restores a typo introduced by 90a0ebbf2
(`getEntitiesFromParentChildRelagetEntitiesFromParentChildRelationtion`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: TUP-3066a: fetchDefaultProjectIdPatiently sort by project.code
The project table has no `name` column (that lived on entity_hierarchy). Sort
by `code` to mirror the original alphabetical-fallback behaviour.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: TUP-3066a: backport CI fixups (data-broker stub, EventBuilder fixture, types regen)
Same drifts surfaced on this PR as on TUP-3066b; smaller subset because
this branch only renames the column, doesn't drop the legacy tables:
- data-broker DhisService stubs: swap entityHierarchy → project to match
EventsPuller.ts's new lookup path
- central-server EventBuilder.test.js: ancestor_descendant_relation now
keyed by project_id (the column was renamed)
- @tupaia/types models.ts + schemas.ts:
- Analytics{,Create,Update}.type now QuestionType (analytics MV column
is question_type on CI's fresh build)
- EntityType enum order: document_group before document, tamanu_country
between ird_village and srh_district (matches pg_enum.enumsortorder)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* remove comments
* Update Entity.test.js
* Update Entity.js
* fix: TUP-3066a: address review-hero + bugbot feedback
- surveyDataExport.fetchProjectId: return project.id (not entity_hierarchy_id)
- migration: delete orphan ancestor_descendant_relation rows before SET NOT NULL
- Entity.fetchDefaultProjectIdPatiently: dedupe project_id in SQL via DISTINCT
- data-broker models: extract named Project type to match file convention
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: TUP-3066b: drop entity_relation, entity_parent_child_relation, entity_hierarchy (#6787)
* feat(database): TUP-3068: simplify ancestor_descendant_relation rebuild
Splits out of #6761. Second of three stacked PRs — stacks on #6776 (TUP-3060
foundation: project_country, ProjectCountry model, findOneByCodeInProject).
The `ancestor_descendant_relation` closure cache is kept as the read source
for hierarchy walks. What changes is how it's rebuilt — TUP-3068's brief was
to simplify the rebuild algorithm now that hierarchy edges live on
`entity.parent_id` + `project_country`.
The legacy 3-class pipeline (EntityHierarchyCacher → EntityHierarchySubtreeRebuilder
→ EntityParentChildRelationBuilder) with conditional `entity_relation` vs
`entity.parent_id` branching at each hierarchy level collapses into a single
recursive CTE per project.
## New: AncestorDescendantCacheBuilder
`AncestorDescendantCacheBuilder.rebuildForProject(projectId)` is the whole rebuild:
- One recursive CTE walking `entity.parent_id` (sub-country edges) ∪
`project_country` (project ↔ country bridge), scoped to one project's hierarchy.
- Wipe-and-rebuild per-project — DELETE for the hierarchy + INSERT ... SELECT
from the CTE, inside one transaction.
- Filters `child.type NOT IN ('project', 'country')` from the parent_id leg:
project.parent_id and country.parent_id both point at the world entity,
which is meta in the project-hierarchy model. Without this filter, world
surfaces as `parent_code` of every project/country and the frontend walks
up into 403s.
## Rewritten: EntityHierarchyCacher
Change-handler translators now key on `project_id` (was: `hierarchyId +
rootEntityId` pairs). Listens to `entity` / `projectCountry` / `entityHierarchy`
changes; each translates to "rebuild these project_ids' caches".
## Deleted
- `EntityHierarchySubtreeRebuilder.js` — its job is now the recursive CTE.
- `EntityParentChildRelationBuilder.js` — its `entity_relation` / parent_id
branching is gone. `entity_parent_child_relation` is no longer maintained
(drop is TUP-3066).
- Two old test fixture/test files for the deleted classes.
## Redirected
- `buildEntityParentChildRelationIfEmpty` — function name preserved for the
central-server bootstrap call site, but now checks
`ancestor_descendant_relation` (the surviving cache) and invokes
`AncestorDescendantCacheBuilder.rebuildAll`.
## Migration TRUNCATE
The TUP-3068 data migration (`20260507000001-backfillProjectCountry`) ends
with `TRUNCATE ancestor_descendant_relation`. This invalidates stale rows
from the pre-3068 cacher (which reference entity ids that RN-1853
duplicated/replaced) so the bootstrap rebuild on next central-server boot
sees an empty cache and runs `rebuildAll` against current state. Self-healing
for clean prod deploys.
The cleanup of `20201006211507-BuildAncestorDescendantRelationCache-modifies-data.js`
reduces it to a no-op shell so db-migrate's directory scan still loads cleanly
once the old builder it imported is gone.
The `repointSurveyResponses` `ANALYZE` step added to the RN-1853 migration is
an adjacent perf fix: post-entity-backfill stats are stale, the planner picks
a bad plan for the four-table-join repoint UPDATEs (observed 15 min on 400k
responses on tom-db; ANALYZE expected to drop runtime to a fraction).
## Entity.js rewrite
`getEntitiesFromParentChildRelation` (and its `getAncestorsFromParentChildRelation`
/ `getDescendantsFromParentChildRelation` wrappers) used to read from
`entity_parent_child_relation`. That table is no longer maintained — rewrite
walks `entity.parent_id` directly with a recursive CTE, scoped by project via
`(project_id IS NULL OR project_id = ?)`.
Also drops `getChildrenViaHierarchy` — no live callers, the only reader was
the legacy `entity_relation` table lookup.
## Test fixture rewrite (bundled here, not PR1)
`buildAndInsertProjectsAndHierarchies` rewritten to write `entity.parent_id`
directly + `project_country` for project ↔ country edges, mirroring RN-1853's
per-project entity duplication. This belongs with the cacher rewrite (PR2)
because the new cacher walks parent_id + project_country and would otherwise
find no edges in tests.
`clearTestData` orders `project_country` before `entity` (FK is
`ON DELETE RESTRICT`).
`CentralSyncManager.syncLookup.test.ts` seeds a `project_country` row to
exercise the new bridge.
## Tests
- New regression suite `Entity/getDescendantsAncestorsProjectScoping.test.js`
(9 tests). Triggers `AncestorDescendantCacheBuilder.rebuildForProject` in
`beforeEach`, asserts via the public `getDescendants` / `getAncestors` API.
- Entity-server test setup explicitly invokes
`AncestorDescendantCacheBuilder.rebuildAll()` in `beforeAll` — the
change-handler-driven cacher doesn't run in test mode.
- Datatrak-web `useSubmitSurveyResponse` — removed the offline shim that
manually wrote `entity_parent_child_relation` rows when datatrak inserted
new entities. With the cacher rebuilt from `entity.parent_id` directly, no
shim needed.
Stacked PR plan:
1. #6776 — TUP-3060 foundation (merged)
2. **This PR** — TUP-3068 closure cache rebuild + test fixture rewrite
3. #6778 — TUP-3065 entity_relation consumer retirement
* refactor(central-server): TUP-3065: switch consumers from entity_relation to project_country
Splits out of #6761. Third of three stacked PRs — stacks on #6777 (TUP-3068
closure cache rebuild). #6776 (TUP-3060 foundation) already merged.
Mechanical replacement of `entity_relation`-based project↔country lookups
with `project_country` joins, plus retiring the now-orphaned `/entityRelations`
admin route and pruning multi-hierarchy test fixtures.
## Consumer switches
All four consumer paths joined `entity_relation` to discover a project's
countries; they now join `project_country` directly. Country code comes from
the country entity itself (`entity.code`), not from `entity.country_code`-on-a-child.
- `Project.countries()` — single source of truth
- `createDashboardRelationsDBFilter` — dashboards filtered by project access
- `assertMapOverlaysPermissions` — map overlay access checks
- `GETProjects.permissionsFilteredInternally` — project listing
- `hasAccessToEntityForVisualisation` / `hasTupaiaAdminAccessToEntityForVisualisation`
## CreateProject
`createProjectCountries` writes `project_country` rows on project creation
(was: `entity_relation`).
## Retired
- `/v1/entityRelations/:id` admin route + its assertion helper
- `DeleteEntity` no longer fires `entityRelation.delete` cascade
- Multi-hierarchy `Entity.test.js` cases
## Test fixture switch
Central-server test fixtures + the database-level
`buildAndInsertProjectsAndHierarchies` helper switched to seed
`project_country` only (PR2 had it write both to bridge the consumer switch).
## Out of scope
- Dropping the legacy tables — TUP-3066, gated on TUP-3067 (MediTrak compat)
Stacked PR plan:
1. #6776 — TUP-3060 foundation (merged)
2. #6777 — TUP-3068 closure cache rebuild
3. **This PR** — TUP-3065 entity_relation consumer retirement
* rename
* Update EntityHierarchyCacher.js
* Update Entity.js
* Update AncestorDescendantCacheBuilder.js
* Update Entity.js
* refactor
* plan
* refactor: TUP-3066: rename hierarchyId → projectId in entity hierarchy code path
Each project has exactly one hierarchy, so the indirection through
`entity_hierarchy_id` is dead weight. Renames `ancestor_descendant_relation.entity_hierarchy_id`
to `project_id` (schema migration + types regen) and ripples the parameter
rename `hierarchyId` → `projectId` across:
- `Entity.js` hierarchy walk methods (record + model)
- `AncestorDescendantRelation.js` getImmediateRelations + caches
- `AncestorDescendantCacheBuilder.js` rebuildForProject
- entity-server hierarchy routes + middleware (CommonContext)
- datatrak-web entity accessors
- web-config-server `fetchHierarchyId` → `fetchProjectId`
- `fetchDefaultEntityHierarchyIdPatiently` → `fetchDefaultProjectIdPatiently`
(rewritten to query projects via ancestor_descendant_relation)
Drops:
- `getChildrenViaHierarchy` (all callers retired in PR #6778)
- `findOneOrThrow({entity_hierarchy_id})` lookup inside
`getEntitiesFromParentChildRelation` — closes review-hero comment
#3217296820 on PR #6777
Also restores a typo introduced by 90a0ebbf2
(`getEntitiesFromParentChildRelagetEntitiesFromParentChildRelationtion`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: TUP-3066b: drop entity_relation, entity_parent_child_relation, entity_hierarchy
Subtractive cleanup once TUP-3066a's rename is in: the closure cache is the only
hierarchy reader still standing and it's keyed by project_id, so the three legacy
tables can go.
⚠ GATED ON TUP-3067 (MediTrak compatibility). The schema migration must not run in
production until mobile sync is no longer reading entity_parent_child_relation
through the boilerplate sync lookup. Draft now to review the code shape; merge
once 3067 ships.
Schema migration drops `project.entity_hierarchy_id` then the three tables in
FK-dependency order (down migration restores structure + minimal seed; the
relation-row data is reproducible from entity.parent_id + project_country).
Code:
- Delete model classes: EntityRelation, EntityParentChildRelation, EntityHierarchy
- Drop ENTITY_RELATION / ENTITY_HIERARCHY / ENTITY_PARENT_CHILD_RELATION records,
syncing-tables list, post-migration trigger list, clearTestData list
- Rewrite Entity.buildSyncLookupQueryDetails to compute project_ids without
entity_parent_child_relation joins
- Retire EntityHierarchyCacher.translateEntityHierarchyChange (the table is gone)
- Drop CreateProject.createEntityHierarchy + project.entity_hierarchy_id field
- Drop DeleteEntity entity_relation walk; now walks per-project via project_country
- Drop test fixture's entity_hierarchy creation
- Delete admin routes:
- central-server: /entityHierarchy GET + PUT routes + tests
- entity-server: /hierarchies route + integration tests
- Drop project.entity_hierarchy_id projection in GETProjects + Project.find
Types are NOT regenerated in this commit because the legacy tables still exist on
the dev DB; the regen happens when the schema migration is applied alongside
TUP-3067.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Update Entity.test.js
* fix: TUP-3066b: drop type references to deleted entity-hierarchy models
Followups noticed during local test setup:
- tsmodels: delete EntityHierarchy.ts and EntityParentChildRelation.ts stubs +
remove their re-exports from models/index.ts
- server-boilerplate: drop EntityHierarchyModel/Record re-export
- sync-server: drop deleted models from SyncServerModelRegistry +
TestSyncServerModelRegistry shapes
- datatrak-web: drop entityHierarchy/entityParentChildRelation fields from
DatatrakWebModelRegistry
Also: in fetchDefaultProjectIdPatiently, sort projects by `code` instead of `name`
(project table has no `name` column — that column lived on entity_hierarchy
which is gone).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: TUP-3066a: fetchDefaultProjectIdPatiently sort by project.code
The project table has no `name` column (that lived on entity_hierarchy). Sort
by `code` to mirror the original alphabetical-fallback behaviour.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: TUP-3066b: clear remaining references to dropped legacy tables in tests + types
Catches the CI fallout from PR #6787:
- Regenerate @tupaia/types (drops EntityHierarchy / EntityRelation /
EntityParentChildRelation from models.ts + schemas.ts)
- entity-server src/types.ts + __tests__/types.ts: drop EntityHierarchyModel /
EntityRelationModel from registries
- data-broker DhisService stubs: replace ENTITY_HIERARCHIES fixture with
PROJECTS stub so EventsPuller's models.project.findOne resolves
- Database tests:
- Entity/getDescendantsAncestorsProjectScoping.test.js: stop creating
entity_hierarchy + entity_hierarchy_id on project
- Central-server tests:
- apiV2/landingPages/utils.js: drop entity_hierarchy create
- apiV2/projects/CreateProject.test.js: drop the "creates a valid entity
hierarchy record" assertion + entityHierarchy cleanup
- apiV2/projects/EditProject.test.js, GETProjects.test.js: same cleanup
- apiV2/surveys/CreateAndEditSurveys.test.js: project fixture without
entity_hierarchy_id
- dhis/EventBuilder.test.js: ancestor_descendant_relation now keyed by
project_id; create project instead of entity_hierarchy
- Sync-server tests:
- CentralSyncManager.pull.test.ts: drop entityHierarchy from prepareData
- CentralSyncManager.syncLookup.test.ts: drop entity_parent_child_relation
insert; entity2's parent linked via entity.parent_id instead
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: TUP-3066b: include analytics types + clear leftover test references
CI surfaced three remaining issues after the first pass:
- @tupaia/types Validate: regenerate against a DB that has the analytics
materialized view, so `Analytics` / `AnalyticsCreate` / `AnalyticsUpdate`
interfaces are present alongside the dropped legacy table cleanup
- entity-server permissions.test.ts: remove the "filters hierarchies when
requested for some with access" case (route `/v1/hierarchies` is gone)
and the unused `getHierarchiesWithFields` import
- sync-server:
- CentralSyncManager.pull.test.ts: snapshotRecords expectation now 3
(country + userAccount + project) — entityHierarchy isn't seeded anymore
- CentralSyncManager.syncLookup.test.ts: drop the now-unused `entity2`
let-binding (the entity is still inserted, just not held for later use)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: TUP-3066b: match types regen against CI (analytics.type enum, entity_type order)
CI generates types against a freshly-built DB; my local dev DB has older
state and produced two drifts:
- Analytics{,Create,Update}.type: my dev DB has the analytics MV with column
type 'text'. CI's fresh MV has 'question_type', so the generated TS type is
QuestionType. Match CI by changing 'string | null' → 'QuestionType | null'
in the three Analytics interfaces.
- EntityType enum order: pg_enum.enumsortorder differs between my dev DB and
CI. Reorder document_group/document and tamanu_country to match CI's
insertion order.
schemas.ts uses alphabetical ordering and didn't need changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: TUP-3066b: match types schemas.ts against CI regen
Three more drifts caught by the Validate types CI step:
- ENTITY_TYPE_CHILDREN map (used by UserAccountPreferencesSchema):
document_group before document, and tamanu_country between ird_village
and srh_district — matches Postgres pg_enum.enumsortorder on CI
- Analytics{,Create,Update}Schema.type now includes a `QuestionType` enum
array (the analytics MV column is question_type on CI's fresh build).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: TUP-3066a: backport CI fixups (data-broker stub, EventBuilder fixture, types regen)
Same drifts surfaced on this PR as on TUP-3066b; smaller subset because
this branch only renames the column, doesn't drop the legacy tables:
- data-broker DhisService stubs: swap entityHierarchy → project to match
EventsPuller.ts's new lookup path
- central-server EventBuilder.test.js: ancestor_descendant_relation now
keyed by project_id (the column was renamed)
- @tupaia/types models.ts + schemas.ts:
- Analytics{,Create,Update}.type now QuestionType (analytics MV column
is question_type on CI's fresh build)
- EntityType enum order: document_group before document, tamanu_country
between ird_village and srh_district (matches pg_enum.enumsortorder)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Update index.js
* specs
* fix: TUP-3066b: restore review-fix changes lost in merge conflict
Merging epic-entity-hierarchy into tup-3066b-drop-legacy-tables dropped
five fixes that were correctly squash-merged via PRs #6778 and #6786.
Restored:
- surveyDataExport.fetchProjectId: return project.id (not entity_hierarchy_id)
- migration: delete orphan ancestor_descendant_relation rows before SET NOT NULL
- Entity.fetchDefaultProjectIdPatiently: dedupe project_id in SQL via DISTINCT
- Entity.findOneByCodeInProject: forward 4th options arg (was silently dropped)
- data-broker models: extract named Project type to match file convention
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: TUP-3066b: restore central-server project changes lost in merge
The recent merge of epic-entity-hierarchy reintroduced entity_hierarchy
references in CreateProject.js and the project test rollback helpers.
This was the same merge-conflict-lost-work pattern as commit ae6a9d8fc;
restoring from 55ddae54c (last known-good CI state):
- CreateProject.js: drop createEntityHierarchy method, its call site,
and the entity_hierarchy_id field on project create
- CreateProject.test.js: drop "creates a valid entity hierarchy record"
test and entityHierarchy.delete from rollbackRecords
- EditProject.test.js: drop entityHierarchy.delete from rollbackRecords
Fixes the 5 failing central-server tests on CI — the after-each hook
crash on the missing entityHierarchy model was cascading sinon stub
leakage into the next two test files.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Delete TUP-3066-refinement.md
* Update entity-hierarchy-improvements.md
* refactor: TUP-3156: delete MS1 sync (#6789)
Andrew confirmed in the TUP-3156 refinement that MS1 is no longer used:
last response in 2023, prior in 2021. The sync is bare entity-code
lookups outside any request context — post-3056 these silently return
arbitrary copies of duplicated entities. Cleaner to remove than to
project-scope a dead path.
Deletes:
- packages/central-server/src/ms1/ directory (sync code)
- Ms1SyncLog + Ms1SyncQueue model wrappers (only callers were in /ms1/)
- startSyncWithMs1 call in index.js
Out of scope:
- DB tables ms1_sync_queue / ms1_sync_log stay (separate cleanup)
- types/EntityMetadata.ms1 field stays (no readers/writers, harmless)
- clearTestData entries stay (tables still exist, no-op cleanup is fine)
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Update index.js
* clean up
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: TUP-3156: project-scope KoBo + data-broker context (#6790)
* feat(database): TUP-3068: simplify ancestor_descendant_relation rebuild
Splits out of #6761. Second of three stacked PRs — stacks on #6776 (TUP-3060
foundation: project_country, ProjectCountry model, findOneByCodeInProject).
The `ancestor_descendant_relation` closure cache is kept as the read source
for hierarchy walks. What changes is how it's rebuilt — TUP-3068's brief was
to simplify the rebuild algorithm now that hierarchy edges live on
`entity.parent_id` + `project_country`.
The legacy 3-class pipeline (EntityHierarchyCacher → EntityHierarchySubtreeRebuilder
→ EntityParentChildRelationBuilder) with conditional `entity_relation` vs
`entity.parent_id` branching at each hierarchy level collapses into a single
recursive CTE per project.
## New: AncestorDescendantCacheBuilder
`AncestorDescendantCacheBuilder.rebuildForProject(projectId)` is the whole rebuild:
- One recursive CTE walking `entity.parent_id` (sub-country edges) ∪
`project_country` (project ↔ country bridge), scoped to one project's hierarchy.
- Wipe-and-rebuild per-project — DELETE for the hierarchy + INSERT ... SELECT
from the CTE, inside one transaction.
- Filters `child.type NOT IN ('project', 'country')` from the parent_id leg:
project.parent_id and country.parent_id both point at the world entity,
which is meta in the project-hierarchy model. Without this filter, world
surfaces as `parent_code` of every project/country and the frontend walks
up into 403s.
## Rewritten: EntityHierarchyCacher
Change-handler translators now key on `project_id` (was: `hierarchyId +
rootEntityId` pairs). Listens to `entity` / `projectCountry` / `entityHierarchy`
changes; each translates to "rebuild these project_ids' caches".
## Deleted
- `EntityHierarchySubtreeRebuilder.js` — its job is now the recursive CTE.
- `EntityParentChildRelationBuilder.js` — its `entity_relation` / parent_id
branching is gone. `entity_parent_child_relation` is no longer maintained
(drop is TUP-3066).
- Two old test fixture/test files for the deleted classes.
## Redirected
- `buildEntityParentChildRelationIfEmpty` — function name preserved for the
central-server bootstrap call site, but now checks
`ancestor_descendant_relation` (the surviving cache) and invokes
`AncestorDescendantCacheBuilder.rebuildAll`.
## Migration TRUNCATE
The TUP-3068 data migration (`20260507000001-backfillProjectCountry`) ends
with `TRUNCATE ancestor_descendant_relation`. This invalidates stale rows
from the pre-3068 cacher (which reference entity ids that RN-1853
duplicated/replaced) so the bootstrap rebuild on next central-server boot
sees an empty cache and runs `rebuildAll` against current state. Self-healing
for clean prod deploys.
The cleanup of `20201006211507-BuildAncestorDescendantRelationCache-modifies-data.js`
reduces it to a no-op shell so db-migrate's directory scan still loads cleanly
once the old builder it imported is gone.
The `repointSurveyResponses` `ANALYZE` step added to the RN-1853 migration is
an adjacent perf fix: post-entity-backfill stats are stale, the planner picks
a bad plan for the four-table-join repoint UPDATEs (observed 15 min on 400k
responses on tom-db; ANALYZE expected to drop runtime to a fraction).
## Entity.js rewrite
`getEntitiesFromParentChildRelation` (and its `getAncestorsFromParentChildRelation`
/ `getDescendantsFromParentChildRelation` wrappers) used to read from
`entity_parent_child_relation`. That table is no longer maintained — rewrite
walks `entity.parent_id` directly with a recursive CTE, scoped by project via
`(project_id IS NULL OR project_id = ?)`.
Also drops `getChildrenViaHierarchy` — no live callers, the only reader was
the legacy `entity_relation` table lookup.
## Test fixture rewrite (bundled here, not PR1)
`buildAndInsertProjectsAndHierarchies` rewritten to write `entity.parent_id`
directly + `project_country` for project ↔ country edges, mirroring RN-1853's
per-project entity duplication. This belongs with the cacher rewrite (PR2)
because the new cacher walks parent_id + project_country and would otherwise
find no edges in tests.
`clearTestData` orders `project_country` before `entity` (FK is
`ON DELETE RESTRICT`).
`CentralSyncManager.syncLookup.test.ts` seeds a `project_country` row to
exercise the new bridge.
## Tests
- New regression suite `Entity/getDescendantsAncestorsProjectScoping.test.js`
(9 tests). Triggers `AncestorDescendantCacheBuilder.rebuildForProject` in
`beforeEach`, asserts via the public `getDescendants` / `getAncestors` API.
- Entity-server test setup explicitly invokes
`AncestorDescendantCacheBuilder.rebuildAll()` in `beforeAll` — the
change-handler-driven cacher doesn't run in test mode.
- Datatrak-web `useSubmitSurveyResponse` — removed the offline shim that
manually wrote `entity_parent_child_relation` rows when datatrak inserted
new entities. With the cacher rebuilt from `entity.parent_id` directly, no
shim needed.
Stacked PR plan:
1. #6776 — TUP-3060 foundation (merged)
2. **This PR** — TUP-3068 closure cache rebuild + test fixture rewrite
3. #6778 — TUP-3065 entity_relation consumer retirement
* refactor(central-server): TUP-3065: switch consumers from entity_relation to project_country
Splits out of #6761. Third of three stacked PRs — stacks on #6777 (TUP-3068
closure cache rebuild). #6776 (TUP-3060 foundation) already merged.
Mechanical replacement of `entity_relation`-based project↔country lookups
with `project_country` joins, plus retiring the now-orphaned `/entityRelations`
admin route and pruning multi-hierarchy test fixtures.
## Consumer switches
All four consumer paths joined `entity_relation` to discover a project's
countries; they now join `project_country` directly. Country code comes from
the country entity itself (`entity.code`), not from `entity.country_code`-on-a-child.
- `Project.countries()` — single source of truth
- `createDashboardRelationsDBFilter` — dashboards filtered by project access
- `assertMapOverlaysPermissions` — map overlay access checks
- `GETProjects.permissionsFilteredInternally` — project listing
- `hasAccessToEntityForVisualisation` / `hasTupaiaAdminAccessToEntityForVisualisation`
## CreateProject
`createProjectCountries` writes `project_country` rows on project creation
(was: `entity_relation`).
## Retired
- `/v1/entityRelations/:id` admin route + its assertion helper
- `DeleteEntity` no longer fires `entityRelation.delete` cascade
- Multi-hierarchy `Entity.test.js` cases
## Test fixture switch
Central-server test fixtures + the database-level
`buildAndInsertProjectsAndHierarchies` helper switched to seed
`project_country` only (PR2 had it write both to bridge the consumer switch).
## Out of scope
- Dropping the legacy tables — TUP-3066, gated on TUP-3067 (MediTrak compat)
Stacked PR plan:
1. #6776 — TUP-3060 foundation (merged)
2. #6777 — TUP-3068 closure cache rebuild
3. **This PR** — TUP-3065 entity_relation consumer retirement
* rename
* Update EntityHierarchyCacher.js
* Update Entity.js
* Update AncestorDescendantCacheBuilder.js
* Update Entity.js
* refactor
* plan
* refactor: TUP-3066: rename hierarchyId → projectId in entity hierarchy code path
Each project has exactly one hierarchy, so the indirection through
`entity_hierarchy_id` is dead weight. Renames `ancestor_descendant_relation.entity_hierarchy_id`
to `project_id` (schema migration + types regen) and ripples the parameter
rename `hierarchyId` → `projectId` across:
- `Entity.js` hierarchy walk methods (record + model)
- `AncestorDescendantRelation.js` getImmediateRelations + caches
- `AncestorDescendantCacheBuilder.js` rebuildForProject
- entity-server hierarchy routes + middleware (CommonContext)
- datatrak-web entity accessors
- web-config-server `fetchHierarchyId` → `fetchProjectId`
- `fetchDefaultEntityHierarchyIdPatiently` → `fetchDefaultProjectIdPatiently`
(rewritten to query projects via ancestor_descendant_relation)
Drops:
- `getChildrenViaHierarchy` (all callers retired in PR #6778)
- `findOneOrThrow({entity_hierarchy_id})` lookup inside
`getEntitiesFromParentChildRelation` — closes review-hero comment
#3217296820 on PR #6777
Also restores a typo introduced by 90a0ebbf2
(`getEntitiesFromParentChildRelagetEntitiesFromParentChildRelationtion`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: TUP-3066b: drop entity_relation, entity_parent_child_relation, entity_hierarchy
Subtractive cleanup once TUP-3066a's rename is in: the closure cache is the only
hierarchy reader still standing and it's keyed by project_id, so the three legacy
tables can go.
⚠ GATED ON TUP-3067 (MediTrak compatibility). The schema migration must not run in
production until mobile sync is no longer reading entity_parent_child_relation
through the boilerplate sync lookup. Draft now to review the code shape; merge
once 3067 ships.
Schema migration drops `project.entity_hierarchy_id` then the three tables in
FK-dependency order (down migration restores structure + minimal seed; the
relation-row data is reproducible from entity.parent_id + project_country).
Code:
- Delete model classes: EntityRelation, EntityParentChildRelation, EntityHierarchy
- Drop ENTITY_RELATION / ENTITY_HIERARCHY / ENTITY_PARENT_CHILD_RELATION records,
syncing-tables list, post-migration trigger list, clearTestData list
- Rewrite Entity.buildSyncLookupQueryDetails to compute project_ids without
entity_parent_child_relation joins
- Retire EntityHierarchyCacher.translateEntityHierarchyChange (the table is gone)
- Drop CreateProject.createEntityHierarchy + project.entity_hierarchy_id field
- Drop DeleteEntity entity_relation walk; now walks per-project via project_country
- Drop test fixture's entity_hierarchy creation
- Delete admin routes:
- central-server: /entityHierarchy GET + PUT routes + tests
- entity-server: /hierarchies route + integration tests
- Drop project.entity_hierarchy_id projection in GETProjects + Project.find
Types are NOT regenerated in this commit because the legacy tables still exist on
the dev DB; the regen happens when the schema migration is applied alongside
TUP-3067.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Update Entity.test.js
* fix: TUP-3066b: drop type references to deleted entity-hierarchy models
Followups noticed during local test setup:
- tsmodels: delete EntityHierarchy.ts and EntityParentChildRelation.ts stubs +
remove their re-exports from models/index.ts
- server-boilerplate: drop EntityHierarchyModel/Record re-export
- sync-server: drop deleted models from SyncServerModelRegistry +
TestSyncServerModelRegistry shapes
- datatrak-web: drop entityHierarchy/entityParentChildRelation fields from
DatatrakWebModelRegistry
Also: in fetchDefaultProjectIdPatiently, sort projects by `code` instead of `name`
(project table has no `name` column — that column lived on entity_hierarchy
which is gone).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: TUP-3066a: fetchDefaultProjectIdPatiently sort by project.code
The project table has no `name` column (that lived on entity_hierarchy). Sort
by `code` to mirror the original alphabetical-fallback behaviour.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: TUP-3066b: clear remaining references to dropped legacy tables in tests + types
Catches the CI fallout from PR #6787:
- Regenerate @tupaia/types (drops EntityHierarchy / EntityRelation /
EntityParentChildRelation from models.ts + schemas.ts)
- entity-server src/types.ts + __tests__/types.ts: drop EntityHierarchyModel /
EntityRelationModel from registries
- data-broker DhisService stubs: replace ENTITY_HIERARCHIES fixture with
PROJECTS stub so EventsPuller's models.project.findOne resolves
- Database tests:
- Entity/getDescendantsAncestorsProjectScoping.test.js: stop creating
entity_hierarchy + entity_hierarchy_id on project
- Central-server tests:
- apiV2/landingPages/utils.js: drop entity_hierarchy create
- apiV2/projects/CreateProject.test.js: drop the "creates a valid entity
hierarchy record" assertion + entityHierarchy cleanup
- apiV2/projects/EditProject.test.js, GETProjects.test.js: same cleanup
- apiV2/surveys/CreateAndEditSurveys.test.js: project fixture without
entity_hierarchy_id
- dhis/EventBuilder.test.js: ancestor_descendant_relation now keyed by
project_id; create project instead of entity_hierarchy
- Sync-server tests:
- CentralSyncManager.pull.test.ts: drop entityHierarchy from prepareData
- CentralSyncManager.syncLookup.test.ts: drop entity_parent_child_relation
insert; entity2's parent linked via entity.parent_id instead
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: TUP-3066b: include analytics types + clear leftover test references
CI surfaced three remaining issues after the first pass:
- @tupaia/types Validate: regenerate against a DB that has the analytics
materialized view, so `Analytics` / `AnalyticsCreate` / `AnalyticsUpdate`
interfaces are present alongside the dropped legacy table cleanup
- entity-server permissions.test.ts: remove the "filters hierarchies when
requested for some with access" case (route `/v1/hierarchies` is gone)
and the unused `getHierarchiesWithFields` import
- sync-server:
- CentralSyncManager.pull.test.ts: snapshotRecords expectation now 3
(country + userAccount + project) — entityHierarchy isn't seeded anymore
- CentralSyncManager.syncLookup.test.ts: drop the now-unused `entity2`
let-binding (the entity is still inserted, just not held for later use)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: TUP-3066b: match types regen against CI (analytics.type enum, entity_type order)
CI generates types against a freshly-built DB; my local dev DB has older
state and produced two drifts:
- Analytics{,Create,Update}.type: my dev DB has the analytics MV with column
type 'text'. CI's fresh MV has 'question_type', so the generated TS type is
QuestionType. Match CI by changing 'string | null' → 'QuestionType | null'
in the three Analytics interfaces.
- EntityType enum order: pg_enum.enumsortorder differs between my dev DB and
CI. Reorder document_group/document and tamanu_country to match CI's
insertion order.
schemas.ts uses alphabetical ordering and didn't need changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: TUP-3066b: match types schemas.ts against CI regen
Three more drifts caught by the Validate types CI step:
- ENTITY_TYPE_CHILDREN map (used by UserAccountPreferencesSchema):
document_group before document, and tamanu_country between ird_village
and srh_district — matches Postgres pg_enum.enumsortorder on CI
- Analytics{,Create,Update}Schema.type now includes a `QuestionType` enum
array (the analytics MV column is question_type on CI's fresh build).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: TUP-3066a: backport CI fixups (data-broker stub, EventBuilder fixture, types regen)
Same drifts surfaced on this PR as on TUP-3066b; smaller subset because
this branch only renames the column, doesn't drop the legacy tables:
- data-broker DhisService stubs: swap entityHierarchy → project to match
EventsPuller.ts's new lookup path
- central-server EventBuilder.test.js: ancestor_descendant_relation now
keyed by project_id (the column was renamed)
- @tupaia/types models.ts + schemas.ts:
- Analytics{,Create,Update}.type now QuestionType (analytics MV column
is question_type on CI's fresh build)
- EntityType enum order: document_group before document, tamanu_country
between ird_village and srh_district (matches pg_enum.enumsortorder)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Update index.js
* specs
* fix: TUP-3066b: restore review-fix changes lost in merge conflict
Merging epic-entity-hierarchy into tup-3066b-drop-legacy-tables dropped
five fixes that were correctly squash-merged via PRs #6778 and #6786.
Restored:
- surveyDataExport.fetchProjectId: return project.id (not entity_hierarchy_id)
- migration: delete orphan ancestor_descendant_relation rows before SET NOT NULL
- Entity.fetchDefaultProjectIdPatiently: dedupe project_id in SQL via DISTINCT
- Entity.findOneByCodeInProject: forward 4th options arg (was silently dropped)
- data-broker models: extract named Project type to match file convention
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: TUP-3066b: restore central-server project changes lost in merge
The recent merge of epic-entity-hierarchy reintroduced entity_hierarchy
references in CreateProject.js and the project test rollback helpers.
This was the same merge-conflict-lost-work pattern as commit ae6a9d8fc;
restoring from 55ddae54c (last known-good CI state):
- CreateProject.js: drop createEntityHierarchy method, its call site,
and the entity_hierarchy_id field on project create
- CreateProject.test.js: drop "creates a valid entity hierarchy record"
test and entityHierarchy.delete from rollbackRecords
- EditProject.test.js: drop entityHierarchy.delete from rollbackRecords
Fixes the 5 failing central-server tests on CI — the after-each hook
crash on the missing entityHierarchy model was cascading sinon stub
leakage into the next two test files.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Delete TUP-3066-refinement.md
* Update entity-hierarchy-improvements.md
* refactor: TUP-3156: delete MS1 sync
Andrew confirmed in the TUP-3156 refinement that MS1 is no longer used:
last response in 2023, prior in 2021. The sync is bare entity-code
lookups outside any request context — post-3056 these silently return
arbitrary copies of duplicated entities. Cleaner to remove than to
project-scope a dead path.
Deletes:
- packages/central-server/src/ms1/ directory (sync code)
- Ms1SyncLog + Ms1SyncQueue model wrappers (only callers were in /ms1/)
- startSyncWithMs1 call in index.js
Out of scope:
- DB tables ms1_sync_queue / ms1_sync_log stay (separate cleanup)
- types/EntityMetadata.ms1 field stays (no readers/writers, harmless)
- clearTestData entries stay (tables still exist, no-op cleanup is fine)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: TUP-3156: project-scope KoBo entity lookups + data-broker context
Sub-country entity codes are duplicated per project after TUP-3056, so
bare findOne({ code }) in the KoBo sync path silently returned an
arbitrary copy. KoBo always syncs one project at a time, so we can
thread the survey's project_id through.
- central-server/kobo/startSyncWithKoBo.js: fetch the survey from the
sync group's data_group_code, pass survey.project_id to dataBroker
pullSyncGroupResults, and use entity.findOneByCodeInProject in
writeKoboDataToTupaia.
- data-broker KoBoService: accept projectId in pullSyncGroupResults
options, forward to translator.
- data-broker KoBoTranslator: accept projectId, use …
…ed + email after timeout) (#6834) * feat(entityHierarchy): TUP-3181: speed up entity import (skip unchanged + email after timeout) Re-importing a whole exported sheet after editing a few rows was doing per-row work for every row, timing out for large projects. Two changes: - Skip unchanged rows: snapshot the project's entities once before the write transaction (loadExistingEntities) and skip any row that would change nothing (isEntityUnchanged) — no per-row queries for unchanged rows, so editing a few rows in a big sheet is fast. Conservative: anything not cheaply comparable counts as changed, so real updates are never skipped. - Email after timeout: wrap the entities import route in emailAfterTimeout (as survey-response import already does) so a genuinely large import (e.g. a new hierarchy) responds immediately and emails the result on completion instead of hanging until the request times out. The admin panel sends a 10s timeout. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(entityHierarchy): TUP-3181: address import-perf review feedback - isEntityUnchanged: treat blank (null) coordinate cells as "no coordinates" rather than Number(null)=0 — xlsx parses blanks to null (defval), so without this the skip never fired for coordinate-less entities (most of them). - loadExistingEntities: only load the countries being imported (AND country_code = ANY(?)), not the whole project, so a small single-country import doesn't pull the whole project into memory. - Keep the per-sheet duplicate-code check ahead of the skip, so accidental duplicate lines still error even when a row would be skipped. - Test cleanup so the skip-unchanged "create" assertion holds on repeat runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(entityHierarchy): TUP-3181: make skip-unchanged work for located entities An unchanged re-import of a project with coordinates still ran the full per-row path for every located entity, so it never sped up. Two causes: - The exporter writes full ST_X/ST_Y precision but the importer reads cells with raw:false, which rounds them — so exact coordinate equality flagged every located entity as changed. Compare coordinates rounded to ~6 dp (~0.1m). - Duplicate-code detection used an array .includes (O(n^2)); use a Set. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * tweak(admin-panel): TUP-3181: bump entity import email-timeout to 30s Give a large project's skip-unchanged import (xlsx parse + snapshot query) room to finish synchronously before falling back to the email-on-completion response. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
upsertEntities ran all entities_upserted through a single Promise.all, so a new parent and new child in the same submission were inserted concurrently and the child's parent_id FK could be evaluated before the parent row existed, failing with entity_parent_fk. Resolve all ids against the pre-batch DB state first (semantics unchanged), then upsert parent-before-child sequentially via orderEntitiesParentFirst. That topological pass also throws on an in-batch cycle, covering mutually-referential submissions the resolveCanonicalEntityForProject cycle guard never sees. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…true full deletion A code can have multiple entity rows post-epic (one per project), but MediTrak sees one row per code. Deleting a duplicate must not tell MediTrak to drop the entity, which still exists in another project. The sync queue stores no code/snapshot, so the decision is made at enqueue time (where the deleted row's code is available): MeditrakSyncRecordUpdater only enqueues an entity delete when no rows remain for that code. canonicalEntityFilter then passes entity deletes through, safe because the enqueuer pre-filters them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…yncRecordUpdater test The test cast getTestModels() to TestModelRegistry but passed it to MeditrakSyncRecordUpdater, which expects the sibling MeditrakAppServerModelRegistry, failing CI with TS2345. The earlier build-only `tsc` run predated this test file, so it slipped through locally; ts-jest type-checks the file. Cast the constructor arg to the expected registry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nel countries Narrow the admin-panel entities listing so it matches the entity export: GETEntities now filters by getEntitiesAllowed(TUPAIA_ADMIN_PANEL_PERMISSION_GROUP) instead of all-access getEntitiesAllowed(). A non-BES user now only sees and exports entities for countries they have Tupaia Admin Panel access to (Option B from the QA thread). BES Admins are unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…paia into epic-entity-hierarchy
… from a project The project-scope mutation guard rejected country entities because they have project_id NULL, returning a 404 on save. Make ownership project-aware so a country in the active project's project_country set is editable, mirroring the list filter. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue #:
Changes:
🦸 Review Hero