Skip to content

Refactor Assign Entities workflow and update related documentation - #174

Merged
gibahjoe merged 6 commits into
mainfrom
2661-complex-batch-assign---part-3-choose-which-rows-to-assign-entities-to
Jul 23, 2026
Merged

Refactor Assign Entities workflow and update related documentation#174
gibahjoe merged 6 commits into
mainfrom
2661-complex-batch-assign---part-3-choose-which-rows-to-assign-entities-to

Conversation

@gibahjoe

@gibahjoe gibahjoe commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

What type of PR is this? (check all applicable)

  • Refactor
  • Feature
  • Bug Fix
  • Optimization
  • Documentation Update

Description

Adds selection support to the Assign Entities flow so users can choose which new entity references should be processed before continuing to preview.

This updates the Assign Entities results page to support entity checkboxes, Dedup selection changes, replacement async requests using selected_entities and selected_redirects, and preview rendering from async-generated new-entities and old-entity output. It also removes the old GitHub entity_redirects dispatch path, since redirects now come from the completed async response.

Documentation has been added for the Assign Entities architecture, gotchas, and debugging steps.

Related Tickets & Documents

  • Related Issue #2661
  • Docs: docs/assign-entities/architecture.md

QA Instructions, Screenshots, Recordings

Test the Assign Entities journey:

  1. Start an Assign Entities request for a resource with unknown entities.
  2. On the Entities tab, confirm new rows have enabled checkboxes and existing/platform rows are disabled.
  3. Change the entity selection and confirm the button changes to “Process entities”.
  4. If Dedup rows are present, change a Dedup selection and confirm the button also changes to “Process entities”.
  5. Process the selection and confirm the replacement request keeps unselected entities visible while placing selected/processed entities in the async output.
  6. Continue to preview and confirm:
    • “Rows that will create new entities” matches pipeline-summary.new-entities
    • “Number of redirects” matches pipeline-summary.old-entity
  7. Confirm the GitHub workflow payload no longer includes entity_redirects.

Added/updated tests?

  • Yes
  • No, and this is why: please replace this line with details on why tests have not been included
  • I need help with writing tests

Focused validation run:

./.venv/bin/pytest tests/unit/blueprints/datamanager/controllers/test_add.py tests/unit/blueprints/datamanager/services/test_github.py tests/unit/blueprints/datamanager/controllers/test_transform.py tests/acceptance/blueprints/datamanager/test_flagged_resources.py -q
./.venv/bin/black --check application/blueprints/datamanager/router.py application/blueprints/datamanager/controllers/transform.py application/blueprints/datamanager/controllers/preview.py application/blueprints/datamanager/services/github.py tests/unit/blueprints/datamanager/controllers/test_add.py tests/unit/blueprints/datamanager/services/test_github.py tests/unit/blueprints/datamanager/controllers/test_transform.py tests/acceptance/blueprints/datamanager/test_flagged_resources.py
./.venv/bin/flake8 application/blueprints/datamanager/router.py application/blueprints/datamanager/controllers/transform.py application/blueprints/datamanager/controllers/preview.py application/blueprints/datamanager/services/github.py tests/unit/blueprints/datamanager/controllers/test_add.py tests/unit/blueprints/datamanager/services/test_github.py tests/unit/blueprints/datamanager/controllers/test_transform.py tests/acceptance/blueprints/datamanager/test_flagged_resources.py

[optional] Are there any post deployment tasks we need to perform?

No.

[optional] Are there any dependencies on other PRs or Work?

Depends on async support for selected_entities, selected_redirects, pipeline-summary.all-entities, and pipeline-summary.old-entity.

Summary by CodeRabbit

  • New Features
    • Added entity assignment selection controls to the Assign Entities workflow, including select-all, per-row checkboxes, and selection state updates in the UI.
    • Enabled duplicate redirection selection and locking in the flagged resource “Assign Entities” flow, with selections submitted to the async process.
  • Bug Fixes
    • Improved the old-entity redirect preview (including accurate “Number of redirects” and clearer handling of empty values).
  • Documentation
    • Added Assign Entities end-to-end architecture documentation and updated related guidance.
  • Tests
    • Expanded unit and acceptance coverage for the updated UI, selection state, and async submission behaviour.

- Added detailed architecture documentation for the Assign Entities process, outlining the flow from user input to async processing and GitHub commit.
- Updated existing documentation to reference the new Assign Entities architecture.
- Modified the entities preview template to ensure consistent display of entity organisation CSV and old entity summary.
- Enhanced acceptance tests to validate the correct handling of selected entities and redirects in the Assign Entities flow.
- Adjusted unit tests to reflect changes in the handling of entity redirects, ensuring they are not passed to the GitHub workflow.
- Improved logic in the duplicate candidate preparation to correctly manage auto-selection and locking of redirects based on entity selection.
@coderabbitai

coderabbitai Bot commented Jul 20, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: b25e7cb1-5281-4d6a-a281-2037ac0f1819

📥 Commits

Reviewing files that changed from the base of the PR and between 67d8f15 and 064b494.

📒 Files selected for processing (13)
  • application/blueprints/datamanager/controllers/flagged_resources.py
  • application/blueprints/datamanager/controllers/preview.py
  • application/blueprints/datamanager/controllers/transform.py
  • application/blueprints/datamanager/router.py
  • application/db/models.py
  • application/templates/components/check-transform-base.html
  • application/templates/datamanager/assign-entities-check-results.html
  • application/templates/datamanager/entities_preview.html
  • docs/assign-entities/architecture.md
  • migrations/versions/7b8c9d0e1f2a_drop_request_meta_entity_redirects.py
  • tests/acceptance/blueprints/datamanager/test_flagged_resources.py
  • tests/unit/blueprints/datamanager/controllers/test_add.py
  • tests/unit/blueprints/datamanager/controllers/test_transform.py
💤 Files with no reviewable changes (1)
  • application/db/models.py
🚧 Files skipped from review as they are similar to previous changes (4)
  • application/templates/components/check-transform-base.html
  • tests/unit/blueprints/datamanager/controllers/test_add.py
  • application/templates/datamanager/assign-entities-check-results.html
  • tests/unit/blueprints/datamanager/controllers/test_transform.py

Walkthrough

The Assign Entities flow now supports selecting entities and redirects across paginated results, validating and resubmitting changed selections asynchronously, deriving preview data from pipeline-summary, and removing redirects from the GitHub workflow payload.

Changes

Assign Entities workflow

Layer / File(s) Summary
Selection-aware transform and contracts
application/blueprints/datamanager/controllers/flagged_resources.py, application/blueprints/datamanager/controllers/transform.py, tests/unit/blueprints/datamanager/controllers/test_transform.py
Entity and redirect candidates now carry selection, locking, and pagination metadata; _prepare_duplicate_candidates computes auto-select and redirect-locked states from async old-entity rows; pagination supports optional selection decoration.
Validated async resubmission
application/blueprints/datamanager/router.py, tests/acceptance/blueprints/datamanager/test_flagged_resources.py
Flagged-resource POST handler validates and merges selections across visible pages, filters redirects by exclusion, submits replacement async requests with excluded_references and selected_redirects, and handles resubmission errors.
Entity and redirect selection UI
application/templates/components/check-transform-base.html, application/templates/datamanager/assign-entities-check-results.html
Check-transform and duplicate-redirection templates render entity selection checkboxes, select-all controls, redirect locking, selection counts, and synchronised client-side checkbox behaviour coordinating entity and redirect availability.
Preview data from pipeline-summary
application/blueprints/datamanager/controllers/preview.py, tests/unit/blueprints/datamanager/controllers/test_preview.py
Preview controller sources old-entity rows and redirect counts from async pipeline-summary instead of persisted RequestMeta, constructs redirect tables with null-value coercion, and calculates excluded-reference counts for preview context.
Preview UI and database schema updates
application/templates/datamanager/entities_preview.html, application/db/models.py, migrations/versions/...
Preview template adds excluded-count display, old-entity redirect count header, and reorganised entity-organisation section headings; RequestMeta model replaces entity_redirects column with endpoints_to_retire and migration removes the old column.
GitHub workflow payload
application/blueprints/datamanager/services/github.py, tests/unit/blueprints/datamanager/services/test_github.py, tests/unit/blueprints/datamanager/controllers/test_add.py
GitHub service no longer accepts or forwards entity_redirects parameter to workflow dispatch; tests verify entity_redirects is absent from payload.
Workflow and integration documentation
docs/assign-entities/architecture.md, docs/datamanager/architecture.md, docs/datamanager/github-add.md
New Assign Entities architecture document describes full flow, async ownership, selection semantics, redirect locking rules, pagination constraints, and debugging guidance; related datamanager and GitHub docs updated with pipeline-summary.old-entity contract.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant User
  participant CheckTransformUI
  participant flagged_resource_detail_post
  participant pipeline-summary
  participant AsyncRequest
  User->>CheckTransformUI: select/deselect entities and redirects
  CheckTransformUI->>flagged_resource_detail_post: submit selection state
  flagged_resource_detail_post->>pipeline-summary: fetch current candidates
  pipeline-summary-->>flagged_resource_detail_post: return entity and redirect rows
  flagged_resource_detail_post->>AsyncRequest: submit validated excluded/selected
  AsyncRequest-->>CheckTransformUI: return replacement request id
Loading

Possibly related PRs

Suggested reviewers: pooleycodes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title is broad but accurately reflects the Assign Entities workflow refactor and documentation updates.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch 2661-complex-batch-assign---part-3-choose-which-rows-to-assign-entities-to

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
application/templates/datamanager/entities_preview.html (1)

205-214: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Prefer GovUK utility classes over inline styles.

To align with GovUK styling conventions, consider replacing the inline styles with the appropriate utility classes, such as govuk-!-padding-3, govuk-!-margin-bottom-5, and a background colour class (or wrapping the content in a standard component like an inset text or panel).

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@application/templates/datamanager/entities_preview.html` around lines 205 -
214, Replace the inline styling on the redirect-count container around the
“Number of redirects” summary row with GOV.UK utility classes, using the
appropriate padding, margin-bottom, and background-colour/component classes
while preserving the existing layout and content.
application/blueprints/datamanager/controllers/preview.py (1)

121-137: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value

Prevent rendering "None" for null JSON values.

If the async API returns null for any of these fields (e.g., {"status": null}), old_entity.get("status", "") will evaluate to None, which str() will convert to the string "None".

Consider appending or "" (and removing the fallback from get()) to ensure None values are coerced to empty strings before string conversion.

♻️ Proposed fix
         row = {
             "old-entity": str(
-                old_entity.get("old-entity", "") or old_entity.get("old_entity", "")
+                old_entity.get("old-entity") or old_entity.get("old_entity") or ""
             ),
-            "status": str(old_entity.get("status", "")),
-            "entity": str(old_entity.get("entity", "")),
-            "notes": str(old_entity.get("notes", "")),
+            "status": str(old_entity.get("status") or ""),
+            "entity": str(old_entity.get("entity") or ""),
+            "notes": str(old_entity.get("notes") or ""),
             "end-date": str(
-                old_entity.get("end-date", "") or old_entity.get("end_date", "")
+                old_entity.get("end-date") or old_entity.get("end_date") or ""
             ),
             "entry-date": str(
-                old_entity.get("entry-date", "") or old_entity.get("entry_date", "")
+                old_entity.get("entry-date") or old_entity.get("entry_date") or ""
             ),
             "start-date": str(
-                old_entity.get("start-date", "") or old_entity.get("start_date", "")
+                old_entity.get("start-date") or old_entity.get("start_date") or ""
             ),
         }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@application/blueprints/datamanager/controllers/preview.py` around lines 121 -
137, Update the row construction in the preview controller so every old_entity
field uses an empty-string fallback before str() conversion, including status,
entity, and notes, while preserving the existing hyphenated/underscored key
fallback behavior for old-entity and date fields. This must render null JSON
values as empty strings rather than "None".
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@application/blueprints/datamanager/controllers/transform.py`:
- Around line 250-254: The existing-row deduplication in _build_entities_data is
ineffective because organisation is removed before
_entity_candidate_reference_key is evaluated. Preserve organisation/reference
data when deriving existing_reference_keys, and ensure candidates with matching
organisation/reference are skipped even when their entity value differs or is
blank, while retaining same-loop deduplication.

---

Nitpick comments:
In `@application/blueprints/datamanager/controllers/preview.py`:
- Around line 121-137: Update the row construction in the preview controller so
every old_entity field uses an empty-string fallback before str() conversion,
including status, entity, and notes, while preserving the existing
hyphenated/underscored key fallback behavior for old-entity and date fields.
This must render null JSON values as empty strings rather than "None".

In `@application/templates/datamanager/entities_preview.html`:
- Around line 205-214: Replace the inline styling on the redirect-count
container around the “Number of redirects” summary row with GOV.UK utility
classes, using the appropriate padding, margin-bottom, and
background-colour/component classes while preserving the existing layout and
content.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 1a0bc287-138b-48e6-817d-3a918fd10648

📥 Commits

Reviewing files that changed from the base of the PR and between 49204b0 and 1a5283d.

📒 Files selected for processing (15)
  • application/blueprints/datamanager/controllers/flagged_resources.py
  • application/blueprints/datamanager/controllers/preview.py
  • application/blueprints/datamanager/controllers/transform.py
  • application/blueprints/datamanager/router.py
  • application/blueprints/datamanager/services/github.py
  • application/templates/components/check-transform-base.html
  • application/templates/datamanager/assign-entities-check-results.html
  • application/templates/datamanager/entities_preview.html
  • docs/assign-entities/architecture.md
  • docs/datamanager/architecture.md
  • docs/datamanager/github-add.md
  • tests/acceptance/blueprints/datamanager/test_flagged_resources.py
  • tests/unit/blueprints/datamanager/controllers/test_add.py
  • tests/unit/blueprints/datamanager/controllers/test_transform.py
  • tests/unit/blueprints/datamanager/services/test_github.py
💤 Files with no reviewable changes (1)
  • application/blueprints/datamanager/services/github.py

Comment thread application/blueprints/datamanager/controllers/transform.py Outdated
@gibahjoe

Copy link
Copy Markdown
Contributor Author

@coderabbitai

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

@gibahjoe Hi! How can I help with this PR? I can review the changes, investigate a specific area, or answer questions.

@gibahjoe

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

…selected entities and update related templates and tests

@pooleycodes pooleycodes left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, maybe just need to remove the RequestMeta.entity_redirects db column and writing to it as i don't think needed anymore?

…ate architecture documentation, and adjust tests accordingly

@pooleycodes pooleycodes left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gibahjoe
gibahjoe merged commit bb527b3 into main Jul 23, 2026
6 checks passed
@gibahjoe
gibahjoe deleted the 2661-complex-batch-assign---part-3-choose-which-rows-to-assign-entities-to branch July 23, 2026 07:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Complex Batch Assign - Part 3: Choose which rows to assign entities to

2 participants