Add support for selected entities and redirects in add-data workflow

[gibahjoe]

What type of PR is this? (check all applicable)

• Refactor
• Feature
• Bug Fix
• Optimization
• Documentation Update

Description

Adds selected entity handling to the add-data pipeline. Selected entities are prioritised for entity number assignment, new-entities reports only selected rows, and all-entities reports the full set of newly assigned rows.

Also adds old-entity redirect generation from selected redirects and duplicate candidates. Complete duplicate matches and single matches above 85% are automatically redirected, with duplicate evidence included in old-entity.notes.

Related Tickets & Documents

• Ticket Link
• Related Issue #
• Closes #

QA Instructions, Screenshots, Recordings

Run the focused pipeline tests:

cd request-processor
./venv/bin/python -m pytest tests/unit/src/application/core/test_pipeline.py::test_fetch_add_data_response_includes_selected_old_entity_redirects tests/unit/src/application/core/test_pipeline.py::test_create_old_entity_redirects_from_selected_redirects tests/unit/src/application/core/test_pipeline.py::test_create_auto_old_entity_redirects_for_complete_matches tests/unit/src/application/core/test_pipeline.py::test_create_auto_old_entity_redirects_for_single_matches_above_85_percent tests/unit/src/application/core/test_pipeline.py::test_create_auto_old_entity_redirects_ignores_existing_redirects_and_unselected_entities -q

Run formatting check:

cd request-processor
./venv/bin/python -m black --check src/application/core/pipeline.py tests/unit/src/application/core/test_pipeline.py

Added/updated tests?

• Yes
• No, and this is why:
• I need help with writing tests

[optional] Are there any post deployment tasks we need to perform?

None.

[optional] Are there any dependencies on other PRs or Work?

None known.

Summary by CodeRabbit

• New Features
  • Added add-data inputs to exclude specific references from entity selection and redirect generation.
  • Added support for explicitly selecting redirects, alongside automatic redirects for likely duplicates.
  • Enhanced add-data responses to include selected new entities and the generated old-entity redirect rows.
• Bug Fixes
  • Improved behaviour when selection inputs are missing, empty, null, or include invalid/unassigned redirect data.
  • Ensured redirect results are de-duplicated and excluded selections are respected.
• Tests
  • Added unit tests covering selection filtering, assignment prioritisation, and redirect generation scenarios.

[coderabbitai]

Warning

Review limit reached

@gibahjoe, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 49 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 10ebb477-a375-409a-ba67-00c94d903541

📥 Commits

Reviewing files that changed from the base of the PR and between 6f8b97c and 7432667.

📒 Files selected for processing (1)

• request-processor/src/application/core/pipeline.py

Walkthrough

Changes

Add-data selection and redirects

Layer / File(s)	Summary
Selection request contract and wiring request_model/schemas.py, request-processor/src/application/core/workflow.py, request-processor/src/tasks.py, request-processor/tests/unit/src/application/core/test_workflow.py, request-processor/tests/unit/src/test_tasks.py	Add-data requests accept optional excluded references and selected redirects, which are forwarded through the task and workflow layers.
Selected entity assignment and transformation request-processor/src/application/core/pipeline.py, request-processor/tests/unit/src/application/core/test_pipeline.py	Excluded references are assigned after non-excluded entries and used to filter entity-organisation and new-entity summaries.
Explicit and automatic redirect generation request-processor/src/application/core/pipeline.py, request-processor/tests/unit/src/application/core/test_pipeline.py	Redirect helpers support explicit selections, duplicate matching, similarity thresholds, evidence notes, validation, dates, and de-duplication.
Selection-aware pipeline response request-processor/src/application/core/pipeline.py, request-processor/tests/unit/src/application/core/test_pipeline.py	The response includes filtered entity breakdowns and merged old-entity redirect rows.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant add_data_task
  participant add_data_workflow
  participant fetch_add_data_response
  participant RedirectHelpers
  Client->>add_data_task: Submit excluded_references and selected_redirects
  add_data_task->>add_data_workflow: Forward selection parameters
  add_data_workflow->>fetch_add_data_response: Request selection-aware response
  fetch_add_data_response->>RedirectHelpers: Build and merge old-entity redirects
  RedirectHelpers-->>fetch_add_data_response: Return redirect rows
  fetch_add_data_response-->>add_data_workflow: Return pipeline summary
  add_data_workflow-->>add_data_task: Return add-data response

Loading

Possibly related PRs

• digital-land/async-request-backend#123: Adds duplicate candidate detection used by this PR’s automatic old-entity redirect generation.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: adding support for selected entities and redirects in the add-data workflow.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch 2661-complex-batch-assign---part-3-choose-which-rows-to-assign-entities-to

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

request-processor/src/application/core/pipeline.py (1)

82-85: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Remove unused selected_entity_ids.

selected_entity_ids is computed here but never referenced afterwards in _create_old_entity_redirects - the exclusion is already enforced via entity_by_reference, which is built from the same selected_new_entities. Flake8 flags this as F841.

🧹 Proposed fix

     selected_new_entities = _filter_selected_entities(new_entities, excluded_references)
-    selected_entity_ids = {
-        str(entity.get("entity", "")).strip() for entity in selected_new_entities
-    }
 
     old_entity_rows = []

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@request-processor/src/application/core/pipeline.py` around lines 82 - 85,
Remove the unused selected_entity_ids computation from
_create_old_entity_redirects, leaving selected_new_entities and the existing
entity_by_reference-based exclusion logic unchanged.

Source: Linters/SAST tools

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@request-processor/src/application/core/pipeline.py`:
- Around line 82-85: Remove the unused selected_entity_ids computation from
_create_old_entity_redirects, leaving selected_new_entities and the existing
entity_by_reference-based exclusion logic unchanged.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 905c7db7-7f44-4b74-9a4a-1b9aba82edcb

📥 Commits

Reviewing files that changed from the base of the PR and between debe04e and 9d2bbaa.

📒 Files selected for processing (7)

• request-processor/src/application/core/pipeline.py
• request-processor/src/application/core/workflow.py
• request-processor/src/tasks.py
• request-processor/tests/unit/src/application/core/test_pipeline.py
• request-processor/tests/unit/src/application/core/test_workflow.py
• request-processor/tests/unit/src/test_tasks.py
• request_model/schemas.py

🚧 Files skipped from review as they are similar to previous changes (1)

• request-processor/tests/unit/src/test_tasks.py

[pooleycodes]

LGTM