Add missing endpoints for Local Plan datasets in batches

[Swati-Dash]

Background

Review the Local Plan dataset tracker and identify Local Planning Authorities (LPAs) that have not yet provided data for the planning datasets. Create endpoints for these LPAs in batches to improve coverage and support onboarding.
https://mhclg-my.sharepoint.com/:x:/g/personal/shannon_franklin_communities_gov_uk/IQDrNNxB4k6tT71Ky9GgG8lHAcd_kEHy0vtWd4HPAgJokkE?e=c0nXNz
Scope
Review the Local Plan tracker and identify:

• LPAs that have not yet provided data.
• The relevant planning datasets for each LPA.
• Any existing endpoints to avoid duplication.

Create endpoints in batches for the identified LPAs.

• Group endpoint creation into manageable batches.
• Ensure endpoint metadata follows the standard conventions.

Validate the newly created endpoints:

• Endpoint URL is accessible.
• Dataset and organisation are correctly assigned.
• Collection/pipeline (where applicable) is correctly configured.

Update the tracker to record:

• Endpoints added
• Any LPAs that require follow-up because a source URL and endpoint URL could not be identified or added

Acceptance Criteria

• All LPAs missing Local Plan dataset endpoints have been identified.
• Endpoints have been created in batches for eligible LPAs.
• No duplicate endpoints have been created.
• All new endpoints pass validation checks.
• The tracker is updated to show completed and outstanding LPAs.

Notes

• Prioritise LPAs where a suitable source URL is already available.
• Record any LPAs requiring manual investigation separately.

[sianteesdale]

The Plan Making team have been keeping track of the LPAs who have given notice to commence their new emerging local plans. They have created, and is continually updating, a spreadsheet of this progress.

We have created a copy of the spreadsheet, available on Sharepoint here: Plan timetable checks 300726.xlsx

Currently, we have 72 plan timetable datasets on our platform. This spreadsheet has identified there could be an additional 47 LPAs that need data adding. Some of them have been marked as 'no csv found', which requires more investigation on the LPA websites to see if a CSV version can be found.

We are provisionally planning to split data adding between myself, @Swati-Dash and @gmatchett

[Swati-Dash]

@sianteesdale as the LPAs that have not submitted these datasets would not lets is know if their links have changed, do you think we can use AI to flag us if links change from these doc-url pages?

[gmatchett]

@sianteesdale @Swati-Dash that's a really good point. Swati made. We are short cutting the process which means LPAs may not see the need to engage with us when changes happen. Given the plan making team are going through the process of getting this information, would it be more useful for us to get them to make sure the LPA provides the data via the correct method? I mean its not fair that the LPAs who have provided the data ion thee way which we can track and raise queries on where necessary.

[Swati-Dash]

100% @gmatchett, we should also raise this issue to amend the legislation which right now doesn't ask LPAs to submit this datasets