Dataset endpoint update: Nationally Significant Infrastructure Projects

[drewhardy77]

Guidance

Current dataset is an artefact of scraping from 3 years ago and is not updated. This new endpoint is regularly updated and has significantly more data. ALIGN are asking for this as a data feed from the platform

Current dataset page: https://www.planning.data.gov.uk/dataset/infrastructure-project

Endpoints

Spatial data download: https://national-infrastructure-consenting.planninginspectorate.gov.uk/projects-map/download-boundaries (geojson download)

Data attribute download: https://national-infrastructure-consenting.planninginspectorate.gov.uk/api/applications-download (csv download)

Data joining

Spatial data (filename: all-project-boundaries) maps to data attribute data (filename: applications) on field all-project-boundaries.case-reference = application.Project reference. No other data apart from the geometry needs to be extracted from the all-project-boundaries file.

Attribute mapping

csv header	current field
Project reference	reference
Project name	name
Date of application	start-date
Application type	infrastructure-project-type*
Stage	infrastructure-project-decision**
No match***	documentation-url

*will need us to update the Infrastructure Project type to use these values in the name field
BC03 - An Industrial Process or Processes
BC04 - Storage or Distribution of Goods
EN01 - Generating Stations
EN02 - Electric Lines
EN03 - Underground Gas Storage Facilities
EN06 - Gas Transporter Pipe-lines
EN07 - Other Pipe-lines
TR01 - Highways
TR02 - Airports
TR03 - Harbour Facilities
TR04 - Railways
TR05 - Rail Freight Interchanges
WA01 - Dams and Reservoirs
WA02 - Transfer of Water Resources
WS01 - Hazardous Waste Facilities
WW01 - Waste Water Treatment Plants

** Update allowable values in https://dataset-editor.development.planning.data.gov.uk/dataset/infrastructure-project-decision.csv

Pre-application
Examination
Post-decision
Withdrawn
Decision
Recommendation
Pre-examination

*** documentation-url can be easily constructed from the data as the website follows a standard formula.
https://national-infrastructure-consenting.planninginspectorate.gov.uk/projects/[reference]
Example: https://national-infrastructure-consenting.planninginspectorate.gov.uk/projects/EN010115

Additional fields which would be useful but not currently mapped

csv header	proposed header	notes
Applicant name	applicant	Organisation behind the application. Not PII as always corporate body
Location	location	description of the land associated with the application
Description	description	Description of the project
Date Examination started	examination-started-date
Date Examination closed	examination-closed-date
Date of decision	decision-date
Date withdrawn	withdrawn-date

[Swati-Dash]

@drewhardy77 do you mean, you need to update the spec with extra fields? if so, can you pls do that before we add this data. Also you can update the project-type using dataset editor?

[Swati-Dash]

@drewhardy77 I think we should add this with the current spec for now. Can you pls create another ticket to update spec and map the new fields? We will at least have up-to-date data for this dataset on our platform and improve it later. Pls confirm what you agreed with psd

cc: @gmatchett

[drewhardy77]

@drewhardy77 I think we should add this with the current spec for now. Can you pls create another ticket to update spec and map the new fields? We will at least have up-to-date data for this dataset on our platform and improve it later. Pls confirm what you agreed with psd

cc: @gmatchett

Yes I can do that. Are you proposing just updating the endpoint and nothing else (just so I can understand the scope of the new ticket)? Worth noting that if we don't do anything at this stage to update the allowable fields for

• infrastructure-project-type
• infrastructure-project-decision
  then most of the entries will revert to NULL values. Also, the documentation-url will also be NULL if we don't create the url using the concatenation method outlined.

I'm happy to add all these in to the new ticket, just wanted to be explicit about what is and isn't included in this initial piece of work

[Swati-Dash]

@drewhardy77 this will added and columns will be mapped to the existing fields in the spec. If the field you have mentioned already exists then we can add values by concatenating.
For your new ticket the scope should be to update the spec with new fields and send us back with col mapping. Thanks