Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ those documents into a versioned database that downstream users can query as a C
workbook, or GeoPackage.

.. image:: https://raw.githubusercontent.com/NatLabRockies/COMPASS/main/docs/source/_static/overview.png
:alt: High-level overview of the INFRA-COMPASS pipeline
:alt: High-level overview of the COMPASS pipeline
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we dropping the INFRA part of the name?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, just wanted to be a bit more consistency in the README with the logo

:align: center
:width: 100%

Expand All @@ -64,17 +64,18 @@ architecture around the LLM call:
jurisdictions.


Read more about the tool in the `documentation <https://natlabrockies.github.io/COMPASS/misc/about.html>`_.
Read more about the tool in the `documentation <https://natlabrockies.github.io/COMPASS/misc/about.html>`_,
or check out summaries of the validation results in the `validation report <https://natlabrockies.github.io/COMPASS/val/validation.html>`_.


Where is the extracted ordinance data?
======================================
The National Laboratories of the Rockies (NLR) typically runs the INFRA-COMPASS pipeline
The National Laboratories of the Rockies (NLR) typically runs the COMPASS pipeline
annually and publishes refreshed datasets to OpenEI. The latest published ordinance datasets
are available here:

- Solar: https://data.openei.org/submissions/8519
- Wind: https://data.openei.org/submissions/8602
- Solar: https://data.openei.org/submissions/8602
- Wind: https://data.openei.org/submissions/8519


Installing COMPASS
Expand Down
4 changes: 4 additions & 0 deletions docs/source/misc/about.rst
Original file line number Diff line number Diff line change
Expand Up @@ -162,3 +162,7 @@ See the development guides for full details:
`plugin development <https://natlabrockies.github.io/COMPASS/dev/plugin_development.html>`_
and
`advanced plugin development <https://natlabrockies.github.io/COMPASS/dev/advanced_plugin_development.html>`_.

Validation
==========
Take a look at some high-level validation results in the `validation report <https://natlabrockies.github.io/COMPASS/val/validation.html>`_.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/source/val/solar_val_aug_2025.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
91 changes: 91 additions & 0 deletions docs/source/val/validation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,95 @@ Here we give a brief overview of the results of known COMPASS validation efforts

COMPASS validation is an ongoing effort, and we will update this page as new results become available.

Geothermal Electricity Document Collection Validation (May 2026)
----------------------------------------------------------------
This validation was for the document collection portion only.

Info
^^^^

- **COMPASS Version**: `v0.15.2 <https://github.com/NatLabRockies/COMPASS/releases/tag/v0.15.2>`_
- **Number of Documents**: 100 (Assuming 10,000 jurisdictions, there is a 95% chance that the metrics are within ±9.75% of the reported value)
- **Features**: None
- **Procedure Validated**: Document collection from web
- **LLM(s) used**: OpenAI GPT-4.1, OpenAI GPT-5


Results
^^^^^^^

.. image:: geothermal_electricity_web_scraping_may_2026.png


Wind Document Collection Validation (September 2026)
----------------------------------------------------
This validation was for the document collection portion only.

Info
^^^^

- **COMPASS Version**: `v0.8.2 <https://github.com/NatLabRockies/COMPASS/releases/tag/v0.8.2>`_
- **Number of Documents**: 100 (Assuming 10,000 jurisdictions, there is a 95% chance that the metrics are within ±9.75% of the reported value)
- **Features**: None
- **Procedure Validated**: Document collection from web
- **LLM(s) used**: OpenAI GPT-4o-mini, OpenAI GPT-4.1-nano, OpenAI GPT-4.1-mini, OpenAI GPT-4.1


Results
^^^^^^^

.. image:: wind_web_scraping_september_2025.png


Solar Validation (August 2025)
------------------------------

This validation was for the ordinance value extraction portion only (assume documents are correct and belong
to the correct jurisdiction).

This validation focused on the model ability to extract structured ordinance data from unstructured wind ordinance text documents.


Info
^^^^

- **COMPASS Version**: `v0.7.0 <https://github.com/NatLabRockies/COMPASS/releases/tag/v0.7.0>`_
- **Number of Documents**: 78 (Assuming 10,000 jurisdictions, there is a 95% chance that the metrics are within ±11.05% of the reported value)
- **Features**:

- structures (participating)
- property line (participating)
- structures (non-participating)
- property line (non-participating)
- roads
- railroads
- transmission
- water
- noise
- maximum height
- maximum project size
- minimum lot size
- maximum lot size
- density
- coverage
- prohibitions
- decommissioning
- glare
- visual impact
- primary use districts
- special use districts
- accessory use districts
- ordinance effective year


- **Procedure Validated**: Ordinance extraction from documents
- **LLM(s) used**: OpenAI GPT-4.1-nano, OpenAI GPT-4.1-mini, OpenAI GPT-4.1

Results
^^^^^^^

.. image:: solar_val_aug_2025.png


Mini Wind Validation (June 2025)
--------------------------------
Expand Down Expand Up @@ -104,3 +193,5 @@ Results
^^^^^^^

.. image:: wind_jan_2024.png

.. Margin of error calculator: https://www.calculator.net/sample-size-calculator.html
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
326 changes: 326 additions & 0 deletions support/validation/May_2026_Geothermal_webscraping_validation.ipynb

Large diffs are not rendered by default.

File renamed without changes.