Conversation
…iers are now logged.
[Security] Removed logging of direct user inputs
…hts logging setup
…mplate Update appsettings deploy template
Ensure all "not found" (404) responses in DataController, MetadataController, and TablesController are logged via the audit logging service, using a placeholder in the logging scope. Update LoggerConsts with a NOT_FOUND_PLACEHOLDER. Add and extend unit tests to verify audit logging occurs for missing database/table scenarios in GET, POST, HEAD, and OPTIONS endpoints. Remove redundant audit log calls and ensure consistent audit behavior across endpoints.
[Fix] Add audit logging for 404s in controllers and tests
…ler and DatabaseListingExample
…endpoint-paths [Feature] Add headers to endpoint paths
Introduced /health and /info endpoints with HealthController and InfoController, including response models and unit tests. The /health endpoint checks all database connections and reports their status; /info returns app metadata. Removed fail-fast database connection validation from startup—health is now checked via the endpoint. Updated test conventions and added XML docs for new code.
Replaced Assert.Multiple with Assert.EnterMultipleScope (using statement) in InfoControllerTests to align with updated NUnit best practices for grouping multiple assertions.
Introduce CheckConnectionAsync to IDataBaseConnector and all implementations, enabling lightweight connectivity checks for health endpoints. Update HealthController to use this method instead of GetAllFilesAsync, rename the action to GetHealthAsync, and update related tests. Apply [ApiKeyAuth] to InfoController and HealthController. Improve service registration in tests and enhance documentation.
Added assertion in InfoController test to ensure version is not "unknown". Updated FileShareDataBaseConnector to use GetPropertiesAsync for a more robust connection check.
Explicitly rethrow OperationCanceledException in HealthController to ensure proper cancellation propagation. Added a unit test to verify that GetHealthAsync throws when cancellation is requested.
[Feature] Add health and info endpoints; remove startup DB validation
- Add global OperationCancelledExceptionFilter to return 499 on OperationCanceledException/TaskCanceledException - Add ClientClosedRequestOperationFilter to document 499 in OpenAPI for endpoints with CancellationToken - Add CancellationToken support to all relevant controller actions and data/cache calls - Remove catch-all 500 error handling; let exceptions propagate - Update XML docs and remove 500 response codes where not needed - Add/expand unit tests for cancellation and error propagation
- Updated all references from OperationCancelledExceptionFilter to OperationCanceledExceptionFilter to match .NET conventions. - Added security guidelines for user input validation and sanitization to copilot-instructions.md. - User input no longer logged in OperationCanceledExceptionFilter
Introduce GlobalSuppressions.cs to suppress S3267 and CA1873 rules with project-level justifications. Refactor test helper methods in ClientClosedRequestOperationFilterTests.cs to be private static and suppress unused parameter warnings. Update reflection logic to use correct binding flags for private static methods.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
[Feature] TableSummary "geo" property
…/dot-github/skills/openapi-validation/npm_and_yarn-053c9c4054 Bump fast-uri from 3.1.0 to 3.1.2 in /.github/skills/openapi-validation in the npm_and_yarn group across 1 directory
…-tablesummary Revert geo field from table summary
…t for content types
… for non-ASCII dimension codes
…ging [Feature] Information level logging for controller responses
…-transformer Dimension codes are no longer simplified automatically
…sion [Dependency] Update package references in project files to latest versions
sakari-malkki
previously approved these changes
Jun 5, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
This “Release 0.5.9” PR expands PxApi’s operational surface and metadata capabilities by adding health/info endpoints, introducing Elasticsearch-backed search behind a feature flag, standardizing structured logging scopes, and updating several API routes and OpenAPI filters accordingly.
Changes:
- Add Elasticsearch search plumbing (models, service abstraction, config, feature flag) plus logging/input-sanitization utilities.
- Add operational endpoints and reliability improvements (health/info controllers, cancellation-to-499 handling, DI connectivity checks).
- Refactor/modernize metadata & listing DTOs (TableSummary + builder), remove deprecated grouping/TableStatus infrastructure, and align routes + OpenAPI examples/filters.
Reviewed changes
Copilot reviewed 129 out of 131 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| PxApi/Utilities/WebApplicationExtensions.cs | Removes startup-time DB validation extension. |
| PxApi/Utilities/TableSummaryBuilder.cs | Adds builder for TableSummary DTO from PX metadata. |
| PxApi/Utilities/MetaCodeTools.cs | Removes legacy code “Convert()” utility. |
| PxApi/Utilities/LoggerScopeExtensions.cs | Adds standardized logging scope helpers (db/file/search). |
| PxApi/Utilities/LoggerConsts.cs | Adds constants for search query and not-found placeholder. |
| PxApi/Utilities/InputSanitizer.cs | Adds whitelist sanitizer for logging/search input. |
| PxApi/Services/Search/ElasticsearchDocument.cs | Adds ES _source mapping DTO. |
| PxApi/Services/Search/DisabledSearchService.cs | Adds stub ISearchService when feature flag is off. |
| PxApi/Services/ISearchService.cs | Introduces search service abstraction. |
| PxApi/PxApi.csproj | Bumps version and updates/adds packages (Elastic, OTEL, etc.). |
| PxApi/Program.cs | Adds CORS, global MVC filters, and conditional search registration. |
| PxApi/OpenApi/UnhandledErrorResponseOperationFilter.cs | Aligns 500 response schema to string. |
| PxApi/OpenApi/Examples/DatabaseListingExample.cs | Updates example links to new /meta/databases/... route. |
| PxApi/OpenApi/DocumentFilters/DataControllerPostEndpointDocumentFilter.cs | Updates path matching for new data route template. |
| PxApi/OpenApi/DocumentFilters/DataControllerGetEndpointDocumentFilter.cs | Updates path matching for new data route template. |
| PxApi/OpenApi/DocumentFilters/DatabasesControllerGetEndpointDocumentFilter.cs | Updates docs injection to /meta/databases. |
| PxApi/OpenApi/ClientClosedRequestOperationFilter.cs | Adds 499 response for cancellable operations. |
| PxApi/OpenApi/ApiExplorerConventions.cs | Hides SearchController when feature flag disabled; always hides CacheController. |
| PxApi/nlog.config | Disables internal log by default; adds dev console target. |
| PxApi/Models/TimeRange.cs | Adds TimeRange DTO. |
| PxApi/Models/TableSummary.cs | Adds TableSummary DTO for richer table listings/results. |
| PxApi/Models/TableStatus.cs | Removes TableStatus enum. |
| PxApi/Models/TableListingItem.cs | Refactors listing item to embed TableSummary. |
| PxApi/Models/TableGroup.cs | Removes TableGroup DTO. |
| PxApi/Models/Search/SearchTarget.cs | Adds search target enum. |
| PxApi/Models/Search/SearchResultItem.cs | Adds enriched search result DTO. |
| PxApi/Models/Search/SearchResponse.cs | Adds search response envelope DTO. |
| PxApi/Models/Search/SearchQueryInfo.cs | Adds query echo DTO for search responses. |
| PxApi/Models/Search/SearchHitResponse.cs | Adds raw hit response DTO (pre-enrichment). |
| PxApi/Models/Search/SearchHit.cs | Adds raw hit DTO. |
| PxApi/Models/Search/SearchDatabaseRef.cs | Adds DB reference DTO for search responses. |
| PxApi/Models/Search/MatchType.cs | Adds match type enum. |
| PxApi/Models/Search/MatchInfo.cs | Adds match info DTO. |
| PxApi/Models/QueryFilters/MetaFiltering.cs | Stops using MetaCodeTools; URL-encodes filter params. |
| PxApi/Models/QueryFilters/FilterJsonConverter.cs | Improves validation/errors; uses UnreachableException for default. |
| PxApi/Models/MetricInfo.cs | Adds MetricInfo DTO. |
| PxApi/Models/JsonStat/TableGroupJsonStatExtension.cs | Removes JSON-stat grouping extension model. |
| PxApi/Models/InfoResponse.cs | Adds InfoResponse record. |
| PxApi/Models/HealthResponse.cs | Adds health response models/status constants. |
| PxApi/Models/DimensionInfo.cs | Adds DimensionInfo DTO. |
| PxApi/ModelBuilders/JsonStat2Builder.cs | Removes grouping support; stops “Convert()” usage for dimension codes. |
| PxApi/GlobalSuppressions.cs | Adds controller-parameter-count suppression. |
| PxApi/Filters/OperationCanceledExceptionFilter.cs | Converts client cancellations to HTTP 499. |
| PxApi/Filters/LoggingScopeActionFilter.cs | Pushes controller/action names into logging scope globally. |
| PxApi/Exceptions/SearchUnavailableException.cs | Adds exception type for search backend failures. |
| PxApi/DataSources/MountedDataBaseConnector.cs | Implements CheckConnectionAsync for mounted connector. |
| PxApi/DataSources/IDataBaseConnector.cs | Adds CheckConnectionAsync to connector interface. |
| PxApi/DataSources/FileShareDataBaseConnector.cs | Implements CheckConnectionAsync for file share connector. |
| PxApi/DataSources/DataBaseConnector.cs | Adds default CheckConnectionAsync (delegates to GetAllFilesAsync). |
| PxApi/DataSources/BlobDataBaseConnector.cs | Implements CheckConnectionAsync for blob connectors. |
| PxApi/Controllers/MetadataController.cs | Updates route, adds CancellationToken, refactors logging scopes and audit behavior. |
| PxApi/Controllers/InfoController.cs | Adds hidden /info endpoint reporting version. |
| PxApi/Controllers/HealthController.cs | Adds hidden /health endpoint checking DB/search connectivity. |
| PxApi/Controllers/ErrorController.cs | Simplifies error responses to strings; updates logging messages. |
| PxApi/Controllers/CacheController.cs | Updates route to /cache/databases/... and adopts new logging scopes. |
| PxApi/Configuration/SearchConfig.cs | Adds typed config for Elasticsearch. |
| PxApi/Configuration/MemoryCacheConfig.cs | Replaces DefaultTableGroupSize with DefaultAliasSize. |
| PxApi/Configuration/FeatureFlagsConfig.cs | Adds SearchController feature flag. |
| PxApi/Configuration/DatabaseCacheConfig.cs | Removes Groupings cache config. |
| PxApi/Configuration/CorsConfig.cs | Adds typed CORS config with validation. |
| PxApi/Configuration/AuthenticationConfig.cs | Adds Search/Health API key config sections. |
| PxApi/Configuration/AppSettings.cs | Wires Search and CORS into AppSettings. |
| PxApi/Caching/ICachedDataSource.cs | Removes groupings cache API. |
| PxApi/Caching/DatabaseCache.cs | Removes groupings cache; adjusts alias cache sizing/config usage. |
| PxApi/Caching/CachedDataSource.cs | Removes groupings building/caching logic. |
| PxApi/Authentication/ApiKeyAuthAttribute.cs | Switches API key comparison to constant-time comparison. |
| PxApi/appsettings.Template.json | Adds new template config file for deployments/dev. |
| PxApi/appsettings.Development.json | Removes committed dev settings file. |
| PxApi/appsettings.deploy.json | Removes committed deploy settings file. |
| PxApi.UnitTests/Utils/TestConfigFactory.cs | Updates test config to new cache/feature flags shape. |
| PxApi.UnitTests/Utils/TableGroupTestUtils.cs | Removes groupings test helper. |
| PxApi.UnitTests/UtilitiesTests/TableSummaryBuilderTests.cs | Adds TableSummaryBuilder unit tests. |
| PxApi.UnitTests/UtilitiesTests/ServiceCollectionExtensionsTests.cs | Updates tests for config shape changes. |
| PxApi.UnitTests/UtilitiesTests/InputSanitizerTests.cs | Adds sanitizer tests. |
| PxApi.UnitTests/Utilities/LoggerScopeExtensionsTests.cs | Adds tests for new logger scope helpers. |
| PxApi.UnitTests/Services/ElasticSearchServiceTests.cs | Adds tests for ES mapping/query field selection helpers. |
| PxApi.UnitTests/PxApi.UnitTests.csproj | Updates test dependencies and adds OTEL dependency. |
| PxApi.UnitTests/OpenApi/OperationFilters/UnhandledErrorResponseOperationFilterTests.cs | Updates expectations for 500 schema change. |
| PxApi.UnitTests/OpenApi/OperationFilters/OperationIdOperationFilterTests.cs | Adjusts pragmas for analyzers. |
| PxApi.UnitTests/OpenApi/OperationFilters/ClientClosedRequestOperationFilterTests.cs | Adds tests for 499 operation filter. |
| PxApi.UnitTests/OpenApi/Examples/DatabaseListingExampleTests.cs | Updates expected links for new routes. |
| PxApi.UnitTests/OpenApi/DocumentFilters/DataControllerPostEndpointDocumentFilterTests.cs | Updates expected data route and constants. |
| PxApi.UnitTests/OpenApi/DocumentFilters/DataControllerGetEndpointDocumentFilterTests.cs | Updates expected data route and constants. |
| PxApi.UnitTests/Models/Search/SearchResultItemTests.cs | Adds JSON serialization tests for search DTOs. |
| PxApi.UnitTests/Models/QueryFilters/MetaFilteringTests.cs | Updates expected URL encoding; adds non-ASCII coverage. |
| PxApi.UnitTests/Models/QueryFilters/FilterJsonConverterTests.cs | Updates exception type expectations and allowed chars. |
| PxApi.UnitTests/ModelBuilderTests/TestMockMetaBuilder.cs | Adds parameterization for tableId. |
| PxApi.UnitTests/ModelBuilderTests/BuildJsonStat2Tests.cs | Updates for removal of groupings support. |
| PxApi.UnitTests/GlobalSuppressions.cs | Adds analyzer suppressions for test project. |
| PxApi.UnitTests/Filters/OperationCanceledExceptionFilterTests.cs | Adds tests for cancellation filter behavior. |
| PxApi.UnitTests/Filters/LoggingScopeActionFilterTests.cs | Adds tests for logging scope action filter behavior. |
| PxApi.UnitTests/ControllerTests/InfoControllerTests.cs | Adds tests for info endpoint result. |
| PxApi.UnitTests/ControllerTests/ErrorControllerTests.cs | Updates expectations for simplified error responses. |
| PxApi.UnitTests/ControllerTests/DataControllerStreamTests.cs | Updates expected dimension keys (case/diacritics). |
| PxApi.UnitTests/ControllerTests/DatabasesControllerTests.cs | Updates expected links and adds cancellation test. |
| PxApi.UnitTests/ConfigurationTests/MemoryCacheConfigurationTests.cs | Updates tests for cache config changes. |
| PxApi.UnitTests/ConfigurationTests/CorsConfigTests.cs | Adds tests for CORS config validation/normalization. |
| PxApi.UnitTests/ConfigurationTests/CacheSettingsTests.cs | Updates expected cache defaults (removes table group size). |
| PxApi.UnitTests/ConfigurationTests/AppSettingsTests.cs | Adds DefaultAliasSize assertions. |
| PxApi.UnitTests/ConfigurationTests/ApiExplorerConventionsFactoryTests.cs | Adds search feature-flag visibility tests. |
| PxApi.UnitTests/Caching/DatabaseCacheTests.cs | Removes groupings eviction behavior tests. |
| PxApi.UnitTests/Caching/CachedDataSourceTests.cs | Removes groupings cache tests. |
| PxApi.UnitTests/Authentication/ApiKeyAuthAttributeTests.cs | Adds Health controller auth tests. |
| PxApi.sln | Adds architecture docs and authentication doc to solution items. |
| docs/architecture/testing.md | Adds testing architecture documentation. |
| docs/architecture/services-and-utilities.md | Adds services/utilities architecture documentation. |
| docs/architecture/overview.md | Adds overall architecture overview doc. |
| docs/architecture/models-and-builders.md | Adds models/builders architecture documentation. |
| docs/architecture/data-access.md | Adds data access/connectors architecture documentation. |
| docs/architecture/controllers.md | Adds controller/routes architecture documentation. |
| docs/architecture/configuration.md | Adds configuration architecture documentation. |
| docs/architecture/caching.md | Adds caching architecture documentation. |
| .gitignore | Ignores additional local/dev artifacts and node_modules. |
| .github/skills/openapi-validation/SKILL.md | Adds OpenAPI validation “skill” documentation. |
| .github/skills/openapi-validation/package.json | Adds Node deps for OpenAPI validation script. |
| .github/skills/dotnet-code-validation/SKILL.md | Adds .NET validation “skill” documentation. |
| .github/copilot-instructions.md | Updates coding/test/security guidance. |
Files not reviewed (1)
- .github/skills/openapi-validation/package-lock.json: Language not supported
[Fix] Groupings reference cleanup
sakari-malkki
approved these changes
Jun 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.