Solutions Analyzer: v9.9 marketplace-authoritative publish status + category taxonomy#14476
Open
oshezaf wants to merge 7 commits into
Open
Solutions Analyzer: v9.9 marketplace-authoritative publish status + category taxonomy#14476oshezaf wants to merge 7 commits into
oshezaf wants to merge 7 commits into
Conversation
…ategory taxonomy Refresh all CSV outputs after merging master. Replace ~430 blanket is_published=true overrides with marketplace lookup-redirects (double-prefix fix, offerId filter-query fallback, pre-check lookup-key redirects). Adds category_primary taxonomy, ARM-expression table-name filter, and override-driven discovered-connector corrections. Net data: +7 solutions, +11 connectors.
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Updates the Solutions Analyzer toolchain to v9.9, improving marketplace-derived publish status accuracy and introducing a normalized table category taxonomy, with regenerated CSV/doc outputs.
Changes:
- Make marketplace publish-status checks catalog-authoritative (fix legacy-id double-prefix, add offerId filter fallback, and allow pre-check lookup-key overrides).
- Add
category_primarytaxonomy + diagnostics totables.csv, and adjust solution category flattening semantics. - Refresh generated CSV/doc artifacts to reflect updated mapping logic and latest repo content.
Reviewed changes
Copilot reviewed 15 out of 24 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| Tools/Solutions Analyzer/map_solutions_connectors_tables.py | Implements marketplace lookup fixes/fallbacks, pre-check overrides, category_primary taxonomy, and improved table-name filtering. |
| Tools/Solutions Analyzer/README.md | Documents v9.9 behavior changes (marketplace checks, taxonomy, overrides, discovered-flag corrections). |
| Tools/Solutions Analyzer/solution_analyzer_overrides.csv | Removes blanket is_published=true rows and adds targeted lookup redirects + connector not_in_solution_json corrections. |
| Tools/Solutions Analyzer/tables_reference.csv | Updates reference rows used for table metadata and categorization inputs. |
| Tools/Solutions Analyzer/solutions_connectors_tables_mapping_simplified.csv | Refreshes simplified solution→connector→table mappings (new solutions/connectors). |
| Tools/Solutions Analyzer/solutions_connectors_tables_issues_and_exceptions_report.csv | Refreshes issues/exceptions output to reflect new discovery/method resolution. |
| Tools/Solutions Analyzer/playbook_connectors.csv | Refreshes extracted playbook connector usage with new solutions/playbooks. |
| Tools/Solutions Analyzer/parsers.csv | Refreshes parser inventory (IDs/versions and new parsers). |
| Tools/Solutions Analyzer/asim_parsers.csv | Refreshes ASIM parser inventory (new sub-parsers and metadata updates). |
| Tools/Solutions Analyzer/asim_parsers_unmatched_report.csv | Refreshes unmatched ASIM parser report after parser inventory changes. |
| Tools/Solutions Analyzer/artifact_doc_links.csv | Refreshes generated doc-link index for new/updated artifacts. |
| Tools/Solutions Analyzer/script-docs/map_solutions_connectors_tables.md | Documents new override semantics and marketplace fallback behavior + taxonomy resolution details. |
| Tools/Solutions Analyzer/script-docs/csv/tables.md | Updates tables CSV documentation to pivot reporting guidance to category_primary. |
| Tools/Solutions Analyzer/script-docs/csv/solutions.md | Clarifies override-able solution offer/publisher IDs and updated categories semantics. |
| Tools/Solutions Analyzer/script-docs/csv/solution_analyzer_overrides.md | Documents new marketplace lookup-key overrides and taxonomy override guidance. |
| "ATCExpressRouteCircuitIpfix","Reference for ATCExpressRouteCircuitIpfix table in Azure Monitor Logs.","Azure Resources","","","LogManagement","microsoft.networkfunction/azuretrafficcollectors","","Yes","No","No","Yes","No","No","https://learn.microsoft.com/en-us/azure/azure-monitor/reference/tables/atcexpressroutecircuitipfix","","No","Yes","No","" | ||
| "ATCMicrosoftPeeringMetadata","Reference for ATCMicrosoftPeeringMetadata table in Azure Monitor Logs.","Azure Resources","","","LogManagement","microsoft.networkfunction/azuretrafficcollectors","","Yes","No","No","Yes","No","No","https://learn.microsoft.com/en-us/azure/azure-monitor/reference/tables/atcmicrosoftpeeringmetadata","","No","Yes","No","" | ||
| "ATCPrivatePeeringMetadata","Reference for ATCPrivatePeeringMetadata table in Azure Monitor Logs.","Azure Resources","","","LogManagement","microsoft.networkfunction/azuretrafficcollectors","","Yes","No","No","Yes","No","No","https://learn.microsoft.com/en-us/azure/azure-monitor/reference/tables/atcprivatepeeringmetadata","","No","Yes","No","" | ||
| "ATCPrivatePeeringMetadata","","Azure Resources","","","","","","Yes","No","No","Yes","No","No","https://learn.microsoft.com/en-us/azure/azure-monitor/reference/tables/atcprivatepeeringmetadata","","","Yes","No","" |
| "BitwardenEventLogs","","","","","","","","No","No","No","No","No","Yes","","","","No","No","No" | ||
| "BlockchainApplicationLog","Reference for BlockchainApplicationLog table in Azure Monitor Logs.","Azure Resources","","","LogManagement","microsoft.blockchain/blockchainmembers","","Yes","No","No","Yes","No","No","https://learn.microsoft.com/en-us/azure/azure-monitor/reference/tables/blockchainapplicationlog","","No","Yes","No","" | ||
| "BlockchainProxyLog","Reference for BlockchainProxyLog table in Azure Monitor Logs.","Azure Resources","","","LogManagement","microsoft.blockchain/blockchainmembers","","Yes","No","No","Yes","No","No","https://learn.microsoft.com/en-us/azure/azure-monitor/reference/tables/blockchainproxylog","","No","Yes","No","" | ||
| "BlockchainProxyLog","","Azure Resources","","","","","","Yes","No","No","Yes","No","No","https://learn.microsoft.com/en-us/azure/azure-monitor/reference/tables/blockchainproxylog","","","Yes","No","" |
| "LightningIOEResults_CL","","","","","","","","No","No","No","No","No","Yes","","","","No","Yes","No" | ||
| "LightningTier0Nodes_CL","","","","","","","","No","No","No","No","No","Yes","","","","No","Yes","No" | ||
| "LinuxAuditLog","Reference for LinuxAuditLog table in Azure Monitor Logs.","Security","","","Security, SecurityInsights","-","","Yes","No","No","Yes","No","No","https://learn.microsoft.com/en-us/azure/azure-monitor/reference/tables/linuxauditlog","","No","Yes","No","" | ||
| "LinuxAuditLog","","Security","","","","","","Yes","No","No","Yes","No","No","https://learn.microsoft.com/en-us/azure/azure-monitor/reference/tables/linuxauditlog","","","Yes","No","" |
| "OAuthAppInfo","Microsoft 365-connected OAuth applications registered with Microsoft Entra ID and available in the Defender for Cloud Apps app governance capability","Security, XDR","","","SecurityInsights","-","","Yes","Yes","No","No","No","No","https://learn.microsoft.com/en-us/azure/azure-monitor/reference/tables/oauthappinfo","https://learn.microsoft.com/en-us/defender-xdr/advanced-hunting-oauthappinfo-table","Yes","","No","" | ||
| "OCI_LogsV2_CL","","","","","","","","No","No","No","No","No","Yes","","","","Yes","Yes","Yes" | ||
| "OEPAirFlowTask","Reference for OEPAirFlowTask table in Azure Monitor Logs.","Azure Resources","","","LogManagement","microsoft.openenergyplatform/energyservices","","Yes","No","No","Yes","No","No","https://learn.microsoft.com/en-us/azure/azure-monitor/reference/tables/oepairflowtask","","No","Yes","No","" | ||
| "OEPAirFlowTask","","Azure Resources","","","","","","Yes","No","No","Yes","No","No","https://learn.microsoft.com/en-us/azure/azure-monitor/reference/tables/oepairflowtask","","","Yes","No","" |
Comment on lines
+11020
to
+11041
| # Redirect the marketplace lookup key BEFORE checking availability. | ||
| # | ||
| # The published signal is derived purely from the public Azure Marketplace | ||
| # catalog API, keyed by the solution's "<publisher_id>.<offer_id>" legacy | ||
| # id taken from SolutionMetadata.json. When a solution ships under a | ||
| # different marketplace offer than its repo metadata records — e.g. a | ||
| # renamed/re-published offer, a publisher hand-off, or a repo folder that | ||
| # carries no SolutionMetadata.json at all — the direct lookup 404s and the | ||
| # solution is mis-reported as unpublished even though it is live. | ||
| # | ||
| # Rather than masking that with a blanket `is_published` override (which | ||
| # hard-codes the conclusion and hides any future marketplace change), a | ||
| # Solution-scoped `solution_publisher_id` / `solution_offer_id` override | ||
| # redirects WHAT is looked up so the public marketplace check itself | ||
| # returns the correct, self-maintaining answer. These overrides are applied | ||
| # here, before the check, because the standard solution-override pass runs | ||
| # later (after marketplace status has already been resolved). The mapper | ||
| # never calls the authenticated Content Hub APIs; only the public | ||
| # marketplace catalog is consulted. | ||
| if overrides: | ||
| for _sol_info in all_solutions_info.values(): | ||
| apply_overrides_to_row(_sol_info, overrides, 'solution', 'solution_name') |
Comment on lines
+5394
to
+5402
| odata_filter = ( | ||
| "(categoryIds/any(cat: cat eq 'AzureSentinelSolution') " | ||
| f"or keywords/any(key: contains(key,'{AZURE_MARKETPLACE_SENTINEL_KEYWORD}'))) " | ||
| f"and (offerId eq '{offer_part}')" | ||
| ) | ||
| api_url = ( | ||
| f"{AZURE_MARKETPLACE_API_URL}?api-version={AZURE_MARKETPLACE_API_VERSION}" | ||
| f"&$filter={quote(odata_filter, safe='')}" | ||
| ) |
Comment on lines
+5359
to
+5360
| prefix = f"{publisher_id}." | ||
| return offer_id if offer_id.startswith(prefix) else f"{publisher_id}.{offer_id}" |
…x SlashNext index drop) Mapper now always emits an empty-table placeholder row when a connector resolves to zero tables (table_detection_failed, parser_tables_only, partial_parser_tables, no_table_definitions), instead of silently dropping the connector and its solution from the index. Restores SlashNext, which a v9.9 reported_table_exclusions override had reduced to zero tables. Both doc generators (generate_connector_docs.py, generate_interactive_docs.py) now union-seed by_solution from solutions.csv so every solution is indexed even with no mapping rows (also covers deprecated solutions). Updated README v9.9 changelog and script-docs; regenerated CSVs.
…onnectors - Extract table names from connectorUiConfig.dataTypes for standalone *_ConnectorDefinition.json files - Establishes dataTypes as Priority 0 (authoritative) source above companion files and query analysis - Fixes 24 CCF v3 connectors that previously reported zero tables despite having 90+ defined tables - Primary impact: AlibabaCloudNetworkingConnector now correctly exports 3 tables (VPCFlowLogs, WAFLogs, APIGatewayLogs) - Updated README.md v9.9 changelog with priority hierarchy details - Regenerated sentinelninja connector docs with extracted table corrections
…'outputStream' fields - Priority 2 DCR extraction now checks both standard 'streams' array and legacy 'outputStream' field - Fixes missed tables in connectors using Azure DCR standard format (e.g., AlibabaCloudNetworking_DCR.json) - Enables Priority 2 to properly extract 'Microsoft-Sentinel*' prefixed stream names from dataFlows - Improves robustness for connectors that may only have DCR files without ConnectorDefinition.json - AlibabaCloudNetworking now correctly extracts all 3 tables from dataTypes (Priority 0) regardless
- DCR table extraction now strips leading Sentinel token after Microsoft-/Custom- prefix normalization - Supports stream forms like Microsoft-Sentinel<TableName> and maps to canonical table names - Aligns DCR-derived tables with connector dataTypes for CCF connectors (Alibaba Cloud Networking now matches) - Update mapper script-docs to document outputStream/streams handling and normalization behavior
- Add TABLE_SCHEMA_REFERENCES mapping for SecurityAlert, DnsEvents, DnsInventory, AMA_DNS - Implement get_schema_references() function with table-specific and fallback logic - Add 'Schema References' section to generated table documentation pages - Include schema references in Table of Contents generation - Update script documentation with configuration details - Update version history to v9.10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Updates the Solutions Analyzer tooling (
Tools/Solutions Analyzer/) to v9.9 and refreshes all generated CSV outputs after syncing withmaster. All changes are scoped toTools/Solutions Analyzer/— no solution content is touched.Code / logic changes (
map_solutions_connectors_tables.py)Marketplace publish status is now fully catalog-authoritative and self-correcting:
_build_legacy_id()fixes a double-prefix bug whereofferIdalready prefixed with<publisherId>.produced a 404-ing id (e.g.azuresentinel.azuresentinel.trendmicrocas), wrongly marking published solutions asmp_is_published=false._query_marketplace_by_offer_id()adds a catalog$filterfallback on 404, recovering offers republished under a different publisherId (e.g. Zscaler Internet Access). Recovery-only — never flips a published solution to unpublished.Solution-scopedsolution_publisher_id/solution_offer_idoverrides are applied before the marketplace check, redirecting what the public catalog looks up rather than hard-coding a verdict.Solution,…,is_published,trueoverride rows — replaced with 11 marketplace lookup-redirect override pairs. Publish status now derives from the live public catalog (anonymouscatalogapi.azure.com/offersonly; no authenticated APIs).Other:
category_primarytaxonomy column ontables.csv(Cloud/Endpoint/Syslog/CEF/3rd Party (SaaS)/Defender/ASIM/Internal/Unknown) withcategory_source/category_candidatesdiagnostics; overridable viaTable,…,category_primary.solution_categoriesnow lists domain/vertical values instead of JSON key names.is_true_table_name()filters ARM-template expressions so placeholders like[parameters('PlaybookName')]_CLno longer leak intotables.csv.not_in_solution_json=falsecorrections for three published connectors flagged as "discovered" due to source-side gaps (MailGuard365, CiscoMerakiNativePoller, Pathlock_TDnR).Data deltas (after merge + rerun, no cache invalidation)
Docs / metadata
## Version Historyupdated for v9.9.script-docs/updated for the mapper,solution_analyzer_overrides.md,solutions.md,tables.md.Generated docs and CSV mirrors are published out-of-band to the docs site and CSV output branch.