starknet_transaction_prover: proving-job duration + outcome metrics

[avi-starkware]

Adds Prometheus counters / histograms recorded by VirtualSnosProver for
each proving job: total count by outcome (success, validation_error,
internal_error, l1_provider_error) and end-to-end duration.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

[cursor]

PR Summary

Low Risk
Observability-only changes around existing proving flow; no auth, data, or proof logic changes beyond recording metrics after each request.

Overview
Adds Prometheus observability for each prove_transaction call: end-to-end latency, per-outcome counters, and sub-step histograms for virtual OS run and Stwo proving.

prove_transaction now delegates to prove_transaction_inner and always records prover_prove_transaction_duration_seconds (success or failure) and prover_prove_transaction_outcome_total{outcome} with a fixed label set (success, failure_validation, failure_blocked, failure_runner, failure_output_parse, failure_proving). VirtualSnosProverError::metric_outcome() maps error variants to those labels to keep cardinality bounded. run_and_prove additionally emits prover_os_run_duration_seconds and prover_stwo_prove_duration_seconds. Metric names and outcome strings live in server::metrics.

^{Reviewed by Cursor Bugbot for commit 931d783. Bugbot is set up for automated code reviews on this repo. Configure here.}

[avi-starkware]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• starknet_transaction_prover: document observability surface in README #14203
• starknet_transaction_prover: origin-level breadcrumbs for validation errors #14172
• starknet_transaction_prover: /health returns 503 when service is saturated #14171
• starknet_transaction_prover: bump prover_panics_total on panic hook fire #14170
• starknet_transaction_prover: HTTP request count + latency + in-flight metrics #14169
• starknet_transaction_prover: proving-job duration + outcome metrics #14168 👈 (View in Graphite)
• starknet_transaction_prover: Prometheus /metrics endpoint with build_info #14167
• starknet_transaction_prover: global panic hook + graceful SIGTERM shutdown #14166
• starknet_transaction_prover,tower_ohttp: OHTTP-unlinkable request-id for decapsulated content #14222
• starknet_transaction_prover: per-request structured log with request-id #14165
• starknet_transaction_prover: unify HTTP middleware stack via macro #14286
• main-v0.14.3

This stack of pull requests is managed by Graphite. Learn more about stacking.

[reviewable-StarkWare]

This change is 

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 931d783. Configure here.}

[cursor]

Sub-step histograms skip failures

Medium Severity

In run_and_prove, OS_RUN_DURATION_SECONDS and STWO_PROVE_DURATION_SECONDS are recorded only after run_virtual_os and prove_virtual_snos_run succeed. When either step returns an error via ?, the elapsed time for that step is never observed, so failure-path latency is missing from sub-step histograms while the end-to-end histogram still includes it.

 

^{Reviewed by Cursor Bugbot for commit 931d783. Configure here.}