Skip to content

[WIP] feat: add MLflow tracing integration via OpenTelemetry bridge#310

Open
luis5tb wants to merge 4 commits into
RHEcosystemAppEng:mainfrom
luis5tb:worktree-mlflow
Open

[WIP] feat: add MLflow tracing integration via OpenTelemetry bridge#310
luis5tb wants to merge 4 commits into
RHEcosystemAppEng:mainfrom
luis5tb:worktree-mlflow

Conversation

@luis5tb

@luis5tb luis5tb commented Jun 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add MLflow LLM observability support via OpenTelemetry bridge — Google ADK natively generates OTel traces, MLflow (>= 3.6.0) accepts them at its /v1/traces OTLP endpoint
  • MLflow tracing is independently toggleable from OTel tracing (MLFLOW_ENABLED works without OTEL_ENABLED)
  • On Cloud Run, the agent connects to an external MLflow instance (Red Hat infrastructure) — no new GCP services deployed
  • On OpenShift, MLflow Tracking Server is deployed as an optional Helm component with dedicated PostgreSQL, artifact PVC, NetworkPolicy, and optional Route

Changes

Agent code:

  • config/settings.py: 6 new MLflow settings (enabled, tracking_uri, experiment_name, experiment_id, log_prompts, run_tags)
  • telemetry/setup.py: _add_mlflow_processor() adds a second OTLP span processor targeting MLflow; setup_telemetry() restructured so MLflow works independently of otel_enabled
  • pyproject.toml: new mlflow optional dependency group (mlflow-tracing>=3.6.0, opentelemetry-exporter-otlp-proto-http>=1.20.0)
  • .env.example, Makefile: env var docs + lock-mlflow target

Cloud Run: env vars in service.yaml pointing at external MLflow

OpenShift: 7 new Helm templates (Deployment, Service, Route, PVC, PostgreSQL, PostgreSQL PVC, NetworkPolicy) + values.yaml, configmap.yaml, secret.yaml, _helpers.tpl updates

Tests: 20 test cases covering settings, enabled/disabled paths, independent MLflow (no OTel), experiment ID headers

Test plan

  • ruff check — all checks passed
  • pytest tests/ — 470 passed, 0 failures
  • Deploy on OpenShift with mlflow.enabled: true and verify traces appear in MLflow UI
  • Deploy on Cloud Run with MLFLOW_ENABLED=true pointing at external MLflow and verify traces
  • Verify MLflow works with OTEL_ENABLED=false (independent mode)
  • Verify no impact when MLFLOW_ENABLED=false (default)

🤖 Generated with Claude Code

@luis5tb luis5tb force-pushed the worktree-mlflow branch 2 times, most recently from 9a3b873 to 2958385 Compare June 26, 2026 10:37
luis5tb and others added 4 commits July 1, 2026 18:22
Add MLflow LLM observability support. Google ADK natively generates
OTel traces; MLflow (>= 3.6.0) accepts them at its /v1/traces OTLP
endpoint. The agent adds a second BatchSpanProcessor targeting MLflow
alongside existing exporters.

- config/settings.py: 6 new MLflow settings (enabled, tracking_uri,
  experiment_name, experiment_id, log_prompts, run_tags)
- telemetry/setup.py: _add_mlflow_processor() sends all settings as
  OTLP headers; setup_telemetry() restructured so MLflow works
  independently of otel_enabled
- pyproject.toml: mlflow optional dep group (otlp-proto-http exporter)
- requirements-mlflow.txt: lock file with hashes
- .env.example, Makefile: env var docs + lock-mlflow target
- tests/test_mlflow.py: 26 test cases covering defaults, env loading,
  enabled/disabled paths, independent MLflow, headers, trailing slash
  handling, and ImportError

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
On Cloud Run the agent connects to an external MLflow instance (Red Hat
infrastructure) — no new GCP services deployed.

- service.yaml: MLFLOW_ENABLED, MLFLOW_TRACKING_URI, MLFLOW_EXPERIMENT_NAME,
  MLFLOW_EXPERIMENT_ID, MLFLOW_LOG_PROMPTS, MLFLOW_RUN_TAGS env vars
- README.md: MLflow Tracing section with configuration table, enabling
  instructions, networking/auth notes, and dependency info

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Deploy MLflow Tracking Server as an optional Helm component with
dedicated PostgreSQL, artifact PVC, NetworkPolicy, and optional Route.

New templates:
- mlflow-deployment.yaml: MLflow server with security contexts,
  readiness probe (/api/2.0/mlflow/experiments/search), liveness
  probe (tcpSocket), artifact PVC + tmp emptyDir mounts
- mlflow-service.yaml: ClusterIP service
- mlflow-route.yaml: optional OpenShift Route
- mlflow-pvc.yaml: artifact storage (10Gi default)
- mlflow-postgresql-deployment.yaml: dedicated PostgreSQL + Service
- mlflow-postgresql-pvc.yaml: PostgreSQL data (5Gi default)
- networkpolicy-mlflow.yaml: restricts ingress to agent pods only

Modified:
- _helpers.tpl: MLflow + PostgreSQL service name and selector helpers
- configmap.yaml: MLflow env vars with auto-resolved trackingUri
  pointing to the ClusterIP Service
- secret.yaml: MLFLOW_DB_PASSWORD for dedicated PostgreSQL
- values.yaml: mlflow section with pinned image (v3.14.0), resource
  defaults, storage, route, auth, and postgresql config
- README.md: MLflow Tracking section with value table and usage docs

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add Helm fail guard when mlflow.postgresql.mode != 'dedicated' to
  prevent silent OTLP trace ingestion failure (file-based backends
  silently drop traces per MLflow docs)
- Replace incorrect mlflow-tracing package references in Cloud Run and
  OpenShift READMEs with the actual dependency
  (opentelemetry-exporter-otlp-proto-http, already in the container)
- Document that MLFLOW_EXPERIMENT_ID requires the experiment to already
  exist on the server (use MLFLOW_EXPERIMENT_NAME for auto-creation)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@luis5tb luis5tb force-pushed the worktree-mlflow branch from 2958385 to 7ec5141 Compare July 1, 2026 16:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant