Staging

.gitignore

@@ -217,3 +217,7 @@ __marimo__/
 
 # Pre-commit cache
 /.pre-commit-cache/
+
+# Agents
+# codex-instructions
+local

AGENTS.md

@@ -7,6 +7,7 @@ This document is a living map for developers building agents in the Codon ecosys
 - `docs/guides/codon-workload-quickstart.md` shows an end-to-end example of composing and running a workload directly with the SDK.
 - [`instrumentation-packages/codon-instrumentation-langgraph/AGENTS.md`](instrumentation-packages/codon-instrumentation-langgraph/AGENTS.md) covers the LangGraph decorators, the `LangGraphWorkloadAdapter`, and how to inherit telemetry automatically.
 - [`instrumentation-packages/codon-instrumentation-openai/AGENTS.md`](instrumentation-packages/codon-instrumentation-openai/AGENTS.md) will track OpenAI-specific instrumentation work. It currently outlines expectations and open tasks.
+- Telemetry initialization now lives in the core SDK (`codon_sdk.instrumentation.initialize_telemetry`), with a hardcoded production default endpoint and API-key header support. Framework packages re-export this entrypoint to stay aligned. CodonWorkload can emit spans on its own when `enable_tracing=True` (default: False); leave it disabled if another instrumentation layer is already wrapping nodes to avoid duplicate spans.
 
 If you are introducing a new framework integration, create an `AGENTS.md` alongside it and link back here.
 
@@ -21,6 +22,7 @@ Telemetry defaults to OpenTelemetry OTLP exporters; align your environment varia
 - The OpenAI instrumentation package is still a stub—see its `AGENTS.md` for the punch list.
 - SDK testing and documentation are light; expect breaking changes as the contracts tighten.
 - Additional frameworks (e.g., LangChain, LiteLLM) may join this layout. Document them here when they land.
+- Telemetry bootstrap is centralized; callers can override endpoint/API key via args or env vars (`OTEL_EXPORTER_OTLP_ENDPOINT`, `CODON_API_KEY`). An opt-in flag (`CODON_ATTACH_TO_EXISTING_OTEL_PROVIDER`) lets Codon attach its OTLP exporter to an existing tracer provider (e.g., when users already run OTEL auto-instrumentation) instead of replacing it.
 
 ## Contribution Guidelines
 - Keep the `AGENTS.md` files in sync with code changes that alter agent-facing behavior.

codex-instructions/sdk-production-refactor.md

@@ -0,0 +1,174 @@
+### Phase 1: The "Front Door" (API Gateway)
+
+**Objective:** Secure the ingestion pipeline immediately with minimal complexity.
+**Timeline:** Immediate / MVP.
+
+#### 1\. SDK Changes (Core Refactoring)
+
+  * **Move Logic:** Move `initialize_telemetry` from `codon.instrumentation.langgraph` to `codon_sdk.instrumentation`.
+  * **Update Signature:** Accept `api_key` and `endpoint` arguments.
+  * **Hardcode Default:** Set the default endpoint to your production load balancer (e.g., `https://ingest.codonops.ai:4317`).
+  * **Header Injection:** Inject `x-codon-api-key` into the exporter.
+
+**Action:** Create `sdk/src/codon_sdk/instrumentation/config.py` (or similar) and migrate the logic there. Update the LangGraph package to import it.
+
+#### 2\. Infrastructure Changes (Server-Side)
+
+  * **DNS:** Point `ingest.codonops.ai` to your Cloud Load Balancer.
+  * **Gateway:** Deploy Nginx (or Kong) behind the LB.
+      * **Config:** specific rule to check `x-codon-api-key` against your DB/Cache.
+      * **Success:** If valid, inject header `X-Codon-Org-ID: <org_id>` and forward to Collector.
+      * **Failure:** Return `401 Unauthorized`.
+  * **OTel Collector:**
+      * Enable `include_metadata: true` on OTLP receivers.
+      * Add `attributes/insert_tenancy` processor to extract `metadata.x-codon-org-id`.
+
+-----
+
+### Phase 2: The "Infinite Scale" (JWT)
+
+**Objective:** Decouple ingestion authorization from the database to handle high-volume telemetry.
+**Timeline:** Post-Launch / Scale-up.
+
+#### 1\. SDK Changes
+
+  * **Token Manager:** Implement a `TokenManager` class in the Core SDK.
+      * **Exchange:** On init, call `https://api.codon.ai/auth/token` with the API Key to get a JWT.
+      * **Refresh:** Automatically refresh the JWT 5 minutes before expiry.
+  * **Exporter Update:** Update `initialize_telemetry` to use the dynamic JWT in the `Authorization: Bearer <token>` header instead of the static API key.
+
+#### 2\. Infrastructure Changes
+
+  * **Auth Service:** Ensure your backend has an endpoint to mint signed JWTs (RS256) containing the `org_id` claim.
+  * **Collector/Gateway:** Reconfigure to use the **OIDC Authenticator**.
+      * It will now validate the **signature** of the JWT locally (CPU operation) instead of calling the DB (I/O operation).
+      * The `attributes` processor will now extract tenancy from `auth.claims.org_id`.
+
+-----
+
+### Architecture Decision Records (ADRs)
+
+Here are the formal records you should commit to your documentation (e.g., `docs/adr/`).
+
+#### ADR 001: Telemetry Authentication Strategy
+
+**Status:** Accepted
+**Date:** 2025-11-24
+
+**Context:**
+We need to secure the OpenTelemetry ingestion endpoint (`ingest.codonops.ai`) to prevent unauthorized data submission and ensure accurate billing/tenancy. We have two options: checking API keys against a database (Gateway pattern) or using signed tokens (JWT pattern).
+
+**Decision:**
+We will implement a **Two-Phase Strategy**:
+
+1.  **Phase 1 (API Gateway):** Validate API Keys at the ingress gateway (Nginx) via a database/cache lookup.
+2.  **Phase 2 (JWT):** Migrate to client-side JWT exchange for stateless authentication at scale.
+
+**Consequences:**
+
+  * **Positive:** Phase 1 allows for rapid implementation using existing API keys without complex SDK logic. Phase 2 provides a clear path to handling massive concurrency without database bottlenecks.
+  * **Negative:** Phase 1 introduces a database dependency on the hot path of telemetry ingestion. If the DB slows down, telemetry ingestion slows down. This is acceptable for MVP volumes but must be replaced (Phase 2) before high-scale adoption.
+
+#### ADR 002: Centralization of Telemetry Configuration
+
+**Status:** Accepted
+**Date:** 2025-11-24
+
+**Context:**
+Currently, `initialize_telemetry` is defined inside the `codon-instrumentation-langgraph` package. As we add support for OpenAI, CrewAI, and other frameworks, duplicating this initialization logic will lead to drift in configuration defaults (endpoints, auth headers, resource attributes).
+
+**Decision:**
+We will move the telemetry initialization logic to the **SDK Core** (`codon_sdk`).
+
+  * **New Location:** `codon_sdk.instrumentation.initialize` (or similar).
+  * **Instrumentation Packages:** Will import and wrap this core function or instruct users to call it directly.
+
+**Consequences:**
+
+  * **Positive:** Single source of truth for endpoints and authentication. Updates to the ingestion architecture (e.g., changing the endpoint URL or auth header format) only need to happen in one place (`codon_sdk`).
+  * **Negative:** Instrumentation packages take a hard dependency on the specific version of `codon_sdk` that contains this utility.
+
+-----
+
+### Recommended Code Structure for Core SDK
+
+Here is how you should implement the **Core SDK** change right now to support Phase 1.
+
+**File:** `sdk/src/codon_sdk/instrumentation/__init__.py` (or a new `config.py` inside it)
+
+```python
+import os
+from typing import Optional, Dict
+from opentelemetry import trace
+from opentelemetry.sdk.resources import Resource
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import BatchSpanProcessor
+from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
+
+# Hardcoded Production Endpoint - The "Front Door"
+DEFAULT_INGEST_ENDPOINT = "https://ingest.codonops.ai:4317"
+
+def initialize_telemetry(
+    api_key: Optional[str] = None,
+    service_name: Optional[str] = None,
+    endpoint: Optional[str] = None,
+) -> None:
+    """
+    Global initialization for Codon telemetry.
+
+    This should be called once at the start of the application. It configures
+    OpenTelemetry to send traces to the Codon Cloud (or a custom collector).
+    """
+
+    # 1. Resolve Identity
+    final_api_key = api_key or os.getenv("CODON_API_KEY")
+    if not final_api_key:
+        # Fallback for local dev without auth (optional, or raise Error)
+        # For Phase 1 production, we might want to enforce this.
+        pass
+
+    # 2. Resolve Context
+    # We prefer the user-provided service name, then env var, then default.
+    final_service_name = (
+        service_name
+        or os.getenv("OTEL_SERVICE_NAME")
+        or "unknown_codon_service"
+    )
+
+    # 3. Resolve Destination
+    # Default to Prod, allow env var override, allow explicit arg override.
+    final_endpoint = (
+        endpoint
+        or os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
+        or DEFAULT_INGEST_ENDPOINT
+    )
+
+    # 4. Configure Headers
+    headers: Dict[str, str] = {}
+    if final_api_key:
+        headers["x-codon-api-key"] = final_api_key
+
+    # 5. Setup OpenTelemetry
+    resource = Resource(attributes={"service.name": final_service_name})
+
+    exporter = OTLPSpanExporter(
+        endpoint=final_endpoint,
+        headers=headers
+    )
+
+    provider = TracerProvider(resource=resource)
+    provider.add_span_processor(BatchSpanProcessor(exporter))
+
+    # Set the global tracer provider so all instrumentation packages pick it up
+    trace.set_tracer_provider(provider)
+```
+
+**Usage in `codon-instrumentation-langgraph`:**
+You would delete the local `initialize_telemetry` definition and re-export or document the usage of the core one.
+
+```python
+# instrumentation-packages/codon-instrumentation-langgraph/codon/instrumentation/langgraph/__init__.py
+
+# Re-export for convenience, or deprecate and point users to core
+from codon_sdk.instrumentation import initialize_telemetry
+```

instrumentation-packages/codon-instrumentation-langgraph/AGENTS.md

@@ -160,7 +160,7 @@ The ledger records each iteration through the loop, and `runtime.state` tracks i
 ---
 
 ## Telemetry & Audit Integration
-- Call `initialize_telemetry(service_name=...)` once during process startup to export spans via OTLP.
+- Call `initialize_telemetry(service_name=...)` once during process startup to export spans via OTLP. The initializer now lives in the core SDK (`codon_sdk.instrumentation.initialize_telemetry`) and is re-exported here. It defaults the endpoint to `https://ingest.codonops.ai:4317`, injects `x-codon-api-key` from the argument or `CODON_API_KEY` env, and respects `OTEL_EXPORTER_OTLP_ENDPOINT`/`OTEL_SERVICE_NAME` overrides. If you already have an OTEL tracer provider (e.g., via auto-instrumentation), set `CODON_ATTACH_TO_EXISTING_OTEL_PROVIDER=true` or pass `attach_to_existing=True` to add Codon’s exporter to the existing provider instead of replacing it.
 - Each node span now carries workload metadata (`codon.workload.id`, `codon.workload.run_id`, `codon.workload.logic_id`, `codon.workload.deployment_id`, `codon.organization.id`) so traces can be rolled up by workload, deployment, or organization without joins.
 - `LangGraphTelemetryCallback` is attached automatically when invoking LangChain runnables; it captures model vendor/identifier, token usage (prompt, completion, total), and response metadata, all of which is emitted as span attributes (`codon.tokens.*`, `codon.model.*`, `codon.node.raw_attributes_json`).
 - Instrumentation writes into the shared `NodeTelemetryPayload` (`runtime.telemetry`) defined by the SDK so future mixins collect the same schema-aligned fields without reimplementing bookkeeping.

instrumentation-packages/codon-instrumentation-langgraph/codon/instrumentation/langgraph/__init__.py

@@ -23,16 +23,13 @@
 
 from opentelemetry import trace
 from opentelemetry.trace import Status, StatusCode
-from opentelemetry.sdk.trace import TracerProvider
-from opentelemetry.sdk.trace.export import BatchSpanProcessor
-from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
-from opentelemetry.sdk.resources import Resource
 
 from .attributes import LangGraphSpanAttributes
 from codon_sdk.instrumentation.schemas.nodespec import NodeSpec, NodeSpecSpanAttributes
 from codon_sdk.agents import Workload
 from codon_sdk.instrumentation.schemas.telemetry.spans import CodonBaseSpanAttributes
 from codon_sdk.instrumentation.telemetry import NodeTelemetryPayload
+from codon_sdk.instrumentation import initialize_telemetry
 
 __all__ = [
     "LangGraphWorkloadMixin",
@@ -45,7 +42,6 @@
     "LangGraphTelemetryCallback",
 ]
 
-SERVICE_NAME: str = os.getenv("OTEL_SERVICE_NAME")
 ORG_NAMESPACE: str = os.getenv("ORG_NAMESPACE")
 __framework__ = "langgraph"
 
@@ -78,31 +74,6 @@ def from_langgraph(
         """Translate a LangGraph graph into a concrete Codon workload."""
 
 
-def initialize_telemetry(service_name: str = SERVICE_NAME or "") -> None:
-    """Initialize OpenTelemetry tracing with OTLP exporter.
-
-    Sets up a TracerProvider with BatchSpanProcessor and OTLPSpanExporter to 
-    export spans to an OTLP collector endpoint.
-
-    Args:
-        service_name: Name for the service in telemetry data. Defaults to 
-                     OTEL_SERVICE_NAME environment variable or empty string.
-
-    Example:
-        >>> initialize_telemetry(service_name="codon-langgraph-demo")
-    """
-    # Set up service name
-    resource = Resource(attributes={"service.name": service_name})
-    # Set up TracerProvider & Exporter
-    provider = TracerProvider(resource=resource)
-    otlp_exporter = OTLPSpanExporter()
-
-    processor = BatchSpanProcessor(otlp_exporter)
-    provider.add_span_processor(processor)
-
-    trace.set_tracer_provider(provider)
-
-
 def current_invocation() -> Optional[NodeTelemetryPayload]:
     """Return the currently active node invocation telemetry (if any)."""
 
@@ -352,7 +323,7 @@ def track_node(
 
     When the decorated function executes, this decorator:
     - Materializes a NodeSpec and captures its ID, signature, and schemas
-    - Wraps execution in an OpenTelemetry span (async and sync supported)  
+    - Wraps execution in an OpenTelemetry span (async and sync supported)
     - Records inputs, outputs, and wall-clock latency via standardized span attributes
 
     Args:

sdk/AGENTS.md

@@ -82,6 +82,8 @@ For a full walkthrough, see `docs/guides/codon-workload-quickstart.md`.
 ## Telemetry Vocabulary
 - `codon_sdk.instrumentation.schemas.telemetry.spans` defines span names and base attributes shared across frameworks.
 - Document additions here before using them in instrumentation packages to keep alignment.
+- Telemetry initialization is centralized in `codon_sdk.instrumentation.initialize_telemetry`, with default endpoint `https://ingest.codonops.ai:4317` and `x-codon-api-key` header support (args override env; env vars `OTEL_EXPORTER_OTLP_ENDPOINT`, `CODON_API_KEY`, `OTEL_SERVICE_NAME` remain valid). Optional attach mode (`attach_to_existing` arg or `CODON_ATTACH_TO_EXISTING_OTEL_PROVIDER` env) lets you add Codon’s exporter to an existing tracer provider instead of replacing it—useful when OTEL auto-instrumentation is already active.
+- CodonWorkload can emit spans natively when `enable_tracing=True` (default: False). It uses the global tracer provider configured via `initialize_telemetry` to create one span per node execution with workload/org/deployment IDs, logic/run IDs, and NodeSpec attributes. Leave it disabled if another instrumentation layer (e.g., LangGraph adapter) is already wrapping nodes to avoid duplicate spans.
 
 ## Extending the SDK
 - Capture requirements for new schema fields inside each class docstring and mirror them here.

sdk/pyproject.toml

@@ -27,6 +27,9 @@ classifiers = [
 ]
 # Core dependencies needed by every part of the SDK
 dependencies = [
+    "opentelemetry-api",
+    "opentelemetry-exporter-otlp-proto-grpc",
+    "opentelemetry-sdk",
     "pydantic",
 ]