Skip to content

feat: decode binary/protobuf payloads with messageType (LOE)#3356

Draft
rossnelson wants to merge 31 commits into
mainfrom
decode-binary-protobuf-payloads
Draft

feat: decode binary/protobuf payloads with messageType (LOE)#3356
rossnelson wants to merge 31 commits into
mainfrom
decode-binary-protobuf-payloads

Conversation

@rossnelson
Copy link
Copy Markdown
Collaborator

Summary

LOE prototype for browser-side decode of System Nexus payloads, where the gRPC envelope (SignalWithStartWorkflowExecutionRequest, future WaitForExternalWorkflow…, StandaloneActivity…, etc.) arrives in workflow history as a single Payload with metadata.encoding = binary/protobuf and metadata.messageType naming a fully-qualified temporal.api.* type.

Adds a Phase 1.5 dispatch in parseRawPayloadToJSON:

encoding === 'binary/protobuf' && metadata.messageType
  → @temporalio/proto: lookupType(messageType).decode(bytes).toObject()
  → recursively decode any nested {metadata, data} Payloads in the result
  → return as if it had been encoding 'json/plain' all along

Every existing renderer (event card, JSON view, compact row, copy/export) inherits the typed view for free. Adding the next system Nexus operation costs zero code — @temporalio/proto already ships every workflowservice message.

Demo

Repro harness lives in temporalio/nexus-endpoint (sibling repo): pinned temporal server + sdk-python spk/signal-with-start test that produces a real SignalWithStartWorkflowExecutionRequest in workflow history.

Before — what every renderer saw before this patch for nexusOperationScheduledEventAttributes.input:

{
  "metadata": {
    "encoding":    "binary/protobuf",
    "messageType": "temporal.api.workflowservice.v1.SignalWithStartWorkflowExecutionRequest"
  },
  "data": "CgdkZWZhdWx0EhhzeXN0ZW0tbmV4… (1068 base64 chars)"
}

After — same render path, no other changes:

{
  "namespace":     "default",
  "workflowId":    "system-nexus-workflow-id",
  "workflowType":  { "name": "EchoWorkflow" },
  "taskQueue":     { "name": "969f0ffd-…" },
  "input":         { "payloads": [ { "driver_claim": { }, "driver_name": "test-driver" } ] },
  "signalName":    "test-signal",
  "signalInput":   { "payloads": [ { "driver_claim": { }, "driver_name": "test-driver" } ] },
  "memo":          { "fields": { "memo-key": { "driver_claim": { }, "driver_name": "test-driver" } } },
  "userMetadata":  { "summary": { }, "details": { } },
  
}

Note the inner Payloads (the user's signal args, memo, etc.) are also decoded — the recursive pass in this PR feeds nested {metadata, data} nodes back through parseRawPayloadToJSON, so the existing JSON-decode path handles them transparently.

Visible UI surfaces (verified live)

  • Event detail panel — Input / Result typed JSON
  • Compact event row in the events table — same shape inline
  • "Copy as JSON" / export — same shape

Open questions for review

  1. CSP — second commit relaxes script-src to add 'unsafe-eval', because protobufjs builds per-message decoders via Function(). Two paths:
    • Keep this commit (accept unsafe-eval in production)
    • Drop this commit and migrate to @bufbuild/protobuf (no eval, smaller, ESM-native; ~2 days)
  2. Bundle size@temporalio/proto is already a dep but only used as types today. Worth a pnpm build before/after to see the runtime delta. If meaningful, switch to dynamic await import('@temporalio/proto') inside the decode helper.
  3. ./atob landmine — the local atob does UTF-8 decoding via decodeURIComponent, which corrupts raw protobuf bytes. The patch uses globalThis.atob directly. Worth either documenting that on ./atob.ts or extracting a rawBase64ToUint8Array helper for future binary work.
  4. UX — should we render Nexus operations as their inner type (e.g. "Signal With Start" as a first-class event row) rather than "Nexus Operation Scheduled" with a typed Input panel? Notion doc is silent. Worth a designer pass.

What this isn't

  • A production-ready feature. It's an LOE prototype. Drafting to share the working approach.
  • Anything beyond binary/protobuf payloads. Other encodings unchanged.
  • A Nexus presentation refactor. The event card still says "Nexus Operation Scheduled".

Test plan

  • pnpm test src/lib/utilities/decode-payload.test.ts -- --run — 27/27 passing (added 2)
  • pnpm check — 0 errors, 84 pre-existing warnings
  • pnpm lint — 0 errors, 210 pre-existing warnings
  • End-to-end: real workflow with SignalWithStartWorkflowExecutionRequest + nested external-storage payloads decoded in browser, screenshots in conversation
  • pnpm build size comparison — TODO before promoting beyond LOE

Notion

System Nexus Endpoint — the page's outstanding "UI" question is whether the browser can decode without a proto library. Answer: yes, it already can; this PR shows how.

rossedfort and others added 26 commits April 9, 2026 15:02
…load component

Consolidates two divergent payload display components into a single
`<Payload />` component at `src/lib/components/payload.svelte`.

Previously, `payload-decoder.svelte` (Svelte 5 runes, required a
`children` snippet, always decoded full attribute trees) and
`metadata-decoder.svelte` (Svelte 4 options API with slot syntax,
decoded single payloads to truncated summary strings) served different
display purposes but used inconsistent APIs and decoding paths across
65+ call sites.

The new component unifies both under a single `mode` prop:
- `code-block` (default): renders a CodeBlock with the decoded value,
  replacing all PayloadDecoder + CodeBlock snippet patterns
- `summary`: truncates to 120 chars and renders a Badge, replacing all
  MetadataDecoder usages
- `inline-truncated`: renders the compact .payload pre block used in
  event detail rows
- `children` snippet: escape hatch for callers that need custom
  rendering (schedule input, timeline SVG text elements)

Both old components used a single decoding path internally
(cloneAllPotentialPayloadsWithCodec -> decodePayloadAttributes ->
stringifyWithBigInt). MetadataDecoder previously used a separate
decodeSingleReadablePayloadWithCodec path; the new component uses the
same unified path for consistency.

The component is written in Svelte 5 runes throughout and imports from
$app/state instead of $app/stores, matching the newer direction of the
codebase.

Migrated files:
- src/lib/components/event/event-card.svelte
- src/lib/components/event/event-details-row.svelte (+ moved .payload CSS)
- src/lib/components/event/event-metadata-expanded.svelte
- src/lib/components/event/event-summary-row.svelte
- src/lib/components/lines-and-dots/svg/group-details-text.svelte
- src/lib/components/lines-and-dots/svg/timeline-graph-row.svelte
- src/lib/components/schedule/schedule-form/schedule-input-payload.svelte
- src/lib/components/standalone-activities/activity-input-and-outcome.svelte
- src/lib/components/workflow/client-actions/update-confirmation-modal.svelte
- src/lib/components/workflow/input-and-results-payload.svelte
- src/lib/components/workflow/metadata/metadata-events.svelte
- src/lib/components/workflow/pending-activity/pending-activity-card.svelte
- src/lib/components/workflow/workflow-json-navigator.svelte
- src/lib/pages/standalone-activity-details.svelte
- src/lib/pages/standalone-activity-search-attributes.svelte
- src/lib/pages/workflow-memo.svelte
- src/lib/pages/workflow-metadata.svelte
- src/lib/pages/workflow-search-attributes.svelte

All 1730 tests pass.
Moves payload.svelte into src/lib/components/payload/ and adds a
USAGE.md report documenting all 31 call sites grouped by feature area.
Updates all 17 import paths accordingly.
… components

Replace the 14-prop multi-mode payload.svelte with four single-purpose
components: PayloadCodeBlock, PayloadSummary, PayloadDecoder, and
PayloadInline. Each component carries only the props relevant to its
render mode, eliminating dead props at call sites.

Also updates the decode paths to use the renamed decode-payload.ts API
(decodeEventAttributes, parsePayloadAttributes) following the #3302
cleanup.
Replace decodeEventAttributes with decodeUserMetadata — the correct
decode path for a single raw Payload (Phase 2 codec only, no attribute
tree walking). Also add onDecode callback prop, called after a
successful decode, consistent with PayloadCodeBlock.
…lue utility

Remove 3-way duplication of getInitialValue/onMount decode logic from
payload-code-block, payload-decoder, and payload-inline by extracting
getInitialPayloadValue and decodePayloadValue into
src/lib/utilities/decode-payload-value.ts.
Components now re-decode when value or fieldName props change after
mount, fixing the reactivity gap that caused stale decoded content
when parent components updated their payload bindings.
Add a Phase 1.5 dispatch in parseRawPayloadToJSON: when a Payload's
metadata says encoding=binary/protobuf and carries a messageType, look
the type up by full name in @temporalio/proto's static namespace and
return the typed JSON. After the outer decode, walk the result tree
and recursively decode any nested {metadata, data} Payload nodes
through the same pipeline so user-side payloads (json/plain, json/
protobuf, json/external-storage-reference, etc.) come out fully
parsed. Unknown messageTypes and decode errors fall through to the
existing path so behavior for non-binary/protobuf payloads is
unchanged.

This is the proposal for handling System Nexus operations like
SignalWithStartWorkflowExecutionRequest, where the gRPC envelope
arrives as binary protobuf and the inner Payloads carry user data.

Tests cover round-tripping a real SignalWithStart fixture and the
graceful fallback for an unknown messageType. 27/27 passing.

Caveat: protobufjs builds its decoders with Function() and so needs
'unsafe-eval' in the script-src CSP. That's relaxed in a separate
commit so reviewers can drop it before merge if we want a different
proto runtime (@bufbuild/protobuf has no eval).
@temporalio/proto uses protobufjs, which compiles per-message
encoders/decoders via the Function() constructor. With the existing
'strict-dynamic' CSP and no 'unsafe-eval', binary/protobuf payload
decode silently fails in the browser.

Relax the dev/auto CSP to permit unsafe-eval so the new decode path
works end-to-end during the LOE evaluation. If the team decides to
ship browser-side proto decoding, the long-term option is to swap
@temporalio/proto for @bufbuild/protobuf (which doesn't use eval) and
drop this commit.

Open question for review: do we keep this, or migrate the runtime?
@vercel
Copy link
Copy Markdown

vercel Bot commented Apr 29, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
holocene Ready Ready Preview, Comment May 4, 2026 5:48pm

Request Review

rossnelson added 2 commits May 4, 2026 13:05
…-binary-protobuf

Moves lookupTemporalProtoType, base64ToUint8Array, looksLikeRawPayload,
recursivelyDecodeNestedPayloads, and the full decode path into a new
decode-binary-protobuf utility. The recursive walker now accepts a recurse
callback rather than calling parseRawPayloadToJSON directly, eliminating
the circular dependency. Adds unit tests covering successful decode, silent
null on unknown type, warn-on-throw, looksLikeRawPayload edge cases, and
the callback injection path.
…quests

Converts payload-decoder.svelte from {#await} to AbortController + $effect
so stale in-flight decodes cannot overwrite a newer result. Rewrites
codeServerRequest in data-encoder.ts to accept an optional AbortSignal and
retry transient errors (network/5xx) up to 3 attempts with 500ms/1000ms
delays, without retrying 4xx. Deletes the now-unused decode-payload-value
module (zero importers confirmed).
Adds a nexus-operation-registry that recognises the 4 system-level Nexus
operation types (StartWorkflow, SignalWorkflow, SignalWithStartWorkflow,
QueryWorkflow) and returns a human-readable descriptor with the embedded
input payloads. NexusOperationRenderer uses the registry to show the
embedded operation directly — users see "Signal With Start Workflow:
EchoWorkflow" and the decoded input, not the raw protobuf wrapper.
input-and-results-payload routes Nexus payloads through the renderer
automatically; all other content paths are unchanged. Adds 6 registry
tests covering the happy path and null returns for unknown types.
Base automatically changed from refactor-payload-component to main May 14, 2026 16:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants