diff --git a/docs/guides/function/builtin-interfaces.md b/docs/guides/function/builtin-interfaces.md index f45582d..0451d24 100644 --- a/docs/guides/function/builtin-interfaces.md +++ b/docs/guides/function/builtin-interfaces.md @@ -166,12 +166,22 @@ const executionId = await tailor.workflow.triggerWorkflow( const result = await tailor.workflow.triggerJobFunction("calculateTax", { amount: 1000, }); + +// Route the dispatch through a workspace-registered execution policy for +// per-key concurrency control (see the SDK Workflow guide for policy setup). +const scoped = await tailor.workflow.triggerJobFunction( + "syncTenant", + { tenantId: "acme" }, + { executionPolicyKey: `tenant-api.acme` }, +); ``` | Function | Returns | Description | | --- | --- | --- | | `triggerWorkflow(name, args?, options?)` | `Promise` | Trigger a workflow. Returns the execution ID | -| `triggerJobFunction(name, args?)` | `any` | Trigger a job function and return its result | +| `triggerJobFunction(name, args?, options?)` | `Promise` | Trigger a job function and return its result. `options.executionPolicyKey` routes the dispatch through a matching execution policy for per-key concurrency control | + +For details on declaring execution policies and the key grammar, see [Execution Policies](/sdk/services/workflow#execution-policies) in the SDK Workflow reference. ## TailorDB Client diff --git a/docs/guides/workflow/index.md b/docs/guides/workflow/index.md index e4c1c06..574e0c0 100644 --- a/docs/guides/workflow/index.md +++ b/docs/guides/workflow/index.md @@ -94,3 +94,141 @@ The execution stack: Each function's result is cached and passed to the next function in the chain. Job functions within a single workflow execute **sequentially**, not in parallel. Each function completes before the next one starts. For concurrent execution, you can start multiple workflows asynchronously using `tailor.workflow.triggerWorkflow()`. + +## Concurrency Control + +In addition to the [platform-wide caps](/reference/platform/platform-limits#workflow-concurrency-limits) (50 per workspace, 20 per workflow), workflows can declare two independent concurrency policies. Neither rejects excess work, but they defer it in different ways: the workflow-level policy keeps new executions in `PENDING` at the scheduler, and the job function execution policy suspends the running workflow to `PENDING_RESUME` at dispatch time. Both resume automatically as slots free up. + +### Workflow-level Concurrency Policy + +Set `concurrencyPolicy.maxConcurrentExecutions` on the workflow definition to cap how many executions of the same workflow may run at once. + +- Enforced by the **scheduler** when it picks up `PENDING` executions. Executions that would exceed the cap stay `PENDING` and are re-evaluated on the next scheduler tick. +- Scoped per workflow definition. Other workflows in the same workspace are unaffected. +- Applies to every entry point (`triggerWorkflow`, executor triggers, CLI `workflow start`) equally. + +```typescript {{ title: 'workflows/import-orders.ts' }} +import { createWorkflow } from "@tailor-platform/sdk"; +import { importOrders } from "./jobs/import-orders"; + +export default createWorkflow({ + name: "import-orders", + mainJob: importOrders, + concurrencyPolicy: { + // At most 3 executions of `import-orders` run concurrently across the + // workspace. The 4th trigger stays PENDING until one of the running + // executions finishes. + maxConcurrentExecutions: 3, + }, +}); +``` + +See [Concurrency Policy](/sdk/services/workflow#concurrency-policy) in the SDK Workflow reference for the SDK API. + +### Job Function Execution Policies + +Declare workspace-scoped execution policies with a per-key `maxConcurrentExecutions` cap, then route job function dispatches through them by passing `executionPolicyKey` on `job.trigger()` / `tailor.workflow.triggerJobFunction()`. + +- Enforced by the **runner** at dispatch time — a separate mechanism from the scheduler-level workflow cap above. The two stack: a workflow that is allowed to start can still have its job function dispatches suspended by an execution policy. +- Dispatches that would exceed the cap are suspended and resume automatically as slots free up. +- A dispatch without `executionPolicyKey` is unaffected by all declared policies, but still counts against the [platform workspace-wide job function limit](/reference/platform/platform-limits#workflow-job-function-concurrency-limits). +- User policies stack **on top of** the platform hard limits (workspace-wide 100 dispatches, per-key fallback 50 for policies without a user-defined cap). The most restrictive applicable cap wins. + +**Declare policies and register them on the SDK config:** + +```typescript {{ title: 'workflows/policies.ts' }} +import { defineWorkflowExecutionPolicies } from "@tailor-platform/sdk"; + +export const executionPolicies = defineWorkflowExecutionPolicies((define) => ({ + // Exact-key policy: one shared pool for dispatches keyed "premium". + premium: define({ concurrencyPolicy: { maxConcurrentExecutions: 5 } }), + // Wildcard policy: independent pool of size 3 per resolved key + // (`tenant-api.acme`, `tenant-api.beta`, ...). + tenantApi: define({ + name: "tenant-api", + matchType: "prefix", + concurrencyPolicy: { maxConcurrentExecutions: 3 }, + }), +})); +``` + +```typescript {{ title: 'tailor.config.ts' }} +import { defineConfig } from "@tailor-platform/sdk"; +import { executionPolicies } from "./workflows/policies"; + +export default defineConfig({ + workflow: { + files: ["workflows/**/*.ts"], + executionPolicies, + }, +}); +``` + +**Route a dispatch through a policy:** + +```typescript {{ title: 'workflows/jobs/sync-tenant.ts' }} +import { createWorkflowJob } from "@tailor-platform/sdk"; +import { executionPolicies } from "../policies"; +import { syncOrders } from "./sync-orders"; +import { pushMetrics } from "./push-metrics"; + +export const syncTenant = createWorkflowJob({ + name: "sync-tenant", + body: async (input: { tenantId: string }) => { + // Exact-key policy: pass `.key` directly (typed). + await pushMetrics.trigger( + { tenantId: input.tenantId }, + { executionPolicyKey: executionPolicies.premium.key }, + ); + + // Wildcard policy: build the concrete key with `.keyFor(suffix)`. + // Resolves to e.g. "tenant-api.acme"; each tenant gets its own pool of 3. + await syncOrders.trigger( + { tenantId: input.tenantId }, + { executionPolicyKey: executionPolicies.tenantApi.keyFor(input.tenantId) }, + ); + }, +}); +``` + +The same `executionPolicyKey` option is available on `tailor.workflow.triggerJobFunction(name, args, options)` when dispatching by name from a Function-service script. + +**Matching modes:** + +| Match type | Declaration | Applies to | Pool granularity | +| ------------------ | ---------------------------------------- | ------------------------------------------- | ---------------------------------------- | +| Exact (default) | `matchType: "exact"` (or omitted) | Dispatches whose key equals the policy key | One pool shared by the exact key | +| Prefix (wildcard) | `matchType: "prefix"` | Dispatches whose key **starts with** the policy key | One independent pool **per resolved key** | + +For wildcard policies, the platform registers the prefix with a trailing `*` and gives every concrete resolved key its own pool of the declared size. In the example above, `tenant-api` with `maxConcurrentExecutions: 3` allows **3 concurrent dispatches per tenant key** — `tenant-api.acme`, `tenant-api.beta`, and `tenant-api.gamma` each run up to 3 in parallel independently, not 3 across all of them combined. + +**Overlapping policies stack (AND-of-caps).** When a dispatch key is covered by more than one policy (any mix of exact and wildcard prefixes), every covering policy applies — the dispatch acquires one slot in each matching pool, and any single saturated cap blocks it. + +```typescript {{ title: 'workflows/policies.ts' }} +export const executionPolicies = defineWorkflowExecutionPolicies((define) => ({ + // Broad safety net: at most 100 dispatches under the "one" tree at once. + one: define({ + name: "one", + matchType: "prefix", + concurrencyPolicy: { maxConcurrentExecutions: 100 }, + }), + // Narrower cap for the "one.two" subtree. + oneTwo: define({ + name: "one.two", + matchType: "prefix", + concurrencyPolicy: { maxConcurrentExecutions: 10 }, + }), + // Even tighter cap for one exact key. + oneTwoThree: define({ + name: "one.two.three", + concurrencyPolicy: { maxConcurrentExecutions: 3 }, + }), +})); + +// A dispatch with executionPolicyKey: "one.two.three" counts against all +// three pools simultaneously; the tightest cap (3) is what actually blocks. +// The broader `one*` and `one.two*` caps still guard the whole subtree, so a +// narrower policy cannot silently disable them. +``` + +See [Execution Policies](/sdk/services/workflow#execution-policies) in the SDK Workflow reference for the full declaration API, key grammar, and options for customizing `keyFor()`. diff --git a/docs/reference/platform/platform-limits.md b/docs/reference/platform/platform-limits.md index eb77135..028ccd0 100644 --- a/docs/reference/platform/platform-limits.md +++ b/docs/reference/platform/platform-limits.md @@ -13,6 +13,8 @@ Platform limits are enforced across different services in the Tailor Platform to | Recursive Call Limit | Call Depth | 10 levels | Max depth for nested platform-to-platform requests (pipelines, functions, etc.) | Request rejected with BadRequest error if depth exceeds 10 levels | | Workflow | Workspace Concurrent Executions | 50 executions | Max concurrent workflow executions per workspace | Pending executions remain in `PENDING` status until running executions drop below the cap | | Workflow | Per-Workflow Concurrent Executions | 20 executions | Max concurrent executions of a single workflow | Pending executions remain in `PENDING` status until running executions drop below the cap | +| Workflow | Workspace Concurrent Job Functions | 100 dispatches | Max concurrent job function dispatches per workspace (regardless of `executionPolicyKey`) | Dispatch is suspended and workflow moves to `PENDING_RESUME` until dispatches drop below the cap | +| Workflow | Per-Key Concurrent Job Functions (fallback) | 50 dispatches | Platform fallback per `executionPolicyKey`, applied only when a matching execution policy has no user-defined `concurrencyPolicy.maxConcurrentExecutions` | Dispatch is suspended and workflow moves to `PENDING_RESUME` until dispatches drop below the cap | | Executor | Workspace Concurrent Job Function Operations | 100 executions | Max concurrently running job function operations per workspace | Excess executions remain pending and start automatically, oldest first, as running executions complete | | Function | Memory | 32 MB | Max memory available to a Function execution | Execution terminated with `Memory limit exceeded` error if usage exceeds 32 MB | | JobFunction | Memory | 256 MB | Max memory available to a JobFunction execution | Execution terminated with `Memory limit exceeded` error if usage exceeds 256 MB | @@ -60,6 +62,25 @@ When either limit is reached, new executions are not rejected. Instead, they rem - If the workspace-wide limit is reached, no new executions start in that workspace regardless of per-workflow counts. - If the per-workflow limit is reached for a specific workflow, other workflows in the same workspace can still start new executions (as long as the workspace-wide limit is not reached). +## Workflow Job Function Concurrency Limits + +Separately from the scheduler-level workflow caps above, the workflow runner enforces platform hard limits on individual job function dispatches. These apply on top of any [user-declared execution policies](/guides/workflow/#job-function-execution-policies) and cannot be raised by workflow authors. + +### How It Works + +Two platform hard limits guard job function dispatches: + +- **Workspace-wide limit (100)**: Caps the total number of concurrently running job function dispatches within a single workspace. Applies to every dispatch regardless of whether `executionPolicyKey` is set. +- **Per-key fallback limit (50)**: A safety net that only kicks in when a dispatch supplies an `executionPolicyKey` and the matching execution policy does not declare a user-defined `concurrencyPolicy.maxConcurrentExecutions`. Once a policy sets its own cap, that user cap replaces this fallback and the platform per-key limit is no longer consulted. + +When a dispatch hits any of these caps (or a user-declared execution policy cap), the workflow is suspended to `PENDING_RESUME` and retried after a short delay (default 5s). + +### Behavior + +- Both platform limits and user-declared execution policies are enforced independently. A dispatch must satisfy all applicable caps to run. +- The workspace-wide job function limit is independent of the workflow-execution caps above — a workflow that is already `RUNNING` can still have its dispatches suspended when the workspace hits the 100-dispatch ceiling. +- The per-key fallback protects operators from accidentally leaving a key unbounded when they only declare the policy to register a valid key without a user-defined cap. + ## Executor Job Function Concurrency Limit The Tailor Platform limits the number of concurrently running Executor [job function operations](/guides/executor/job-function-operation) to **100 per workspace**. This ensures fair scheduling across workspaces: a sudden spike of executions in one workspace does not delay job executions in other workspaces.