Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion docs/guides/function/builtin-interfaces.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,12 +166,22 @@ const executionId = await tailor.workflow.triggerWorkflow(
const result = await tailor.workflow.triggerJobFunction("calculateTax", {
amount: 1000,
});

// Route the dispatch through a workspace-registered execution policy for
// per-key concurrency control (see the SDK Workflow guide for policy setup).
const scoped = await tailor.workflow.triggerJobFunction(
"syncTenant",
{ tenantId: "acme" },
{ executionPolicyKey: `tenant-api.acme` },
);
```

| Function | Returns | Description |
| --- | --- | --- |
| `triggerWorkflow(name, args?, options?)` | `Promise<string>` | Trigger a workflow. Returns the execution ID |
| `triggerJobFunction(name, args?)` | `any` | Trigger a job function and return its result |
| `triggerJobFunction(name, args?, options?)` | `Promise<any>` | Trigger a job function and return its result. `options.executionPolicyKey` routes the dispatch through a matching execution policy for per-key concurrency control |

For details on declaring execution policies and the key grammar, see [Execution Policies](/sdk/services/workflow#execution-policies) in the SDK Workflow reference.

## TailorDB Client

Expand Down
138 changes: 138 additions & 0 deletions docs/guides/workflow/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,3 +94,141 @@ The execution stack:
Each function's result is cached and passed to the next function in the chain.

Job functions within a single workflow execute **sequentially**, not in parallel. Each function completes before the next one starts. For concurrent execution, you can start multiple workflows asynchronously using `tailor.workflow.triggerWorkflow()`.

## Concurrency Control

In addition to the [platform-wide caps](/reference/platform/platform-limits#workflow-concurrency-limits) (50 per workspace, 20 per workflow), workflows can declare two independent concurrency policies. Neither rejects excess work, but they defer it in different ways: the workflow-level policy keeps new executions in `PENDING` at the scheduler, and the job function execution policy suspends the running workflow to `PENDING_RESUME` at dispatch time. Both resume automatically as slots free up.

### Workflow-level Concurrency Policy

Set `concurrencyPolicy.maxConcurrentExecutions` on the workflow definition to cap how many executions of the same workflow may run at once.

- Enforced by the **scheduler** when it picks up `PENDING` executions. Executions that would exceed the cap stay `PENDING` and are re-evaluated on the next scheduler tick.
- Scoped per workflow definition. Other workflows in the same workspace are unaffected.
- Applies to every entry point (`triggerWorkflow`, executor triggers, CLI `workflow start`) equally.

```typescript {{ title: 'workflows/import-orders.ts' }}
import { createWorkflow } from "@tailor-platform/sdk";
import { importOrders } from "./jobs/import-orders";

export default createWorkflow({
name: "import-orders",
mainJob: importOrders,
concurrencyPolicy: {
// At most 3 executions of `import-orders` run concurrently across the
// workspace. The 4th trigger stays PENDING until one of the running
// executions finishes.
maxConcurrentExecutions: 3,
},
});
```

See [Concurrency Policy](/sdk/services/workflow#concurrency-policy) in the SDK Workflow reference for the SDK API.

### Job Function Execution Policies

Declare workspace-scoped execution policies with a per-key `maxConcurrentExecutions` cap, then route job function dispatches through them by passing `executionPolicyKey` on `job.trigger()` / `tailor.workflow.triggerJobFunction()`.

- Enforced by the **runner** at dispatch time — a separate mechanism from the scheduler-level workflow cap above. The two stack: a workflow that is allowed to start can still have its job function dispatches suspended by an execution policy.
- Dispatches that would exceed the cap are suspended and resume automatically as slots free up.
- A dispatch without `executionPolicyKey` is unaffected by all declared policies, but still counts against the [platform workspace-wide job function limit](/reference/platform/platform-limits#workflow-job-function-concurrency-limits).
- User policies stack **on top of** the platform hard limits (workspace-wide 100 dispatches, per-key fallback 50 for policies without a user-defined cap). The most restrictive applicable cap wins.

**Declare policies and register them on the SDK config:**

```typescript {{ title: 'workflows/policies.ts' }}
import { defineWorkflowExecutionPolicies } from "@tailor-platform/sdk";

export const executionPolicies = defineWorkflowExecutionPolicies((define) => ({
// Exact-key policy: one shared pool for dispatches keyed "premium".
premium: define({ concurrencyPolicy: { maxConcurrentExecutions: 5 } }),
// Wildcard policy: independent pool of size 3 per resolved key
// (`tenant-api.acme`, `tenant-api.beta`, ...).
tenantApi: define({
name: "tenant-api",
matchType: "prefix",
concurrencyPolicy: { maxConcurrentExecutions: 3 },
}),
}));
```

```typescript {{ title: 'tailor.config.ts' }}
import { defineConfig } from "@tailor-platform/sdk";
import { executionPolicies } from "./workflows/policies";

export default defineConfig({
workflow: {
files: ["workflows/**/*.ts"],
executionPolicies,
},
});
```

**Route a dispatch through a policy:**

```typescript {{ title: 'workflows/jobs/sync-tenant.ts' }}
import { createWorkflowJob } from "@tailor-platform/sdk";
import { executionPolicies } from "../policies";
import { syncOrders } from "./sync-orders";
import { pushMetrics } from "./push-metrics";

export const syncTenant = createWorkflowJob({
name: "sync-tenant",
body: async (input: { tenantId: string }) => {
// Exact-key policy: pass `.key` directly (typed).
await pushMetrics.trigger(
{ tenantId: input.tenantId },
{ executionPolicyKey: executionPolicies.premium.key },
);

// Wildcard policy: build the concrete key with `.keyFor(suffix)`.
// Resolves to e.g. "tenant-api.acme"; each tenant gets its own pool of 3.
await syncOrders.trigger(
{ tenantId: input.tenantId },
{ executionPolicyKey: executionPolicies.tenantApi.keyFor(input.tenantId) },
);
},
});
```

The same `executionPolicyKey` option is available on `tailor.workflow.triggerJobFunction(name, args, options)` when dispatching by name from a Function-service script.

**Matching modes:**

| Match type | Declaration | Applies to | Pool granularity |
| ------------------ | ---------------------------------------- | ------------------------------------------- | ---------------------------------------- |
| Exact (default) | `matchType: "exact"` (or omitted) | Dispatches whose key equals the policy key | One pool shared by the exact key |
| Prefix (wildcard) | `matchType: "prefix"` | Dispatches whose key **starts with** the policy key | One independent pool **per resolved key** |

For wildcard policies, the platform registers the prefix with a trailing `*` and gives every concrete resolved key its own pool of the declared size. In the example above, `tenant-api` with `maxConcurrentExecutions: 3` allows **3 concurrent dispatches per tenant key** — `tenant-api.acme`, `tenant-api.beta`, and `tenant-api.gamma` each run up to 3 in parallel independently, not 3 across all of them combined.

**Overlapping policies stack (AND-of-caps).** When a dispatch key is covered by more than one policy (any mix of exact and wildcard prefixes), every covering policy applies — the dispatch acquires one slot in each matching pool, and any single saturated cap blocks it.

Comment thread
k1LoW marked this conversation as resolved.
Comment thread
k1LoW marked this conversation as resolved.
```typescript {{ title: 'workflows/policies.ts' }}
export const executionPolicies = defineWorkflowExecutionPolicies((define) => ({
// Broad safety net: at most 100 dispatches under the "one" tree at once.
one: define({
name: "one",
matchType: "prefix",
concurrencyPolicy: { maxConcurrentExecutions: 100 },
}),
// Narrower cap for the "one.two" subtree.
oneTwo: define({
name: "one.two",
matchType: "prefix",
concurrencyPolicy: { maxConcurrentExecutions: 10 },
}),
// Even tighter cap for one exact key.
oneTwoThree: define({
name: "one.two.three",
concurrencyPolicy: { maxConcurrentExecutions: 3 },
}),
}));

// A dispatch with executionPolicyKey: "one.two.three" counts against all
// three pools simultaneously; the tightest cap (3) is what actually blocks.
// The broader `one*` and `one.two*` caps still guard the whole subtree, so a
// narrower policy cannot silently disable them.
```

See [Execution Policies](/sdk/services/workflow#execution-policies) in the SDK Workflow reference for the full declaration API, key grammar, and options for customizing `keyFor()`.
21 changes: 21 additions & 0 deletions docs/reference/platform/platform-limits.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ Platform limits are enforced across different services in the Tailor Platform to
| Recursive Call Limit | Call Depth | 10 levels | Max depth for nested platform-to-platform requests (pipelines, functions, etc.) | Request rejected with BadRequest error if depth exceeds 10 levels |
| Workflow | Workspace Concurrent Executions | 50 executions | Max concurrent workflow executions per workspace | Pending executions remain in `PENDING` status until running executions drop below the cap |
| Workflow | Per-Workflow Concurrent Executions | 20 executions | Max concurrent executions of a single workflow | Pending executions remain in `PENDING` status until running executions drop below the cap |
| Workflow | Workspace Concurrent Job Functions | 100 dispatches | Max concurrent job function dispatches per workspace (regardless of `executionPolicyKey`) | Dispatch is suspended and workflow moves to `PENDING_RESUME` until dispatches drop below the cap |
| Workflow | Per-Key Concurrent Job Functions (fallback) | 50 dispatches | Platform fallback per `executionPolicyKey`, applied only when a matching execution policy has no user-defined `concurrencyPolicy.maxConcurrentExecutions` | Dispatch is suspended and workflow moves to `PENDING_RESUME` until dispatches drop below the cap |
| Executor | Workspace Concurrent Job Function Operations | 100 executions | Max concurrently running job function operations per workspace | Excess executions remain pending and start automatically, oldest first, as running executions complete |
| Function | Memory | 32 MB | Max memory available to a Function execution | Execution terminated with `Memory limit exceeded` error if usage exceeds 32 MB |
| JobFunction | Memory | 256 MB | Max memory available to a JobFunction execution | Execution terminated with `Memory limit exceeded` error if usage exceeds 256 MB |
Expand Down Expand Up @@ -60,6 +62,25 @@ When either limit is reached, new executions are not rejected. Instead, they rem
- If the workspace-wide limit is reached, no new executions start in that workspace regardless of per-workflow counts.
- If the per-workflow limit is reached for a specific workflow, other workflows in the same workspace can still start new executions (as long as the workspace-wide limit is not reached).

## Workflow Job Function Concurrency Limits

Separately from the scheduler-level workflow caps above, the workflow runner enforces platform hard limits on individual job function dispatches. These apply on top of any [user-declared execution policies](/guides/workflow/#job-function-execution-policies) and cannot be raised by workflow authors.

### How It Works

Two platform hard limits guard job function dispatches:

- **Workspace-wide limit (100)**: Caps the total number of concurrently running job function dispatches within a single workspace. Applies to every dispatch regardless of whether `executionPolicyKey` is set.
- **Per-key fallback limit (50)**: A safety net that only kicks in when a dispatch supplies an `executionPolicyKey` and the matching execution policy does not declare a user-defined `concurrencyPolicy.maxConcurrentExecutions`. Once a policy sets its own cap, that user cap replaces this fallback and the platform per-key limit is no longer consulted.

When a dispatch hits any of these caps (or a user-declared execution policy cap), the workflow is suspended to `PENDING_RESUME` and retried after a short delay (default 5s).

### Behavior

- Both platform limits and user-declared execution policies are enforced independently. A dispatch must satisfy all applicable caps to run.
- The workspace-wide job function limit is independent of the workflow-execution caps above — a workflow that is already `RUNNING` can still have its dispatches suspended when the workspace hits the 100-dispatch ceiling.
- The per-key fallback protects operators from accidentally leaving a key unbounded when they only declare the policy to register a valid key without a user-defined cap.

## Executor Job Function Concurrency Limit

The Tailor Platform limits the number of concurrently running Executor [job function operations](/guides/executor/job-function-operation) to **100 per workspace**. This ensures fair scheduling across workspaces: a sudden spike of executions in one workspace does not delay job executions in other workspaces.
Expand Down
Loading