Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
800 changes: 0 additions & 800 deletions .jaiph/language_redesign_spec.md

This file was deleted.

26 changes: 16 additions & 10 deletions CHANGELOG.md

Large diffs are not rendered by default.

45 changes: 5 additions & 40 deletions QUEUE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,47 +4,12 @@ Process rules:

1. Tasks are executed top-to-bottom.
2. The first `##` section is always the current task.
3. When a task is completed, remove that section entirely.
4. Every task must be standalone: no hidden assumptions, no "read prior task" dependency.
5. This queue assumes **hard rewrite semantics**:
3. Task that is ready for implementation is marked with `#dev-ready` at the end of the header.
4. When a task is completed, remove that section entirely.
5. Every task must be standalone: no hidden assumptions, no "read prior task" dependency.
6. This queue assumes **hard rewrite semantics**:
* breaking changes are allowed,
* backward compatibility is **not** a design goal unless a task explicitly says otherwise.
6. **Acceptance criteria are non-negotiable.** A task is not done until every acceptance bullet is verified by a test that fails when the contract is violated. "It works on my machine" or "the existing tests pass" is not acceptance.

***

## Performance — investigate and fix slow installation

**Goal**
`jaiph install` (and related dependency or bootstrap steps) feels unreasonably slow; find the dominant cost and improve it without weakening reproducibility (lockfile, shallow clone behavior, etc.).

**Scope**

* Profile or instrument the install path (git clone, lockfile I/O, post-install) and document the top 1–3 contributors to latency.
* Implement targeted fixes (e.g. avoid redundant work, reduce subprocess churn, cache safely) and verify wall-clock improvement on a cold and warm run where applicable.

**Acceptance criteria**

* A short note in the commit or PR description states what was slow and what changed, with before/after rough timings on the same machine.
* `jaiph install` behavior remains correct: same lockfile semantics and failure modes for bad URLs or missing refs.
* `npm test` passes.

***

## Performance — investigate and fix slow workflow start (initial 2–4 s lag)

**Goal**
When starting workflows (e.g. `jaiph run` / first step), users observe a 2–4 second delay before useful work; reduce that lag or explain and eliminate unnecessary startup work (JIT, imports, process spawn, discovery).

**Scope**

* Reproduce the lag with a minimal `.jh` workflow; trace Node startup, module load, and runtime init (`NodeWorkflowRuntime` and friends).
* Address fixable costs (e.g. defer heavy work, lazy imports, avoid redundant file scans) without changing user-visible workflow semantics.

**Acceptance criteria**

* Documented repro (command + minimal file) and what was measured (time to first event / first step).
* Measurable reduction in the cold-start path on a representative case, or a clear justification if the lag is irreducible (e.g. external subprocess).
* `npm test` passes.
7. **Acceptance criteria are non-negotiable.** A task is not done until every acceptance bullet is verified by a test that fails when the contract is violated. "It works on my machine" or "the existing tests pass" is not acceptance.

***
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
- **Parser** (`src/parser.ts`, `src/parse/*`) — `.jh` / `.test.jh` → AST.
- **Validator** (`src/transpile/validate.ts`) — imports and symbol references at compile time.
- **Transpiler** (`src/transpile/*`) — emits atomic `script` files under `scripts/` only (no workflow-level shell).
- **Node workflow runtime** (`src/runtime/kernel/node-workflow-runtime.ts`, `graph.ts`) — interprets the AST; `buildRuntimeGraph()` is parse-only across imports.
- **Node workflow runtime** (`src/runtime/kernel/node-workflow-runtime.ts`, `graph.ts`) — interprets the AST; `buildRuntimeGraph(graph)` consumes the `ModuleGraph` produced by `loadModuleGraph` (no filesystem reads).
- **Node test runner** (`src/runtime/kernel/node-test-runner.ts`) — `*.test.jh` blocks with mocks.
- **JS kernel** (`src/runtime/kernel/`) — prompts, managed scripts, `__JAIPH_EVENT__`, inbox, mocks.
Diagrams, runtime contracts, on-disk artifact layout, and distribution: **[Architecture](docs/architecture.md)**. Test layers and E2E policy: **[Contributing](docs/contributing.md)**.
Expand Down
347 changes: 347 additions & 0 deletions design/2026-05-15-parser-compiler-simplification.md

Large diffs are not rendered by default.

123 changes: 86 additions & 37 deletions docs/architecture.md

Large diffs are not rendered by default.

19 changes: 13 additions & 6 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,9 +94,11 @@ If a `.jh` file is executable and has `#!/usr/bin/env jaiph`, you can run it dir

### Compile-time and process model

The CLI runs `buildScripts()`, which walks the entry file and its import closure. Each reachable module is parsed and `validateReferences` runs before script files are written. Unrelated `.jh` files on disk are not read.
The default `jaiph run` path parses each reachable `.jh` **once**. The CLI calls **`loadModuleGraph`** (`src/transpile/module-graph.ts`) to walk the entry plus its transitive `import` closure, producing a **`ModuleGraph`** record (`{ entryFile, workspaceRoot?, modules: Map<absPath, { filePath, ast, imports }> }`). `parsejaiph(source, filePath)` is itself I/O-pure — `loadModuleGraph` is the only routine that reads `.jh` sources from disk. The entry AST is reused for the banner (`metadataToConfig`), and the same graph is passed to **`buildScriptsFromGraph(graph, outDir)`**, which calls `emitScriptsForModuleFromGraph` per reachable module and writes atomic `script` files. `validateReferences(graph)` runs against the in-memory ASTs — neither validation nor emission re-reads `.jh` files. Unrelated `.jh` files on disk are not read.

After validation, the CLI spawns the Node workflow runner as a detached child. The runner loads the graph with `buildRuntimeGraph()` (parse-only imports; no `validateReferences` here) and executes `NodeWorkflowRuntime`. Prompt steps, script subprocesses, inbox dispatch, and event emission are handled in the runtime kernel — workflows and rules are interpreted in-process; only `script` steps spawn a managed shell. The CLI listens on stderr for `__JAIPH_EVENT__` JSON lines, the single event channel for all execution modes. Stdout carries only plain script output, forwarded to the terminal as-is.
After validation, the CLI spawns the Node workflow runner as a detached child. For local (non-Docker) runs the CLI serializes the graph to `<outDir>/.jaiph-module-graph.json` with `writeModuleGraph` (deterministic JSON: entries sorted by absolute path, ASTs included verbatim) and points the child at it through the internal env var **`JAIPH_MODULE_GRAPH_FILE`**. The runner deserializes the file with `readModuleGraph` and passes the result to `buildRuntimeGraph(graph)`, which produces the `RuntimeGraph` (a type alias for `ModuleGraph`) by injecting `ScriptDef` stubs for `import script` declarations — without touching disk. When the env var is absent — Docker `jaiph run`, `jaiph run --raw`, `jaiph test`, or any other caller — the runner falls back to `loadModuleGraph(sourceFile, workspaceRoot)` on the source file. Either path runs `NodeWorkflowRuntime` with the same `RuntimeGraph` shape — `buildRuntimeGraph` still does **not** run `validateReferences`. Prompt steps, script subprocesses, inbox dispatch, and event emission are handled in the runtime kernel — workflows and rules are interpreted in-process; only `script` steps spawn a managed shell. The CLI listens on stderr for `__JAIPH_EVENT__` JSON lines, the single event channel for all execution modes. Stdout carries only plain script output, forwarded to the terminal as-is.

For the full data flow across the parent → child process boundary, see [Architecture — Local module graph](architecture.md#local-module-graph).

### Run progress and tree output

Expand Down Expand Up @@ -274,7 +276,7 @@ jaiph test e2e/say_hello.test.jh

## `jaiph compile`

Parse modules and run **`validateReferences`** (the same compile-time checks as before `jaiph run`) **without** writing `scripts/`, **without** calling **`buildRuntimeGraph`**, and **without** spawning the workflow runner. Use this for CI gates, pre-commit hooks, or editor diagnostics.
Parse modules and run the same compile-time validation as before `jaiph run` **without** writing `scripts/`, **without** calling **`buildRuntimeGraph`**, and **without** spawning the workflow runner. Use this for CI gates, pre-commit hooks, or editor diagnostics.

```bash
jaiph compile [--json] [--workspace <dir>] <file.jh | directory> ...
Expand All @@ -286,9 +288,11 @@ At least one path is required. **`jaiph compile -h`** or **`jaiph compile --help

**Directory arguments** — The tree is scanned for `*.jh` files whose basename is **not** `*.test.jh` (same rule as `walkjhFiles` in the transpiler: files like `foo.test.jh` are skipped). Each non-test `*.jh` under the tree is treated as an entrypoint and its closure merged into the same validation set. To validate a test module’s graph explicitly, pass that **`*.test.jh` file** as a path (directories never pick up `*.test.jh` as roots).

**Multiple-error reporting.** `jaiph compile` aggregates **all** recoverable validation errors across the import closure before exiting, rather than stopping at the first failure. Internally it calls **`collectDiagnostics(graph)`** (`src/transpile/validate.ts`), which walks every reachable module and returns a `Diagnostics` collector (`src/diagnostics.ts`) populated with every error the validator accumulated through `diag.error(...)` and `diag.capture(...)`. Output is sorted by `(file, line, col)` so a single compile cycle surfaces independent errors together — for example, a duplicate `import` alias on line 2, an undefined channel in a `send` on line 6, and an unknown `run` target on line 7 all appear in one report. **Fatal** errors (parser failures like an unterminated triple-quote, loader failures, etc.) still abort the closure for the affected entry — `jaiph compile` reports them as a single diagnostic for that entry and continues with the next entry. Any non-empty diagnostic set exits **1**.

**Flags:**

- **`--json`** — On success, print `[]` to stdout. On failure, print one JSON **array** of objects `{ "file", "line", "col", "code", "message" }` to stdout and exit **1** (non-JSON errors use a synthetic `E_COMPILE` object when the message is not in `file:line:col CODE …` form).
- **`--json`** — On success, print `[]` to stdout. On failure, print **one** JSON **array** containing every collected diagnostic — objects `{ "file", "line", "col", "code", "message" }` to stdout and exit **1** (non-JSON errors use a synthetic `E_COMPILE` object when the message is not in `file:line:col CODE …` form). Without `--json`, the same set is written to **stderr** as one `path:line:col CODE message` line per diagnostic, in the same sorted order.
- **`--workspace <dir>`** — Override the workspace root used for **library import resolution** (`<workspace>/.jaiph/libs/`, etc.) for **all** modules reached from the given paths. When omitted, the workspace is **auto-detected** from each path’s location (`detectWorkspaceRoot` — same algorithm as `jaiph run`, starting from the file’s directory or from a directory argument).

## `jaiph format`
Expand Down Expand Up @@ -348,9 +352,11 @@ jaiph install [--force]

**With arguments** — clone each repo into `.jaiph/libs/<name>/` (shallow: `--depth 1`) and upsert the entry in `.jaiph/libs.lock`. The library name is derived from the URL: last path segment, stripped of `.git` suffix (e.g. `github.com/you/queue-lib.git` → `queue-lib`). Version pinning is usually written as **`https://…/name.git@<tag-or-branch>`**; other URL shapes with a trailing **`@ref`** are also accepted when the parser can split URL and version unambiguously.

**Without arguments** — restore all libraries from `.jaiph/libs.lock`. Useful after cloning a project or in CI. If the lockfile exists but lists **no** libraries, the command prints `No libs in lockfile.` and exits **0**.
**Without arguments** — restore all libraries from `.jaiph/libs.lock`. Useful after cloning a project or in CI. If the lockfile exists but lists **no** libraries, the command prints `No libs in lockfile.` and exits **0**. Restore mode does **not** invent new lock entries — the lockfile is read but not rewritten.

If `.jaiph/libs/<name>/` already exists, the library is skipped without invoking `git` (warm path) — both for explicit arguments and for restore-from-lock. Use **`--force`** (anywhere in the argument list) to delete and re-clone.

If `.jaiph/libs/<name>/` already exists, the library is skipped. Use **`--force`** (anywhere in the argument list) to delete and re-clone.
**Parallel clones.** Missing libraries are cloned concurrently with a small bounded-concurrency executor (default **4 in flight**); the warm-path skip runs in a pre-pass before any clone work starts. Independent network/process latency therefore overlaps when several libraries are missing. Failures from individual clones still propagate: any non-zero clone exits the command non-zero, and failed libraries are **not** added to `.jaiph/libs.lock`. Successful and warm-skipped libraries are upserted as before.

**Lockfile format** (`.jaiph/libs.lock`):

Expand Down Expand Up @@ -419,6 +425,7 @@ These variables apply to `jaiph run` and workflow execution. Variables marked **
- `JAIPH_META_FILE` — path to the run metadata file (under the CLI’s build output directory for that invocation). Set on the **detached workflow child** only; the parent strips any inherited value so leftover exports do not collide. The runner writes `run_dir=` / `summary_file=` lines for the host to read after exit.
- `JAIPH_SOURCE_ABS` — absolute path to the entry `.jh` file; set by the CLI for **`jaiph run`** before spawn. Required by the runner (local and Docker).
- `JAIPH_SCRIPTS` — directory containing emitted **`script`** files for this run; set after **`buildScripts()`**. Any **`JAIPH_SCRIPTS`** exported in the parent shell is cleared before launch so nested toolchains do not point at the wrong tree.
- `JAIPH_MODULE_GRAPH_FILE` — absolute path to a `ModuleGraph` JSON snapshot (`<outDir>/.jaiph-module-graph.json`) the CLI wrote with `writeModuleGraph`. Set by the CLI **only** for the default local (non-Docker, non-`--raw`) `jaiph run` path so the spawned `node-workflow-runner.js` builds the runtime graph from the cached ASTs instead of re-reading the import closure. The file is internal and may move; do not depend on its path or contents. When the variable is absent (Docker `jaiph run`, `jaiph run --raw`, `jaiph test`), `buildRuntimeGraph()` falls back to `loadModuleGraph` on disk. See [Architecture — Local module graph](architecture.md#local-module-graph).
- `JAIPH_RUN_DIR`, `JAIPH_RUN_ID`, `JAIPH_RUN_SUMMARY_FILE` — for a normal (**non-raw**) **`jaiph run`**, the host generates **`JAIPH_RUN_ID`** once per invocation (UUID), passes it through to the detached child (and into Docker when sandboxed), and Docker failure-path discovery can match summaries by this id. The runtime uses **`JAIPH_RUN_ID`** as the stable run identifier; if it is absent, the runtime may assign its own UUID. **`JAIPH_RUN_DIR`** and **`JAIPH_RUN_SUMMARY_FILE`** are set inside the runner once the UTC run directory exists.
- `JAIPH_SOURCE_FILE` — set automatically by the CLI to the entry file **basename**. Used to name run directories (see [Architecture — Durable artifact layout](architecture.md#durable-artifact-layout)).

Expand Down
Loading
Loading