From 296006af3dd6789fb1559c55b094eda4309dac40 Mon Sep 17 00:00:00 2001 From: Assaf Sapir Date: Sat, 27 Jun 2026 17:50:22 +0300 Subject: [PATCH] M3: Closures (`=` by-value / `:=` by-reference capture) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add lexical closures whose capture mode is inferred from the binding operator, mirroring Quilon's existing mutability rule: - `=` bindings are captured BY VALUE (a frozen, read-only snapshot). - `:=` bindings are captured BY REFERENCE — a single shared GC-boxed cell, so writes escape the closure and persist/accumulate across calls. There is no capture list and no marker; the operator that bound the name is the signal. Closures are monomorphic (concrete-typed params/captures) — generic and polymorphic-capturing closures are deferred to M4. Pipeline: - AST: new `Expr::Lambda` function-literal node. - Parser: lambda expressions (`x => …`, `(a,b) => …`, `() => …`), with a one-shot `no_lambda` guard so a for-loop collection (`for n <- xs => body`) is not mis-parsed as `xs => body`. - Checker: `check_lambda` infers `Type::Function`; the body is checked in a layered scope so capture is visible and the mutability gate still applies. - Codegen: a closure value is `{ ptr fn, ptr env }`; the lifted function takes the captured environment as a trailing pointer. `=` captures are copied into the env by value; `:=` captures are heap-boxed via `__alloc` and shared. A non-capturing nested function is emitted as a plain module function so it can recurse; captures are threaded through arbitrary nesting depth, and a closure value may itself be captured by another closure and called. Tests/docs: - examples/closures.ql (counter via `:=` cell + adder via `=` value -> exit 42), wired into the examples gate (JIT + native AOT under clang and gcc). - tests/closures_test.rs covers accumulation, frozen-snapshot, shared cells, mixed capture, shadowing, nested recursion, and multi-level capture. - LANGUAGE.md: Closures section + feature matrix + limitations boundary. Co-Authored-By: Claude Opus 4.8 (1M context) --- LANGUAGE.md | 47 ++- examples/closures.ql | 27 ++ src/ast/captures.rs | 188 +++++++++ src/ast/mod.rs | 1 + src/ast/nodes.rs | 15 + src/codegen/generator.rs | 773 +++++++++++++++++++++++++++++++++++-- src/parser/ast_parser.rs | 223 ++++++++--- src/typechecker/checker.rs | 63 +++ tests/closures_test.rs | 158 ++++++++ tests/examples_test.rs | 1 + 10 files changed, 1422 insertions(+), 74 deletions(-) create mode 100644 examples/closures.ql create mode 100644 src/ast/captures.rs create mode 100644 tests/closures_test.rs diff --git a/LANGUAGE.md b/LANGUAGE.md index 610478b..464ed9b 100644 --- a/LANGUAGE.md +++ b/LANGUAGE.md @@ -229,6 +229,47 @@ factorial = n -> Num => n == 0 ? 1 : n * factorial(n - 1) ``` (See `examples/factorial.ql`, `examples/fibonacci.ql`.) +### Closures — capture by `=` (value) vs `:=` (reference) + +A function written **inside** another function's body is a **closure**: it can read the +enclosing locals it refers to. How each captured name is captured is decided **by the +operator that bound it** — there is no capture list and no marker, mirroring the +mutability rule for [variables](#variables) and [records](#mutation-in-place-field-writes--setters): + +- a name bound with **`=`** is captured **by value** — a frozen, read-only snapshot taken + when the closure is created; +- a name bound with **`:=`** is captured **by reference** — a single shared, mutable cell. + Writes through it (from inside the closure or from the enclosing code) are visible to + everyone sharing it, and the cell survives even if the closure outlives the frame that + created it. + +```quilon +^ = () -> Num => < + total := 0 ~ `:=` -> captured BY REFERENCE + bump = n => < + total := total + n ~ writes the SHARED cell; the effect persists across calls + total + > + bump(10) ~ total -> 10 + bump(20) ~ total -> 30 (same cell) + + base = 7 ~ `=` -> captured BY VALUE (a frozen copy) + addBase = x => x + base + + total + addBase(5) ~ 30 + 12 = 42 +> +``` + +A non-capturing nested function may **recurse** (`fact = n => … fact(n-1) …`); nested +closures may capture from any enclosing frame (the shared `:=` cell is threaded through +every level), and a closure value may itself be captured by another closure and called. + +Closures are **monomorphic** in this milestone: parameters and captured values are +concrete-typed (the capture rule needs no type variables). Capturing a polymorphic value, +generic closures, passing a closure as a function **parameter**, and returning a closure +from a function (higher-order across frames) are deferred — see +[Known limitations](#known-limitations). (See `examples/closures.ql`.) + --- ## Expressions @@ -396,6 +437,7 @@ message instead. Any compile error exits with status 1. | Named record types + methods (`it`) | ✅ | | In-place mutation of `:=` records: field writes (`obj.f := v`) + setter methods | ✅ | | Functions, recursion, blocks, type inference | ✅ | +| Closures: lexical capture (`=` by value / `:=` by reference), monomorphic | ✅ | | Pipe `\|>` (first-arg injection) | ✅ | | `for n <- collection => body` loops | ✅ | | Pattern matching (numbers, wildcard, identifiers, sum-type variants) | ✅ | @@ -407,7 +449,7 @@ message instead. Any compile error exits with status 1. | Conservative GC (Boehm) | ✅ | | `Text` in records/arrays, or as a sum-type payload (`Ok(text)`) | 🚧 | | Command-line `argv` (argc works; argv is a placeholder) | 🚧 | -| Generics, closures, `while` loops | ❌ | +| Generics, `while` loops; generic / closure-returning closures | ❌ | | Array methods (`map`/`filter`/`reduce`), string interpolation | ❌ | --- @@ -419,7 +461,8 @@ message instead. Any compile error exits with status 1. - **Non-numeric data in composites isn't sound yet.** `Text` inside a record or an array, and non-numeric sum-type payloads such as `Ok("x")` / `NotOk("error")`, do not type-check correctly in 0.9 — numeric payloads and numeric records/arrays work. Planned for a later release. - **Array `.size` works only on a named receiver** (`xs.size`), not on a literal/expression (`[1,2,3].size`). - A user-defined `print`/`eprint` is honored by the type checker but the code generator still lowers the built-in — overriding the runtime body is a follow-up. -- **No generics, closures, or `while` loops.** The module system is minimal (`core.io` built-in + file-path imports). +- **No generics or `while` loops.** The module system is minimal (`core.io` built-in + file-path imports). +- **Closures are monomorphic.** Lexical capture works end-to-end (`=` by value / `:=` by reference; see [Closures](#closures--capture-by--value-vs--reference)), including recursion of non-capturing nested functions, capture across multiple nesting levels, and capturing-then-calling another closure. Deferred to a later milestone (they need the closure's type threaded through inference / defunctionalization): capturing a *polymorphic* value, *generic* closures, passing a closure **as a function parameter**, and **returning a closure from a function**. A closure used in an unsupported position is rejected at compile time (e.g. an unannotated function parameter that is called reports `Not a function`), never miscompiled. - **Sum-type payloads mixing types across variants behind one value aren't unified yet.** Each variant's payload slots have a fixed representation sized to the widest variant; a single value carries one variant's payload. Distinct payload *types* per slot across variants (e.g. a position that is `Num` in one variant and `Text` in another) is a deferred follow-up — the built-in payload set (`Num`/`Text`/`Bool`, consistent per position) works. - `argv` is a placeholder (0); full `[]Text` conversion is planned. diff --git a/examples/closures.ql b/examples/closures.ql new file mode 100644 index 0000000..292cd04 --- /dev/null +++ b/examples/closures.ql @@ -0,0 +1,27 @@ +~ Closures: lexical capture, with the capture mode decided by the binding operator. +~ `=` bindings are captured BY VALUE — a frozen, read-only snapshot. +~ `:=` bindings are captured BY REFERENCE — a shared, mutable cell whose writes +~ escape the closure and persist across calls. +~ There is no capture list and no marker: the operator that bound the name is the +~ signal. (Closures are monomorphic in M3 — concrete-typed params and captures.) + +^ = () -> Num => < + ~ `total` is `:=` -> captured by reference. `bump` mutates the shared cell, so + ~ its writes accumulate across separate calls (they escape the closure). + total := 0 + bump = n => < + total := total + n + total + > + + bump(10) ~ total -> 10 + bump(20) ~ total -> 30 (the same cell, written again) + + ~ `base` is `=` -> captured by value. `addBase` sees a frozen copy; rebinding + ~ `base` afterwards does NOT change what the closure already captured. + base = 7 + addBase = x => x + base + + ~ 30 (accumulated via the :=-captured cell) + 12 (5 + the =-captured 7) = 42. + total + addBase(5) +> diff --git a/src/ast/captures.rs b/src/ast/captures.rs new file mode 100644 index 0000000..a81bcc5 --- /dev/null +++ b/src/ast/captures.rs @@ -0,0 +1,188 @@ +//! Free-variable / capture analysis for closures (M3). +//! +//! A lambda *captures* an identifier when its body references a name bound in an +//! enclosing scope (and not shadowed by one of the lambda's own bindings). Capture is +//! purely lexical; how each captured name is captured (by value vs by reference) is +//! decided by the binding operator at codegen time, not here. +//! +//! The one subtlety is that Quilon's `:=` is BOTH "mutable bind" and "reassign": inside a +//! closure, `x := v` *reassigns the captured cell* when `x` names an enclosing binding, +//! but *declares a fresh local* when it does not. So this analysis is parameterized by +//! the set of enclosing names (`outer`): a `:=` to an outer name is a use (capture), a +//! `:=` to a new name is a local. It never needs to resolve types. + +use super::nodes::{Expr, ForPattern, Item, Pattern, Statement}; +use std::collections::HashSet; + +/// The ordered, de-duplicated names a lambda captures: references in its body to names in +/// `outer` (the enclosing scope) that the lambda has not shadowed with its own parameter +/// or local binding. `params` are the lambda's parameter names (which shadow `outer`). +/// Order follows first textual appearance, giving the closure environment a stable field +/// layout. +pub fn lambda_free_idents(params: &[String], body: &Expr, outer: &HashSet) -> Vec { + // `local` accumulates names bound INSIDE the lambda (params first); a read or write of + // a `local` name is never a capture. A name that is neither local nor outer is a + // top-level/global reference, also not captured. + let mut local: HashSet = params.iter().cloned().collect(); + let mut seen = HashSet::new(); + let mut ordered = Vec::new(); + collect(body, &mut local, outer, &mut seen, &mut ordered); + ordered +} + +/// Record a reference to `name` as a capture if it resolves to an enclosing binding +/// (`outer`) and is not locally shadowed. +fn note( + name: &str, + local: &HashSet, + outer: &HashSet, + seen: &mut HashSet, + out: &mut Vec, +) { + if !local.contains(name) && outer.contains(name) && seen.insert(name.to_string()) { + out.push(name.to_string()); + } +} + +fn collect( + expr: &Expr, + local: &mut HashSet, + outer: &HashSet, + seen: &mut HashSet, + out: &mut Vec, +) { + match expr { + Expr::Ident { name, .. } => note(name, local, outer, seen, out), + Expr::Number { .. } | Expr::String { .. } | Expr::Bool { .. } | Expr::Unit { .. } => {} + Expr::BinOp { left, right, .. } | Expr::Pipeline { left, right, .. } => { + collect(left, local, outer, seen, out); + collect(right, local, outer, seen, out); + } + Expr::UnaryOp { expr, .. } | Expr::FieldAccess { expr, .. } => { + collect(expr, local, outer, seen, out) + } + Expr::Call { func, args, .. } => { + collect(func, local, outer, seen, out); + for a in args { + collect(a, local, outer, seen, out); + } + } + Expr::Lambda { params, body, .. } => { + // A nested lambda's parameters shadow within its own body; names it reads from + // OUR scope are transitively free in us too. Its locals are its own — clone so + // they don't leak back into ours. + let mut inner = local.clone(); + for p in params { + inner.insert(p.name.clone()); + } + collect(body, &mut inner, outer, seen, out); + } + Expr::Block { stmts, .. } => { + // A block opens a nested scope; thread a forward-growing local set through it. + let mut block_local = local.clone(); + for stmt in stmts { + match stmt { + Statement::Expr(e) => collect(e, &mut block_local, outer, seen, out), + Statement::Item(Item::VarDecl(decl)) => { + // The initializer runs BEFORE the name binds. + collect(&decl.value, &mut block_local, outer, seen, out); + // `x := v` where `x` is an outer binding not yet shadowed locally + // is a REASSIGNMENT of the captured cell — a use, so capture `x` + // and do NOT shadow it. Any other binding introduces a local. + let is_outer_reassign = decl.mutable + && !block_local.contains(&decl.name) + && outer.contains(&decl.name); + if is_outer_reassign { + note(&decl.name, &block_local, outer, seen, out); + } else { + block_local.insert(decl.name.clone()); + } + } + Statement::Item(Item::FunctionDecl(decl)) => { + // A nested function is itself a closure: names it reads from OUR + // scope are transitively free in us too. Analyze its body with its + // parameters shadowing (a cloned local set), then bind its name. + let mut inner = block_local.clone(); + for p in &decl.params { + inner.insert(p.name.clone()); + } + collect(&decl.body, &mut inner, outer, seen, out); + block_local.insert(decl.name.clone()); + } + Statement::Item(Item::TypeDecl(_)) => {} + } + } + } + Expr::If { + cond, then, else_, .. + } => { + collect(cond, local, outer, seen, out); + collect(then, local, outer, seen, out); + collect(else_, local, outer, seen, out); + } + Expr::Match { expr, arms, .. } => { + collect(expr, local, outer, seen, out); + for arm in arms { + let mut arm_local = local.clone(); + bind_pattern(&arm.pattern, &mut arm_local); + collect(&arm.body, &mut arm_local, outer, seen, out); + } + } + Expr::FieldAssign { target, value, .. } => { + collect(target, local, outer, seen, out); + collect(value, local, outer, seen, out); + } + Expr::Index { expr, index, .. } => { + collect(expr, local, outer, seen, out); + collect(index, local, outer, seen, out); + } + Expr::Array { elements, .. } => { + for e in elements { + collect(e, local, outer, seen, out); + } + } + Expr::Record { fields, .. } | Expr::Constructor { fields, .. } => { + for (_, e) in fields { + collect(e, local, outer, seen, out); + } + } + Expr::SumConstructor { args, .. } => { + for a in args { + collect(a, local, outer, seen, out); + } + } + Expr::ForLoop { + collection, + pattern, + body, + .. + } => { + collect(collection, local, outer, seen, out); + let mut inner = local.clone(); + match pattern { + ForPattern::Item { name, .. } => { + inner.insert(name.clone()); + } + ForPattern::ItemIndex { item, index, .. } => { + inner.insert(item.clone()); + inner.insert(index.clone()); + } + } + collect(body, &mut inner, outer, seen, out); + } + } +} + +fn bind_pattern(pattern: &Pattern, bound: &mut HashSet) { + match pattern { + Pattern::Ident { name, .. } => { + bound.insert(name.clone()); + } + Pattern::Constructor { args, .. } => { + for a in args { + bind_pattern(a, bound); + } + } + Pattern::Number { .. } | Pattern::Wildcard { .. } => {} + } +} diff --git a/src/ast/mod.rs b/src/ast/mod.rs index b496992..7829fda 100644 --- a/src/ast/mod.rs +++ b/src/ast/mod.rs @@ -1,5 +1,6 @@ // AST (Abstract Syntax Tree) definitions for Quilon +pub mod captures; pub mod nodes; pub mod types; diff --git a/src/ast/nodes.rs b/src/ast/nodes.rs index 5421508..b2f9d3c 100644 --- a/src/ast/nodes.rs +++ b/src/ast/nodes.rs @@ -149,6 +149,20 @@ pub enum Expr { span: Span, }, + // Function literal (lambda / closure): `x => x + 1`, `(a, b) => a + b`, `() => 0`. + // A first-class value, distinct from a top-level `FunctionDecl`. When its body + // references names bound in an enclosing scope, those are *captured*: a name bound + // with `=` is captured by value (read-only copy), one bound with `:=` is captured + // by reference (a shared, mutable GC cell). Capture is inferred entirely from the + // binding operator — there is no capture list. Closures are monomorphic in M3: + // params/captures are concrete-typed; generic closures are deferred to M4. + Lambda { + params: Vec, + return_type: Option, + body: Box, + span: Span, + }, + // Pipeline Pipeline { left: Box, @@ -251,6 +265,7 @@ impl Expr { Expr::BinOp { span, .. } => span, Expr::UnaryOp { span, .. } => span, Expr::Call { span, .. } => span, + Expr::Lambda { span, .. } => span, Expr::Pipeline { span, .. } => span, Expr::Block { span, .. } => span, Expr::If { span, .. } => span, diff --git a/src/codegen/generator.rs b/src/codegen/generator.rs index 4182e29..290004a 100644 --- a/src/codegen/generator.rs +++ b/src/codegen/generator.rs @@ -26,6 +26,29 @@ fn zeroed(ty: BasicTypeEnum<'_>) -> BasicValueEnum<'_> { } } +/// A closure's call ABI: its source-parameter LLVM types and its return type. The +/// implicit trailing environment pointer is NOT included (every closure call appends it). +/// Recovered at a call site because a closure value (`{ ptr fn, ptr env }`) does not +/// encode its callee signature. +type ClosureSig<'ctx> = (Vec>, BasicTypeEnum<'ctx>); + +/// One captured free variable of a closure, resolved at the lambda site. +/// `slot` is the variable's storage in the enclosing frame — for a by-value (`=`) +/// capture it is the source slot we snapshot from; for a by-reference (`:=`) capture it +/// IS the shared GC cell pointer we store into the environment. `value_ty` is the +/// captured value's LLVM type. +struct Capture<'ctx> { + name: String, + slot: PointerValue<'ctx>, + value_ty: BasicTypeEnum<'ctx>, + by_ref: bool, + /// If the captured value is itself a closure, its recorded signature + /// (param types, return type) so the lifted body can re-register it and call it. A + /// closure value is an opaque `{ ptr, ptr }` struct that does not encode its callee + /// signature, and the lifted body starts with a cleared `closure_sigs`. + closure_sig: Option>, +} + pub struct CodeGenerator<'ctx> { context: &'ctx Context, module: Module<'ctx>, @@ -51,6 +74,23 @@ pub struct CodeGenerator<'ctx> { // payloads are sized per-value at construction — see `register_builtin_sum_types`). sum_layouts: HashMap>>, current_function: Option>, + // Names of `:=` (mutable) locals in the CURRENT function that are captured by + // reference by some nested closure. These are allocated as heap GC cells (boxes) + // rather than plain stack allocas, so a closure capturing one shares the very same + // cell — writes from either side are visible to the other and survive the closure + // escaping its defining frame. Their `variables` entry stores the cell pointer + // directly; since a load/store of `value_type` works through any pointer, ordinary + // reads/writes need no special-casing. Recomputed on entry to each function body. + boxed_vars: std::collections::HashSet, + // Monotonic counter for naming the lifted top-level function of each lambda + // (`__lambda_0`, `__lambda_1`, …). Lambdas have no source name of their own. + lambda_counter: usize, + // Signature of each local variable currently bound to a closure value: + // (source-parameter LLVM types, return LLVM type). A closure value is an opaque + // `{ ptr fn, ptr env }` struct that does not encode its callee signature, so calling + // one needs the signature recovered here (recorded when the lambda is bound). The + // trailing env-pointer parameter is implicit and not stored. Cleared per function. + closure_sigs: HashMap>, } impl<'ctx> CodeGenerator<'ctx> { @@ -69,6 +109,9 @@ impl<'ctx> CodeGenerator<'ctx> { sum_variants: HashMap::new(), sum_layouts: HashMap::new(), current_function: None, + boxed_vars: std::collections::HashSet::new(), + lambda_counter: 0, + closure_sigs: HashMap::new(), }; codegen.register_builtin_sum_types(); codegen @@ -109,8 +152,12 @@ impl<'ctx> CodeGenerator<'ctx> { } } - // Generate code for all top-level items + // Generate code for all top-level items. Reset the current-function context + // before each one: a top-level item is never nested, so codegen must not see a + // stale function left over from the previous top-level decl (which would make it + // look like a nested/local declaration). for item in &program.items { + self.current_function = None; self.generate_item(item)?; } @@ -285,11 +332,8 @@ impl<'ctx> CodeGenerator<'ctx> { } // Unannotated return type defaults to Num, except a setter body whose // tail is an in-place field write (`it.field := v`) yields `$` (i8). - let inferred_ret = match &method.return_type { - Some(t) => t.clone(), - None if self.expr_is_unit(&method.body) => Type::Unit, - None => Type::Num, - }; + let inferred_ret = + self.default_return_type(method.return_type.as_ref(), &method.body); let return_type = self.type_to_llvm(&inferred_ret)?; let fn_type = return_type.fn_type(¶m_types, false); let method_fn = self.module.add_function(&mangled, fn_type, None); @@ -328,6 +372,8 @@ impl<'ctx> CodeGenerator<'ctx> { self.builder.position_at_end(entry); self.variables.clear(); + self.closure_sigs.clear(); + self.boxed_vars = self.compute_boxed_vars(&method.body); // Param 0 is the implicit receiver `it` (a pointer to the record struct). let it_param = function.get_nth_param(0).unwrap(); @@ -387,17 +433,51 @@ impl<'ctx> CodeGenerator<'ctx> { self.var_named_types .insert(decl.name.clone(), type_name.clone()); } + // Binding a function literal: remember its signature so a later `name(args)` can + // recover the callee type for the indirect closure call (the closure value itself + // does not encode it). + if let Expr::Lambda { + params, + return_type, + body, + .. + } = &decl.value + { + let sig = self.closure_signature(params, return_type.as_ref(), body)?; + self.closure_sigs.insert(decl.name.clone(), sig); + } let value = self.generate_expr(&decl.value)?; if self.current_function.is_some() { - // Local variable - use alloca let var_type = value.get_type(); - let alloca = self.create_entry_block_alloca(&decl.name, var_type)?; + + // Reassignment of an already-bound mutable local (`counter := counter + 1`): + // store THROUGH the existing slot rather than allocating a fresh one. This is + // what makes a `:=` capture escape-safe — the cell a closure shares is the + // very cell later writes target — and it is equivalent to the old realloc for + // ordinary straight-line code (reads always go through the latest slot). + if decl.mutable + && let Some((slot, _)) = self.variables.get(&decl.name).copied() + { + self.builder + .build_store(slot, value) + .map_err(|e| format!("Failed to build store: {:?}", e))?; + return Ok(()); + } + + // A `:=` local captured by reference by some nested closure lives in a heap + // GC cell (a "box"), so the closure and this frame share one mutable cell. Its + // `variables` slot is the cell pointer; loads/stores work through it unchanged. + let slot = if decl.mutable && self.boxed_vars.contains(&decl.name) { + self.alloc_box(var_type)? + } else { + self.create_entry_block_alloca(&decl.name, var_type)? + }; self.builder - .build_store(alloca, value) + .build_store(slot, value) .map_err(|e| format!("Failed to build store: {:?}", e))?; - self.variables.insert(decl.name.clone(), (alloca, var_type)); + self.variables.insert(decl.name.clone(), (slot, var_type)); } else { // Global variable let global = @@ -410,6 +490,50 @@ impl<'ctx> CodeGenerator<'ctx> { } fn generate_function_decl(&mut self, decl: &FunctionDecl) -> Result<(), String> { + // A function declared INSIDE another function (we are mid-emitting a body) is a + // local declaration. If its body references enclosing locals it is a capturing + // CLOSURE (lowered via the lambda machinery); otherwise it is a self-contained + // local function, which we emit as a plain module function — that preserves + // recursion (`fact = n => … fact(n-1) …`), since a closure value cannot refer to + // itself before it exists. The choice is by ACTUAL captures, not syntax. + if self.current_function.is_some() { + let param_names: Vec = decl.params.iter().map(|p| p.name.clone()).collect(); + let outer: std::collections::HashSet = self.variables.keys().cloned().collect(); + let captures = + crate::ast::captures::lambda_free_idents(¶m_names, &decl.body, &outer); + if !captures.is_empty() { + return self.generate_local_closure(decl); + } + // No captures: emit a plain module function, but save/restore the enclosing + // frame (the shared `variables`/`closure_sigs`/`boxed_vars`/builder state) + // around it, since `emit_module_function` clears and repopulates them. + let saved_block = self.builder.get_insert_block(); + let saved_function = self.current_function; + let saved_vars = std::mem::take(&mut self.variables); + let saved_boxed = std::mem::take(&mut self.boxed_vars); + let saved_sigs = std::mem::take(&mut self.closure_sigs); + + let result = self.emit_module_function(decl); + + self.variables = saved_vars; + self.boxed_vars = saved_boxed; + self.closure_sigs = saved_sigs; + self.current_function = saved_function; + if let Some(block) = saved_block { + self.builder.position_at_end(block); + } + return result; + } + + self.emit_module_function(decl) + } + + /// Emit `decl` as a top-level/module function (internal linkage). Clears and + /// repopulates the per-function emission state (`variables`, `closure_sigs`, + /// `boxed_vars`); the entry point `^` gets the special f64-return / implicit-0 + /// treatment. Used for true top-level functions and for non-capturing nested + /// functions (which can recurse, unlike a closure value). + fn emit_module_function(&mut self, decl: &FunctionDecl) -> Result<(), String> { // Convert parameter types to LLVM types let param_types: Vec = decl .params @@ -426,11 +550,7 @@ impl<'ctx> CodeGenerator<'ctx> { // An unannotated body defaults to `Num`, except a Unit (`$`) tail — e.g. // `log = m => print(m)` — which must be `i8`, not f64, or `build_return` // would emit `ret i8` into an f64 function and fail module verification. - let inferred = match &decl.return_type { - Some(t) => t.clone(), - None if self.expr_is_unit(&decl.body) => Type::Unit, - None => Type::Num, - }; + let inferred = self.default_return_type(decl.return_type.as_ref(), &decl.body); self.type_to_llvm(&inferred)? }; @@ -458,6 +578,9 @@ impl<'ctx> CodeGenerator<'ctx> { // Store parameters in variables map self.variables.clear(); + self.closure_sigs.clear(); + // Which `:=` locals must be heap-boxed because a nested closure captures them. + self.boxed_vars = self.compute_boxed_vars(&decl.body); for (i, param) in decl.params.iter().enumerate() { let llvm_param = function.get_nth_param(i as u32).unwrap(); llvm_param.set_name(¶m.name); @@ -493,6 +616,24 @@ impl<'ctx> CodeGenerator<'ctx> { Ok(()) } + /// Bind a capturing nested function as a local closure value: lower it via the lambda + /// machinery (capturing enclosing locals per the `=`/`:=` rule) and store the + /// resulting `{ ptr fn, ptr env }` in a local slot, recording its signature so + /// `name(args)` resolves to an indirect closure call. + fn generate_local_closure(&mut self, decl: &FunctionDecl) -> Result<(), String> { + let sig = self.closure_signature(&decl.params, decl.return_type.as_ref(), &decl.body)?; + self.closure_sigs.insert(decl.name.clone(), sig); + + let closure = self.generate_lambda(&decl.params, decl.return_type.as_ref(), &decl.body)?; + let slot = self.create_entry_block_alloca(&decl.name, closure.get_type())?; + self.builder + .build_store(slot, closure) + .map_err(|e| format!("Failed to store closure: {:?}", e))?; + self.variables + .insert(decl.name.clone(), (slot, closure.get_type())); + Ok(()) + } + fn create_entry_block_alloca( &self, name: &str, @@ -515,6 +656,510 @@ impl<'ctx> CodeGenerator<'ctx> { .map_err(|e| format!("Failed to build alloca: {:?}", e)) } + // ---- Closures (M3) ----------------------------------------------------------------- + // + // A closure value is a flat `{ ptr fn, ptr env }` struct: a pointer to the lifted + // top-level function, and a pointer to its heap-allocated environment of captured + // values. The lifted function takes the captured environment as an extra TRAILING + // pointer parameter (after the source parameters), so calling through a closure is a + // plain indirect call passing `env` last. Closures are monomorphic (M3): captured + // values and parameters are concrete-typed; generic closures are M4. + + /// The uniform closure representation: `{ ptr fn, ptr env }`. + fn closure_struct_type(&self) -> inkwell::types::StructType<'ctx> { + let ptr = self.context.ptr_type(AddressSpace::default()); + self.context.struct_type(&[ptr.into(), ptr.into()], false) + } + + /// Allocate a GC-managed heap cell large enough to hold one `ty` value and return the + /// pointer to it. Used to "box" a `:=` local captured by reference, so the cell + /// outlives the defining frame and is shared with the closure. + fn alloc_box(&self, ty: BasicTypeEnum<'ctx>) -> Result, String> { + use inkwell::values::AnyValue; + let size = ty + .size_of() + .ok_or_else(|| format!("cannot size box for type {:?}", ty))?; + let alloc_fn = self.get_intrinsic("__alloc")?; + Ok(self + .builder + .build_call(alloc_fn, &[size.into()], "box") + .map_err(|e| format!("Failed to call __alloc for box: {:?}", e))? + .as_any_value_enum() + .into_pointer_value()) + } + + /// The `:=` (mutable) locals of the function body `body` that some nested closure + /// captures by reference, and so must be heap-boxed. A captured `=` local is copied + /// by value into the closure's environment and needs no box; only a captured mutable + /// local must share a single cell with the closure. Computed by collecting the + /// function's `:=` binding names and intersecting with the union of every nested + /// lambda's free variables. + fn compute_boxed_vars(&self, body: &Expr) -> std::collections::HashSet { + let mut mutable_locals = std::collections::HashSet::new(); + Self::collect_mutable_locals(body, &mut mutable_locals); + + // Find which of those mutable locals a nested closure captures. Passing the + // mutable-local set as the lambdas' `outer` scope means a closure's captures are + // already restricted to (and recognize reassignments of) exactly these names. + let mut captured = std::collections::HashSet::new(); + Self::collect_lambda_captures(body, &mutable_locals, &mut captured); + captured + } + + /// Collect the names of all `:=` (mutable) `VarDecl`s bound in THIS function frame — + /// i.e. in `expr` and its nested control-flow, but NOT inside a nested lambda body (a + /// lambda's own `:=` locals live in the lambda's frame, not ours). + fn collect_mutable_locals(expr: &Expr, out: &mut std::collections::HashSet) { + match expr { + // A nested function literal opens its own frame — do not descend. + Expr::Lambda { .. } => {} + Expr::Block { stmts, .. } => { + for stmt in stmts { + match stmt { + crate::ast::Statement::Expr(e) => Self::collect_mutable_locals(e, out), + crate::ast::Statement::Item(Item::VarDecl(decl)) => { + if decl.mutable { + out.insert(decl.name.clone()); + } + Self::collect_mutable_locals(&decl.value, out); + } + crate::ast::Statement::Item(_) => {} + } + } + } + Expr::BinOp { left, right, .. } | Expr::Pipeline { left, right, .. } => { + Self::collect_mutable_locals(left, out); + Self::collect_mutable_locals(right, out); + } + Expr::UnaryOp { expr, .. } | Expr::FieldAccess { expr, .. } => { + Self::collect_mutable_locals(expr, out) + } + Expr::Call { func, args, .. } => { + Self::collect_mutable_locals(func, out); + for a in args { + Self::collect_mutable_locals(a, out); + } + } + Expr::If { + cond, then, else_, .. + } => { + Self::collect_mutable_locals(cond, out); + Self::collect_mutable_locals(then, out); + Self::collect_mutable_locals(else_, out); + } + Expr::Match { expr, arms, .. } => { + Self::collect_mutable_locals(expr, out); + for arm in arms { + Self::collect_mutable_locals(&arm.body, out); + } + } + Expr::FieldAssign { target, value, .. } => { + Self::collect_mutable_locals(target, out); + Self::collect_mutable_locals(value, out); + } + Expr::Index { expr, index, .. } => { + Self::collect_mutable_locals(expr, out); + Self::collect_mutable_locals(index, out); + } + Expr::Array { elements, .. } => { + for e in elements { + Self::collect_mutable_locals(e, out); + } + } + Expr::Record { fields, .. } | Expr::Constructor { fields, .. } => { + for (_, e) in fields { + Self::collect_mutable_locals(e, out); + } + } + Expr::SumConstructor { args, .. } => { + for a in args { + Self::collect_mutable_locals(a, out); + } + } + Expr::ForLoop { + collection, body, .. + } => { + Self::collect_mutable_locals(collection, out); + Self::collect_mutable_locals(body, out); + } + Expr::Number { .. } + | Expr::String { .. } + | Expr::Bool { .. } + | Expr::Unit { .. } + | Expr::Ident { .. } => {} + } + } + + /// Union, over every lambda appearing (at any depth) in `expr`, of the names it + /// captures from `outer`. Used to find which of the enclosing frame's mutable locals + /// a closure shares (and so must be heap-boxed). + fn collect_lambda_captures( + expr: &Expr, + outer: &std::collections::HashSet, + out: &mut std::collections::HashSet, + ) { + Self::for_each_closure(expr, &mut |params, body| { + let names: Vec = params.iter().map(|p| p.name.clone()).collect(); + for name in crate::ast::captures::lambda_free_idents(&names, body, outer) { + out.insert(name); + } + }); + } + + /// Invoke `f(params, body)` for every closure appearing (at any depth) in `expr` — a + /// `Expr::Lambda` OR a nested `Item::FunctionDecl` (both are closures; the latter is + /// only resolved to a plain function at codegen when it captures nothing). Used to + /// gather captures across all closures in a frame. + fn for_each_closure(expr: &Expr, f: &mut impl FnMut(&[crate::ast::Param], &Expr)) { + Self::walk_exprs(expr, &mut |e| match e { + Expr::Lambda { params, body, .. } => f(params, body), + Expr::Block { stmts, .. } => { + for stmt in stmts { + if let crate::ast::Statement::Item(Item::FunctionDecl(decl)) = stmt { + f(&decl.params, &decl.body); + // The function body is an expression position `walk_exprs` does + // not enter (it only descends VarDecl initializers), so recurse to + // find closures nested inside this nested function too. + Self::for_each_closure(&decl.body, f); + } + } + } + _ => {} + }); + } + + /// Pre-order walk over every sub-expression of `expr`, invoking `f` on each. Used by + /// the closure pre-passes above. Does not descend into nested item declarations' + /// signatures (only expression positions), which is all closure analysis needs. + fn walk_exprs(expr: &Expr, f: &mut impl FnMut(&Expr)) { + f(expr); + match expr { + Expr::BinOp { left, right, .. } | Expr::Pipeline { left, right, .. } => { + Self::walk_exprs(left, f); + Self::walk_exprs(right, f); + } + Expr::UnaryOp { expr, .. } | Expr::FieldAccess { expr, .. } => { + Self::walk_exprs(expr, f) + } + Expr::Call { func, args, .. } => { + Self::walk_exprs(func, f); + for a in args { + Self::walk_exprs(a, f); + } + } + Expr::Lambda { body, .. } => Self::walk_exprs(body, f), + Expr::Block { stmts, .. } => { + for stmt in stmts { + match stmt { + crate::ast::Statement::Expr(e) => Self::walk_exprs(e, f), + crate::ast::Statement::Item(Item::VarDecl(d)) => { + Self::walk_exprs(&d.value, f) + } + crate::ast::Statement::Item(_) => {} + } + } + } + Expr::If { + cond, then, else_, .. + } => { + Self::walk_exprs(cond, f); + Self::walk_exprs(then, f); + Self::walk_exprs(else_, f); + } + Expr::Match { expr, arms, .. } => { + Self::walk_exprs(expr, f); + for arm in arms { + Self::walk_exprs(&arm.body, f); + } + } + Expr::FieldAssign { target, value, .. } => { + Self::walk_exprs(target, f); + Self::walk_exprs(value, f); + } + Expr::Index { expr, index, .. } => { + Self::walk_exprs(expr, f); + Self::walk_exprs(index, f); + } + Expr::Array { elements, .. } => { + for e in elements { + Self::walk_exprs(e, f); + } + } + Expr::Record { fields, .. } | Expr::Constructor { fields, .. } => { + for (_, e) in fields { + Self::walk_exprs(e, f); + } + } + Expr::SumConstructor { args, .. } => { + for a in args { + Self::walk_exprs(a, f); + } + } + Expr::ForLoop { + collection, body, .. + } => { + Self::walk_exprs(collection, f); + Self::walk_exprs(body, f); + } + Expr::Number { .. } + | Expr::String { .. } + | Expr::Bool { .. } + | Expr::Unit { .. } + | Expr::Ident { .. } => {} + } + } + + /// The default return TYPE codegen assigns a function with the given (possibly + /// missing) return annotation and body: the annotation if present, else `$` (Unit) + /// for a Unit-tailed body, else `Num`. Codegen lacks the checker's full inference, so + /// this picks the LLVM-level return type for an unannotated function/lambda/method. + /// (The entry point `^` is handled separately — it always returns an f64 exit code.) + fn default_return_type(&self, return_type: Option<&Type>, body: &Expr) -> Type { + match return_type { + Some(t) => t.clone(), + None if self.expr_is_unit(body) => Type::Unit, + None => Type::Num, + } + } + + /// The LLVM signature of a function literal: (source-parameter types, return type). + /// Mirrors the type rules used when emitting the lifted function, but without the + /// trailing env pointer (which is implicit to every closure call). + fn closure_signature( + &self, + params: &[crate::ast::Param], + return_type: Option<&Type>, + body: &Expr, + ) -> Result, String> { + let param_types: Vec = params + .iter() + .map(|p| self.type_to_llvm(&p.type_annotation.clone().unwrap_or(Type::Num))) + .collect::, _>>()?; + let ret = self.default_return_type(return_type, body); + Ok((param_types, self.type_to_llvm(&ret)?)) + } + + /// Lower a function literal to a value: lift its body into a fresh top-level function + /// taking the captured environment as a trailing `ptr` parameter, build and populate + /// that environment on the heap, and return the `{ ptr fn, ptr env }` closure struct. + /// + /// Capture rule, inferred from each captured name's binding operator: + /// `=` binding -> captured BY VALUE: a snapshot is copied into the env (read-only). + /// `:=` binding -> captured BY REFERENCE: the env holds the pointer to the shared + /// GC cell (the box), so reads see — and writes escape to — the one + /// cell, surviving the closure outliving its defining frame. + fn generate_lambda( + &mut self, + params: &[crate::ast::Param], + return_type: Option<&Type>, + body: &Expr, + ) -> Result, String> { + let ptr_ty = self.context.ptr_type(AddressSpace::default()); + + // 1. Determine the captured names: a lambda free variable that is actually a + // binding in the current frame. A by-reference capture is one whose name is in + // the current `boxed_vars` (its storage is a shared cell); the rest are by-value. + let param_names: Vec = params.iter().map(|p| p.name.clone()).collect(); + let outer: std::collections::HashSet = self.variables.keys().cloned().collect(); + let free = crate::ast::captures::lambda_free_idents(¶m_names, body, &outer); + let mut captures: Vec> = Vec::new(); + for name in free { + // Only names with a live local slot are captured; a free name that resolves to + // a top-level function/global is referenced directly inside the lifted body + // (it has module scope) and needs no capture. + if let Some((slot, value_ty)) = self.variables.get(&name).copied() { + let by_ref = self.boxed_vars.contains(&name); + let closure_sig = self.closure_sigs.get(&name).cloned(); + captures.push(Capture { + name, + slot, + value_ty, + by_ref, + closure_sig, + }); + } + } + + // 2. Build the environment struct type. A by-value capture stores the value; a + // by-reference capture stores the cell pointer (`ptr`). + let env_field_types: Vec = captures + .iter() + .map(|c| if c.by_ref { ptr_ty.into() } else { c.value_ty }) + .collect(); + let env_struct_ty = self.context.struct_type(&env_field_types, false); + + // 3. Allocate and populate the environment on the GC heap (so it survives the + // closure escaping). For a by-value capture, snapshot the current value; for a + // by-reference capture, store the shared cell pointer itself. + let env_ptr = if captures.is_empty() { + ptr_ty.const_null() + } else { + let env = self.alloc_box(env_struct_ty.into())?; + for (i, cap) in captures.iter().enumerate() { + let field = self + .builder + .build_struct_gep(env_struct_ty, env, i as u32, "env_field") + .map_err(|e| format!("Failed to GEP env field: {:?}", e))?; + let stored: BasicValueEnum = if cap.by_ref { + cap.slot.into() + } else { + self.builder + .build_load(cap.value_ty, cap.slot, &cap.name) + .map_err(|e| format!("Failed to load capture: {:?}", e))? + }; + self.builder + .build_store(field, stored) + .map_err(|e| format!("Failed to store capture: {:?}", e))?; + } + env + }; + + // 4. Emit the lifted top-level function `__lambda_N(params..., ptr env)`. + let fn_value = + self.emit_lambda_function(params, return_type, body, &captures, env_struct_ty)?; + + // 5. Assemble the closure value `{ fn_ptr, env_ptr }`. + let closure_ty = self.closure_struct_type(); + let fn_ptr = fn_value.as_global_value().as_pointer_value(); + let with_fn = self + .builder + .build_insert_value(closure_ty.get_undef(), fn_ptr, 0, "clo_fn") + .map_err(|e| format!("Failed to insert closure fn: {:?}", e))? + .into_struct_value(); + let closure = self + .builder + .build_insert_value(with_fn, env_ptr, 1, "clo_env") + .map_err(|e| format!("Failed to insert closure env: {:?}", e))? + .into_struct_value(); + Ok(closure.into()) + } + + /// Emit the lifted top-level function for a lambda: its source parameters followed by + /// a trailing `ptr env`. Inside, parameters are bound normally and each captured name + /// is re-bound from the environment — a by-value capture is copied into a local slot, + /// a by-reference capture re-uses the shared cell pointer directly (so writes escape). + /// Saves and restores the enclosing codegen state (current function, variable scope, + /// boxed set, builder position) around the nested emission. + fn emit_lambda_function( + &mut self, + params: &[crate::ast::Param], + return_type: Option<&Type>, + body: &Expr, + captures: &[Capture<'ctx>], + env_struct_ty: inkwell::types::StructType<'ctx>, + ) -> Result, String> { + let ptr_ty = self.context.ptr_type(AddressSpace::default()); + + // Same (param types, return type) the call site reconstructs, plus the trailing + // env pointer — keeping the emitted function and the indirect-call type in lockstep. + let (mut param_types, ret_ty) = self.closure_signature(params, return_type, body)?; + param_types.push(ptr_ty.into()); // trailing env pointer + + let fn_type = ret_ty.fn_type( + ¶m_types + .iter() + .map(|t| (*t).into()) + .collect::>(), + false, + ); + + let name = format!("__lambda_{}", self.lambda_counter); + self.lambda_counter += 1; + let function = self.module.add_function(&name, fn_type, None); + function.set_linkage(inkwell::module::Linkage::Internal); + + // Save enclosing emission state — we are about to emit a DIFFERENT function body. + let saved_block = self.builder.get_insert_block(); + let saved_function = self.current_function; + let saved_vars = std::mem::take(&mut self.variables); + let saved_boxed = std::mem::take(&mut self.boxed_vars); + let saved_sigs = std::mem::take(&mut self.closure_sigs); + // The lifted body has its own frame: recompute which of ITS `:=` locals are boxed. + self.boxed_vars = self.compute_boxed_vars(body); + // A by-reference capture is ALSO a shared cell in this frame: mark it boxed so a + // FURTHER nested closure capturing the same name captures it by reference too + // (sharing the one cell across all nesting levels). Without this, a `:=` value + // mutated through two levels of closures would be silently snapshotted by value. + for cap in captures.iter().filter(|c| c.by_ref) { + self.boxed_vars.insert(cap.name.clone()); + } + self.current_function = Some(function); + + let entry = self.context.append_basic_block(function, "entry"); + self.builder.position_at_end(entry); + + // Bind source parameters (indices 0..n); the env pointer is the last parameter. + for (i, param) in params.iter().enumerate() { + let llvm_param = function.get_nth_param(i as u32).unwrap(); + llvm_param.set_name(¶m.name); + let pty = llvm_param.as_basic_value_enum().get_type(); + let alloca = self.create_entry_block_alloca(¶m.name, pty)?; + self.builder + .build_store(alloca, llvm_param) + .map_err(|e| format!("Failed to store param: {:?}", e))?; + self.variables.insert(param.name.clone(), (alloca, pty)); + } + + // Re-bind captures from the environment pointer (the trailing parameter). + if !captures.is_empty() { + let env_ptr = function + .get_nth_param(params.len() as u32) + .unwrap() + .into_pointer_value(); + for (i, cap) in captures.iter().enumerate() { + let field = self + .builder + .build_struct_gep(env_struct_ty, env_ptr, i as u32, "cap_field") + .map_err(|e| format!("Failed to GEP capture field: {:?}", e))?; + if cap.by_ref { + // The field holds the shared cell pointer; load it and bind the name + // to that cell so reads/writes inside the closure hit the one cell. + let cell = self + .builder + .build_load(ptr_ty, field, &cap.name) + .map_err(|e| format!("Failed to load cell ptr: {:?}", e))? + .into_pointer_value(); + self.variables + .insert(cap.name.clone(), (cell, cap.value_ty)); + } else { + // By-value capture: copy the snapshot into a fresh local slot. + let val = self + .builder + .build_load(cap.value_ty, field, &cap.name) + .map_err(|e| format!("Failed to load capture value: {:?}", e))?; + let alloca = self.create_entry_block_alloca(&cap.name, cap.value_ty)?; + self.builder + .build_store(alloca, val) + .map_err(|e| format!("Failed to store capture value: {:?}", e))?; + self.variables + .insert(cap.name.clone(), (alloca, cap.value_ty)); + } + // If the captured value is itself a closure, re-register its signature so + // a `name(args)` inside this lifted body resolves to an indirect call (the + // lifted body began with a cleared `closure_sigs`). + if let Some(sig) = &cap.closure_sig { + self.closure_sigs.insert(cap.name.clone(), sig.clone()); + } + } + } + + let body_value = self.generate_expr(body)?; + self.builder + .build_return(Some(&body_value)) + .map_err(|e| format!("Failed to build closure return: {:?}", e))?; + + // Restore the enclosing emission state. + self.variables = saved_vars; + self.boxed_vars = saved_boxed; + self.closure_sigs = saved_sigs; + self.current_function = saved_function; + if let Some(block) = saved_block { + self.builder.position_at_end(block); + } + + Ok(function) + } + fn generate_expr(&mut self, expr: &Expr) -> Result, String> { match expr { Expr::Number { value, .. } => { @@ -595,6 +1240,13 @@ impl<'ctx> CodeGenerator<'ctx> { Expr::Call { func, args, .. } => self.generate_call(func, args), + Expr::Lambda { + params, + return_type, + body, + .. + } => self.generate_lambda(params, return_type.as_ref(), body), + Expr::If { cond, then, else_, .. } => self.generate_if(cond, then, else_), @@ -1049,6 +1701,15 @@ impl<'ctx> CodeGenerator<'ctx> { return self.generate_sum_constructor(tag, &type_name, args); } + // A local variable bound to a closure value: call it indirectly, passing the + // captured environment as the trailing argument. Recognized by the variable's + // recorded closure signature (see `closure_sigs`). + if let Some((param_tys, ret_ty)) = self.closure_sigs.get(func_name.as_str()).cloned() + && self.variables.contains_key(func_name.as_str()) + { + return self.generate_closure_call(func_name, ¶m_tys, ret_ty, args); + } + // Get the function from the module. If there is no plain top-level function with this // name, it may be a method call: the parser desugars `recv.method(a, b)` to // `method(recv, a, b)`, so resolve `recv`'s named type and dispatch to `Type_method`. @@ -1082,22 +1743,86 @@ impl<'ctx> CodeGenerator<'ctx> { .build_call(function, &arg_metadata, "calltmp") .map_err(|e| format!("Failed to build call: {:?}", e))?; - // In Inkwell 0.8, try_as_basic_value returns a special ValueKind enum - // We need to pattern match on it, but since it's an opaque type, - // let's just use into_basic_value which works for functions that return values - // For now, we'll unsafely assume all functions return values - use inkwell::values::AnyValue; - let any_val = call_site.as_any_value_enum(); + Self::call_result_to_basic(call_site) + } + + /// Call a closure value held in local variable `var_name`: extract the function and + /// environment pointers from its `{ ptr fn, ptr env }` struct and emit an indirect + /// call passing the source arguments followed by the environment pointer. `param_tys` + /// / `ret_ty` are the closure's recorded signature (excluding the implicit env param). + fn generate_closure_call( + &mut self, + var_name: &str, + param_tys: &[BasicTypeEnum<'ctx>], + ret_ty: BasicTypeEnum<'ctx>, + args: &[Expr], + ) -> Result, String> { + if args.len() != param_tys.len() { + return Err(format!( + "closure `{}` expects {} argument(s), got {}", + var_name, + param_tys.len(), + args.len() + )); + } + let ptr_ty = self.context.ptr_type(AddressSpace::default()); + let closure_ty = self.closure_struct_type(); + + // Load the closure struct from its slot, then split out fn and env pointers. + let (slot, _) = *self.variables.get(var_name).expect("closure var bound"); + let closure_val = self + .builder + .build_load(closure_ty, slot, var_name) + .map_err(|e| format!("Failed to load closure: {:?}", e))? + .into_struct_value(); + let fn_ptr = self + .builder + .build_extract_value(closure_val, 0, "clo_fn") + .map_err(|e| format!("Failed to extract closure fn: {:?}", e))? + .into_pointer_value(); + let env_ptr = self + .builder + .build_extract_value(closure_val, 1, "clo_env") + .map_err(|e| format!("Failed to extract closure env: {:?}", e))? + .into_pointer_value(); + + // Evaluate arguments, then append the environment pointer. + let mut call_args: Vec = + Vec::with_capacity(args.len() + 1); + for arg in args { + call_args.push(self.generate_expr(arg)?.into()); + } + call_args.push(env_ptr.into()); + + // Reconstruct the callee function type: source params + trailing env ptr -> ret. + let mut metadata_params: Vec = + param_tys.iter().map(|t| (*t).into()).collect(); + metadata_params.push(ptr_ty.into()); + let fn_type = ret_ty.fn_type(&metadata_params, false); - // Convert AnyValueEnum to BasicValueEnum - match any_val { + let call = self + .builder + .build_indirect_call(fn_type, fn_ptr, &call_args, "clo_call") + .map_err(|e| format!("Failed to build indirect call: {:?}", e))?; + + Self::call_result_to_basic(call) + } + + /// Convert a call site's result to a `BasicValueEnum`, erroring if the callee returns + /// a non-basic (e.g. void) value. Shared by the direct (`generate_call`) and indirect + /// closure (`generate_closure_call`) call paths so both handle return kinds identically. + fn call_result_to_basic( + call: inkwell::values::CallSiteValue<'ctx>, + ) -> Result, String> { + use inkwell::values::AnyValue; + match call.as_any_value_enum() { inkwell::values::AnyValueEnum::IntValue(v) => Ok(v.into()), inkwell::values::AnyValueEnum::FloatValue(v) => Ok(v.into()), inkwell::values::AnyValueEnum::PointerValue(v) => Ok(v.into()), inkwell::values::AnyValueEnum::ArrayValue(v) => Ok(v.into()), inkwell::values::AnyValueEnum::StructValue(v) => Ok(v.into()), inkwell::values::AnyValueEnum::VectorValue(v) => Ok(v.into()), - _ => Err("Function does not return a basic value".to_string()), + _ => Err("call did not return a basic value".to_string()), } } diff --git a/src/parser/ast_parser.rs b/src/parser/ast_parser.rs index 2756066..bc9899e 100644 --- a/src/parser/ast_parser.rs +++ b/src/parser/ast_parser.rs @@ -9,6 +9,11 @@ use crate::lexer::{Span, Token, TokenKind}; pub struct Parser<'a> { tokens: &'a [Token], pos: usize, + /// While true, a bare `ident =>` / `(params) =>` is NOT taken as a function literal. + /// Set only while parsing a `for`-loop's collection, where the `=>` that follows the + /// collection introduces the loop BODY (`for n <- xs => body`) and must not be + /// swallowed by a lambda. Lambdas in such positions can still be written parenthesized. + no_lambda: bool, } #[derive(Debug, Clone, PartialEq)] @@ -27,7 +32,11 @@ impl std::error::Error for ParseError {} impl<'a> Parser<'a> { pub fn new(tokens: &'a [Token]) -> Self { - Self { tokens, pos: 0 } + Self { + tokens, + pos: 0, + no_lambda: false, + } } pub fn parse(tokens: &'a [Token]) -> Result { @@ -249,52 +258,8 @@ impl<'a> Parser<'a> { return_type: Option, exported: bool, ) -> Result { - let mut params = Vec::new(); - // Parse parameters: (a, b) or (a :: Type, b :: Type) or single param or just => - if self.check(&TokenKind::ParenOpen) { - self.advance(); - - if !self.check(&TokenKind::ParenClose) { - loop { - let param_name = self.expect_ident()?; - let param_type = if self.check(&TokenKind::TypeAnnotation) { - self.advance(); - Some(self.parse_type()?) - } else { - None - }; - - params.push(Param { - name: param_name, - type_annotation: param_type, - span: self.previous_span(), - }); - - if !self.check(&TokenKind::Comma) { - break; - } - self.advance(); - } - } - - self.expect(&TokenKind::ParenClose)?; - } else if self.check(&TokenKind::Ident) { - // Single parameter without parentheses - let param_name = self.expect_ident()?; - let param_type = if self.check(&TokenKind::TypeAnnotation) { - self.advance(); - Some(self.parse_type()?) - } else { - None - }; - - params.push(Param { - name: param_name, - type_annotation: param_type, - span: self.previous_span(), - }); - } + let params = self.parse_param_list()?; // Optional return type annotation with -> let return_type = if self.check(&TokenKind::ReturnArrow) { @@ -575,7 +540,10 @@ impl<'a> Parser<'a> { TokenKind::Assign | TokenKind::MutAssign ) { - // This looks like a declaration + // This looks like a declaration. A nested `name = params => body` stays an + // `Item::FunctionDecl`; codegen decides per-decl whether it is a capturing + // CLOSURE or a plain (recursion-capable) local function, based on whether + // it actually references enclosing locals. let item = self.parse_item()?; stmts.push(Statement::Item(item)); } else { @@ -656,8 +624,11 @@ impl<'a> Parser<'a> { }; // `<- collection`. The collection is a full expression (may itself be a - // pipeline); it stops at the `=>` that introduces the body. + // pipeline); it stops at the `=>` that introduces the body. Suppress bare-lambda + // detection here so `for n <- xs => body` reads `xs` as the collection and the + // `=>` as the loop arrow, not as a `xs => body` lambda. self.expect(&TokenKind::LeftArrow)?; + self.no_lambda = true; let collection = self.parse_ternary()?; // `=> body` — a single expression or a `< ... >` block. @@ -1103,6 +1074,10 @@ impl<'a> Parser<'a> { } fn parse_primary(&mut self) -> Result { + // `no_lambda` is a ONE-SHOT guard: it suppresses a function-literal only at this + // leading primary (the for-loop collection), not in any deeper sub-expression, so + // nested parenthesized lambdas still parse. Consume it here. + let allow_lambda = !std::mem::replace(&mut self.no_lambda, false); let token = self.peek(); match &token.kind { @@ -1135,6 +1110,13 @@ impl<'a> Parser<'a> { Ok(Expr::Unit { span }) } TokenKind::Ident => { + // A bare single-parameter lambda: `x => body` or `x :: Type => body`. + // Detected before consuming the ident as a plain reference: an ident + // followed by `=>` (or `:: Type =>`) is a function literal, not a value. + if allow_lambda && self.looks_like_bare_lambda() { + return self.parse_lambda_expr(); + } + let span = token.span.clone(); let name = token.text.clone(); self.advance(); @@ -1175,6 +1157,13 @@ impl<'a> Parser<'a> { } } TokenKind::ParenOpen => { + // A parenthesized parameter list introducing a lambda — `() => body`, + // `(a, b) => body`, `(n :: Num) => body` — vs. an ordinary parenthesized + // expression `(a + b)`. Distinguished by scanning to the matching `)` + // and checking whether `=>` / `->` follows. + if allow_lambda && self.paren_starts_lambda() { + return self.parse_lambda_expr(); + } self.advance(); let expr = self.parse_expr()?; self.expect(&TokenKind::ParenClose)?; @@ -1189,6 +1178,144 @@ impl<'a> Parser<'a> { } } + /// At a bare identifier in primary position, is this the start of a single-parameter + /// lambda (`x => …` or `x :: Type => …`) rather than a plain value reference? We peek + /// past the ident: a directly-following `=>` is a lambda; an `::` introduces a typed + /// parameter, so we scan the (single) type annotation and require a `=>` after it. + fn looks_like_bare_lambda(&self) -> bool { + debug_assert!(self.check(&TokenKind::Ident)); + match &self.peek_ahead(1).kind { + TokenKind::Arrow => true, + TokenKind::TypeAnnotation => { + // `name :: =>` — find the `=>` that closes the annotation. Types + // here are simple (a name with optional `{ … }` generic args); the first + // top-level `=>` after the `::` ends the param list of a lambda. A `<` + // block or anything else means it was not a lambda parameter. + let mut idx = 2; + let mut brace_depth = 0i32; + while idx < 40 { + match &self.peek_ahead(idx).kind { + TokenKind::BraceOpen => brace_depth += 1, + TokenKind::BraceClose => brace_depth -= 1, + TokenKind::Arrow if brace_depth == 0 => return true, + TokenKind::Eof => return false, + // A return-arrow, block, comma, etc. at depth 0 means this `::` + // was a binding annotation, not a lambda param — bail out. + TokenKind::BlockOpen | TokenKind::Comma if brace_depth == 0 => { + return false; + } + _ => {} + } + idx += 1; + } + false + } + _ => false, + } + } + + /// At `(` in primary position, does a parenthesized parameter list (`(a, b) =>` / + /// `() =>`) follow — making this a lambda — rather than a parenthesized expression? + /// Scans to the matching `)` and checks for a following `=>` or `->` (return type). + fn paren_starts_lambda(&self) -> bool { + debug_assert!(self.check(&TokenKind::ParenOpen)); + let mut depth = 1; + let mut idx = 1; + while idx < 80 && depth > 0 { + match &self.peek_ahead(idx).kind { + TokenKind::ParenOpen => depth += 1, + TokenKind::ParenClose => { + depth -= 1; + if depth == 0 { + let next = &self.peek_ahead(idx + 1).kind; + return *next == TokenKind::Arrow || *next == TokenKind::ReturnArrow; + } + } + TokenKind::Eof => return false, + _ => {} + } + idx += 1; + } + false + } + + /// Parse a function-literal (lambda) expression: `params => body`, where `params` is + /// `()`, a parenthesized list, or a single bare identifier — optionally with a `->` + /// return type. The body is a single expression or a `< >` block. Shares its + /// parameter grammar with `parse_function_decl`. + fn parse_lambda_expr(&mut self) -> Result { + let start = self.current_span(); + let params = self.parse_param_list()?; + + let return_type = if self.check(&TokenKind::ReturnArrow) { + self.advance(); + Some(self.parse_type()?) + } else { + None + }; + + self.expect(&TokenKind::Arrow)?; + + let body = if self.check(&TokenKind::BlockOpen) { + self.parse_block()? + } else { + self.parse_expr()? + }; + + let span = Span::new(start.start, self.previous_span().end); + Ok(Expr::Lambda { + params, + return_type, + body: Box::new(body), + span, + }) + } + + /// Parse a parameter list: `(a, b)`, `(a :: T, b :: T)`, `()`, or a single bare + /// `name` / `name :: T` without parentheses. Stops before the `=>` / `->`. + /// Shared by lambdas and (via the existing inline logic) function declarations. + fn parse_param_list(&mut self) -> Result, ParseError> { + let mut params = Vec::new(); + if self.check(&TokenKind::ParenOpen) { + self.advance(); + if !self.check(&TokenKind::ParenClose) { + loop { + let param_name = self.expect_ident()?; + let param_type = if self.check(&TokenKind::TypeAnnotation) { + self.advance(); + Some(self.parse_type()?) + } else { + None + }; + params.push(Param { + name: param_name, + type_annotation: param_type, + span: self.previous_span(), + }); + if !self.check(&TokenKind::Comma) { + break; + } + self.advance(); + } + } + self.expect(&TokenKind::ParenClose)?; + } else if self.check(&TokenKind::Ident) { + let param_name = self.expect_ident()?; + let param_type = if self.check(&TokenKind::TypeAnnotation) { + self.advance(); + Some(self.parse_type()?) + } else { + None + }; + params.push(Param { + name: param_name, + type_annotation: param_type, + span: self.previous_span(), + }); + } + Ok(params) + } + fn parse_record(&mut self) -> Result { let start = self.current_span(); self.expect(&TokenKind::BraceOpen)?; diff --git a/src/typechecker/checker.rs b/src/typechecker/checker.rs index 0f5f0d0..fd3e264 100644 --- a/src/typechecker/checker.rs +++ b/src/typechecker/checker.rs @@ -770,6 +770,62 @@ impl TypeChecker { Ok(()) } + /// Type-check a function-literal (lambda) expression and return its + /// `Type::Function`. The lambda's body is checked in a fresh scope layered over the + /// enclosing environment, so it may reference (capture) outer bindings — `Environment` + /// scopes are a stack and `lookup` walks outward. Capture-by-value vs by-reference is + /// decided downstream (codegen) from each captured name's binding operator; the + /// checker only needs the outer binding to be visible and concrete. + /// + /// Closures are MONOMORPHIC in M3: parameters are concrete-typed (annotated, else the + /// `Num` default, matching top-level functions) and captured values are concrete. The + /// language has no type variables, so there is nothing polymorphic to capture; generic + /// closures + defunctionalization are deferred to M4. + fn check_lambda( + &mut self, + params: &[Param], + return_type: Option<&Type>, + body: &Expr, + ) -> Result { + let param_types: Vec = params + .iter() + .map(|p| { + p.type_annotation + .as_ref() + .map(|t| self.resolve_type(t)) + .unwrap_or(Type::Num) + }) + .collect(); + + self.env.push_scope(); + for (param, param_type) in params.iter().zip(param_types.iter()) { + self.env.define( + param.name.clone(), + param_type.clone(), + false, + param.span.clone(), + )?; + } + let body_type = self.infer_expr(body)?; + self.env.pop_scope(); + + // Honor an explicit `-> Type` annotation; otherwise the body's inferred type is + // the return type. + let ret = match return_type { + Some(annotated) => { + let annotated = self.resolve_type(annotated); + self.check_type_compatibility(&annotated, &body_type, body.span())?; + annotated + } + None => body_type, + }; + + Ok(Type::Function { + params: param_types, + return_type: Box::new(ret), + }) + } + fn infer_expr(&mut self, expr: &Expr) -> Result { match expr { Expr::Number { .. } => Ok(Type::Num), @@ -797,6 +853,13 @@ impl TypeChecker { Expr::Call { func, args, span } => self.check_call(func, args, span), + Expr::Lambda { + params, + return_type, + body, + .. + } => self.check_lambda(params, return_type.as_ref(), body), + Expr::Pipeline { left, right, span } => { // `left |> right` injects `left` as the first argument of the // right-hand call: `x |> f` => `f(x)`, `x |> f(a)` => `f(x, a)`. diff --git a/tests/closures_test.rs b/tests/closures_test.rs new file mode 100644 index 0000000..acab709 --- /dev/null +++ b/tests/closures_test.rs @@ -0,0 +1,158 @@ +// Closures (M3): execution tests for lexical capture, with the capture mode chosen by +// the binding operator — `=` captures by value (a frozen snapshot), `:=` captures by +// reference (a shared, mutable cell whose writes escape the closure). Each test drives +// the full pipeline (lex -> parse -> typecheck -> codegen -> JIT) and asserts the real +// exit code, the same backbone as run_test.rs. + +use quilon::jit; +use quilon::lexer::Lexer; +use quilon::parser; +use quilon::typechecker::TypeChecker; +use std::sync::Mutex; + +// LLVM JIT / native-target init is not thread-safe; serialize across parallel tests. +static JIT_LOCK: Mutex<()> = Mutex::new(()); + +fn assert_exit(src: &str, expected: i32) { + let _guard = JIT_LOCK.lock().unwrap_or_else(|p| p.into_inner()); + let tokens = Lexer::tokenize(src).expect("lexing failed"); + let program = parser::parse(&tokens).expect("parsing failed"); + TypeChecker::new() + .check_program(&program) + .expect("type checking failed"); + let code = jit::run_program(&program).expect("execution failed"); + assert_eq!(code, expected, "unexpected exit code for source:\n{}", src); +} + +fn assert_type_error(src: &str) { + let tokens = Lexer::tokenize(src).expect("lexing failed"); + let program = parser::parse(&tokens).expect("parsing failed"); + assert!( + TypeChecker::new().check_program(&program).is_err(), + "expected a type error for source:\n{}", + src + ); +} + +// --- `:=` capture is by reference: writes from inside the closure escape and accumulate +// across separate calls (it shares one cell with the enclosing frame). --- +#[test] +fn mutable_capture_counter_accumulates() { + // `count` is `:=`, captured by reference. Each `bump(...)` writes the shared cell, so + // the effect of the first call is visible to the second: 10 then +20 -> 30. + assert_exit( + "^ = () -> Num => <\n count := 0\n bump = n => <\n count := count + n\n count\n >\n bump(10)\n bump(20)\n count\n>", + 30, + ); +} + +// --- A `:=` capture sees writes the OUTER frame makes after the closure is created — +// direct evidence the cell is shared, not copied. --- +#[test] +fn mutable_capture_sees_later_outer_write() { + assert_exit( + "^ = () -> Num => <\n n := 1\n readN = () => n\n n := 100\n readN()\n>", + 100, + ); +} + +// --- `=` capture is by value: the closure carries a frozen snapshot of the captured +// binding, usable repeatedly and independent of anything else. --- +#[test] +fn value_capture_is_frozen_snapshot() { + // `base` is `=` (immutable), captured by value. add(10)=17, add(20)=27 -> 44. + assert_exit( + "^ = () -> Num => <\n base = 7\n add = x => x + base\n add(10) + add(20)\n>", + 44, + ); +} + +// --- A closure capturing nothing is just a function value, callable normally. --- +#[test] +fn closure_with_no_captures() { + assert_exit( + "^ = () -> Num => <\n twice = x => x + x\n twice(21)\n>", + 42, + ); +} + +// --- Two closures capturing the SAME `:=` cell share it: a write through one is seen by +// the other (the box is one shared mutable cell). --- +#[test] +fn two_closures_share_one_mutable_cell() { + assert_exit( + "^ = () -> Num => <\n total := 0\n add = n => <\n total := total + n\n total\n >\n reset = () => <\n total := 0\n total\n >\n add(5)\n add(37)\n reset()\n add(42)\n>", + 42, + ); +} + +// --- A closure may capture by value AND by reference at once. --- +#[test] +fn mixed_value_and_reference_capture() { + // `step` (=) is frozen-copied; `acc` (:=) is the shared cell. acc starts 0, + // bump adds step (3) each call: 3, then 6. + assert_exit( + "^ = () -> Num => <\n step = 3\n acc := 0\n bump = () => <\n acc := acc + step\n acc\n >\n bump()\n bump()\n>", + 6, + ); +} + +// --- A closure parameter still shadows an outer binding of the same name (the param is +// not a capture). --- +#[test] +fn parameter_shadows_outer_binding() { + assert_exit( + "^ = () -> Num => <\n x = 99\n f = x => x + 1\n f(41)\n>", + 42, + ); +} + +// --- Capture does not relax mutability: an `=`-captured name is still immutable, so the +// checker rejects writing it (there is no `:=` reassign of an `=` binding). --- +#[test] +fn value_capture_stays_immutable() { + assert_type_error( + "^ = () -> Num => <\n k = 1\n f = () => <\n k := 2\n k\n >\n f()\n>", + ); +} + +// --- A nested function that captures NOTHING is a plain local function and may recurse +// (a closure value cannot refer to itself before it exists, so non-capturing nested +// functions are emitted as ordinary functions, preserving recursion). --- +#[test] +fn non_capturing_nested_function_recurses() { + assert_exit( + "^ = () -> Num => <\n fact = n -> Num => n == 0 ? 1 : n * fact(n - 1)\n fact(5)\n>", + 120, + ); +} + +// --- A closure may capture another closure value and call it (higher-order use within a +// frame): `g` captures the capturing closure `f` and invokes it. --- +#[test] +fn closure_captures_and_calls_another_closure() { + assert_exit( + "^ = () -> Num => <\n base = 40\n f = x => x + base\n g = () => f(1)\n g() + 1\n>", + 42, + ); +} + +// --- A `:=` cell mutated through TWO levels of closure nesting still shares one cell: +// the inner closure's writes are visible after both calls (10 = 5 + 5). --- +#[test] +fn mutable_capture_shared_across_two_nesting_levels() { + assert_exit( + "^ = () -> Num => <\n total := 0\n mid = () => <\n inner = () => <\n total := total + 5\n total\n >\n inner()\n inner()\n >\n mid()\n total\n>", + 10, + ); +} + +// --- A closure nested two levels deep can read an `=` value from the outermost frame, +// captured transitively through the middle closure (frozen by value). --- +#[test] +fn value_capture_threads_through_nested_closure() { + assert_exit( + "^ = () -> Num => <\n a = 42\n mid = z => <\n inner = w => a + w\n inner(z)\n >\n mid(0)\n>", + 42, + ); +} diff --git a/tests/examples_test.rs b/tests/examples_test.rs index fec7662..7867211 100644 --- a/tests/examples_test.rs +++ b/tests/examples_test.rs @@ -55,6 +55,7 @@ const EXPECTED_EXIT: &[(&str, i32)] = &[ ("sum_types.ql", 42), ("use_module.ql", 5), ("unit.ql", 0), + ("closures.ql", 42), ]; fn ql_files() -> Vec {