diff --git a/docs/information/audit.md b/docs/information/audit.md index 510db2b8..90ae2d67 100644 --- a/docs/information/audit.md +++ b/docs/information/audit.md @@ -1,6 +1,6 @@ # Vocabulary Audit Report -Generated: 2026-05-13 +Generated: 2026-05-14 All audits below are **advisory**. Heuristic audits generate false positives; use this report as a starting point for manual review, not as a correctness gate. diff --git a/docs/information/vocabulary_information.md b/docs/information/vocabulary_information.md index 0cfd5d81..2b38d80d 100644 --- a/docs/information/vocabulary_information.md +++ b/docs/information/vocabulary_information.md @@ -4,7 +4,7 @@ - **Merkle Root**: `39ca671a4dcb3075855cb293380d1796105e2eca0de49b0537279b798b675ee6` - **Pattern Count**: 452 -- **Last Verified**: 2026-05-13 +- **Last Verified**: 2026-05-14 ## Usage diff --git a/paper/sema.pdf b/paper/sema.pdf index f8747fef..a536afe4 100644 Binary files a/paper/sema.pdf and b/paper/sema.pdf differ diff --git a/paper/sema.tex b/paper/sema.tex index c4061cc4..3cbccf77 100644 --- a/paper/sema.tex +++ b/paper/sema.tex @@ -334,9 +334,23 @@ \subsection{The Type System} \label{sec:wiring} \label{sec:compiler} -This subsection describes the type system of the Sema Bootstrap Library. It is a \emph{particular} type system, not \emph{the} type system: the protocol defined in the preceding subsections (hash a structured JSON object, mix the hash into language, compute Merkle roots over dependency fields) operates on any schema a library-author chooses. Keeping the protocol this broad is deliberate---it is what allows multiple agents to speak Sema potentially in \emph{different dialects} while sharing the same cryptographic carrier. Dialects can split across \emph{communities of agents} (one deployment tightens what another leaves loose) or across \emph{domains} (agent protocols, molecular structures, legal clauses, and game-rule engines each want different type systems because the concepts they coordinate are shaped differently). Two libraries with different schemas do not interoperate field-for-field, but each resolves to verifiable hashes, the fail-closed handshake still detects divergence, and the boundary between dialects is explicit rather than silent. This positions Sema less as a single vocabulary and more as a substrate on which vocabularies coexist---analogous to how Unicode does not prescribe a language, only a shared encoding. - -We picked the type system that follows because it seemed useful for agent protocols: structural typing lets the same verb operate across domains as long as subjects satisfy a common contract, and polymorphic dispatch lets an agent request ``any pattern that implements Deep(Discover)'' rather than pinning a specific hash. Another library targeting a different problem might make different choices for equally good reasons, and we invite others to extend, fork, or replace the design below---additional fields the bootstrap library does not yet capture (provenance, confidence, uncertainty bounds, temporal validity, cost models), alternative dependency partitionings, or entirely different schemas for domains we have not addressed are all welcome. The protocol's only hard requirement is that whatever schema is chosen be defined explicitly so its hash is reproducible. Readers interested only in protocol-level guarantees may skip to Section~\ref{sec:standard_library}; readers interested in how Sema handles polymorphism, composition, and wiring across \semaPatternCount{}~patterns should continue here. +This subsection describes the type system of the Sema Bootstrap Library. It is a +\emph{particular} type system, not \emph{the} type system: the protocol above +only requires an explicit schema whose canonical form can be reproducibly +hashed. Different communities or domains can therefore build different Sema +dialects while sharing the same cryptographic carrier. Those dialects may not +interoperate field-for-field, but their boundaries are explicit: hashes still +identify canonical objects, and handshakes still expose divergence. In this +sense Sema is closer to Unicode than to a universal language: it standardizes the +carrier, not every vocabulary spoken through it. + +We chose the agent-protocol schema below because structural typing lets the same +verb operate across domains as long as subjects satisfy a common contract, and +polymorphic dispatch lets an agent request ``any pattern that implements +Deep(Discover)'' rather than pinning a specific hash. Readers interested only in +protocol-level guarantees may skip to Section~\ref{sec:standard_library}; +readers interested in how Sema handles polymorphism, composition, and wiring +across \semaPatternCount{}~patterns should continue here. A central tension in vocabulary design is determining when a change in attributes constitutes a change in identity. The bootstrap library adopts a structural resolution: qualitative differences create distinct patterns, while quantitative differences create parameters with hashed range contracts. For example, \sema{Backoff}{c6d1} with a \texttt{base\_delay} of 100ms and the same pattern with a 5-second delay share the same semantic identity because the hash captures the parameter range contract, not the instance value. However, if changing a variable alters the failure mode, such as switching from busy-waiting to database storage, it becomes a distinct pattern (e.g., \texttt{SpinLock} vs. \texttt{Lease}). This allows the runtime to validate instantiation values against the hashed range contract while preserving the conceptual unity of the pattern. @@ -374,17 +388,47 @@ \section{The Bootstrap Library} \subsection{Methodology} -We emphasize that the initial vocabulary is a pragmatic bootstrap, not the output of a principled selection process. A different author would produce a different vocabulary using the same protocol, and that is by design---the protocol's value is independent of any particular library. The \semaPatternCount~patterns emerged iteratively and opportunistically rather than from systematic derivation, drawing on four loosely organized sources: (1)~established distributed systems primitives (locking, leases, consensus) adapted from engineering literature; (2)~the theoretical architecture developed in~\cite{westerberg2026crp}, which motivated patterns for the Solver Contract surfaces, the marginal-value rule, and cognitive decomposition; (3)~direct prompting of LLMs to identify design patterns, search for gaps in the growing vocabulary, and propose candidates for missing capabilities; and (4)~the author's own domain intuitions about agent failure modes, codified as patterns like \sema{WhyClimb}{cb43} (recursive problem abstraction) and \sema{SteelmanCheck}{2d13} (mandatory counter-argument generation). Creativity protocols such as Adversarial Evolutionary Generation~\cite{westerberg2026alien} were used to push beyond the ``Median Trap'' of standard LLM generation for some frontier patterns. - -The process became \emph{increasingly structured} as the library grew. Early minting was loose and author-driven; as the pattern count rose, the process progressively acquired scaffolding: a recurring ``Taxonomist'' role---an LLM agent invoked at refinement checkpoints to audit the growing graph (rejecting duplicates, merging overlapping concepts, enforcing the Noun/Verb separation); explicit rules and guidelines that tightened whenever a failure mode recurred across patterns (Section~\ref{sec:refinement}); hard structural constraints carried by the database layer itself (DAG acyclicity, dependency-hash transitivity, bidirectional wiring checks, Rule-G layer direction); and explicit tests of \emph{broad use}---validating that a pattern's mechanism and invariants capture what every legitimate deployment context needs rather than overfitting to the author's first scenario. - -The atomic apply step, \texttt{sema apply}, is where these constraints are enforced concretely. It validates every addition and removal before executing any mutation, catching progressively more classes of error---for example, handle-format irregularities, signature-syntax mistakes, layer-direction violations on hard dependencies, embedding near-duplicates, and rigor warnings for patterns with mechanism text but no invariants or contracts. Each class was added in response to a concrete failure that had slipped through earlier: what began as a small set of schema validations grew into a suite of checks that \texttt{sema apply} runs on every mint and on every CI pass. The vocabulary reported here is the product of that progression: free-form in its earliest additions, increasingly disciplined in its later ones, and retroactively brought up to the current standard by the refinement pass of Section~\ref{sec:refinement}. - -Minting is gated by a two-criteria rule: a concept earns a pattern if it meets either \emph{protocol consistency} (multiple agents must coordinate on the exact semantics, so content-addressed definition is the mechanism of shared meaning, as with \sema{StateLock}{5602} for distributed mutex) or \emph{structured thinking} (the act of specifying an invariant-bearing mechanism sharpens a concept that English uses loosely, so downstream patterns invoking it are forced to confront what they mean). Either criterion alone is sufficient; English suffices when neither is met. This is what prevents pre-emptively minting concepts with no callers and no coordinating protocol, which would add library weight without buying either form of value. - -Pattern \emph{generality} is gated by a separate \emph{broad-use test}: for each pattern, enumerate its canonical intended use, its plausible future uses, and the enumerated range of legitimate deployment contexts; the pattern's mechanism and invariants must then capture only what \emph{every} such context needs (the intersection), while features that vary between contexts become descendant territory accessed via \texttt{derived\_from}. The discipline prevents two failure modes simultaneously: a mechanism overfit to the author's first use case (too specific, breaks for legitimate variants) and a mechanism so generic it has no teeth (too vague, underconstrains the concept). A validation of the test across twenty representative patterns (spanning Physics, Infrastructure, Mind, and Society; Rings~0--2; Tiers~0--2; both Nouns and Verbs) is recorded in the companion broad-use analysis, and is what shaped the current separation between general parent patterns and their specializing descendants---for example, \sema{Lock}{051c} with descendant \sema{Mutex}{4f92} (the \(K=1\) specialization), and \sema{Task}{b290} with descendant \sema{BoundedTask}{1063} (the budget-carrying specialization). - -The result, after the subsequent refinement pass described in Section~\ref{sec:refinement}, is a vocabulary with no duplicate mechanisms or semantic collisions detected by the applied audits (hash collision, canonical-form equality, and embedding-similarity sweeps above a threshold) across \semaPatternCount{}~patterns---maintained by those audits and by the Taxonomist role, but not enforced by \texttt{sema apply}, and not a guarantee of functional uniqueness (see §\ref{sec:limitations}, ``The Synonymy Limit''). We make no claim to completeness or optimality. +We emphasize that the initial vocabulary is a pragmatic bootstrap, not the +output of a principled selection process. A different author would produce a +different vocabulary using the same protocol, and that is by design: the +protocol's value is independent of any particular library. The +\semaPatternCount~patterns emerged iteratively from four sources: distributed +systems primitives, the cognitive-decomposition architecture developed +in~\cite{westerberg2026crp}, LLM-assisted gap-finding, and the author's own +domain intuitions about agent failure modes. Creativity protocols such as +Adversarial Evolutionary Generation~\cite{westerberg2026alien} were used for +some frontier patterns. + +The process became more structured as the library grew. Refinement checkpoints +used a recurring ``Taxonomist'' role to reject duplicates, merge overlapping +concepts, and enforce the Noun/Verb separation; repeated failures became rules +and guidelines (Section~\ref{sec:refinement}); and the database layer enforces +DAG acyclicity, dependency-hash transitivity, bidirectional wiring checks, and +Rule-G layer direction. The atomic \texttt{sema apply} step operationalizes +these constraints by validating additions and removals before mutation, catching +handle-format errors, signature mistakes, layer-direction violations, +near-duplicates, and rigor warnings on every mint and CI pass. + +Minting is gated by two criteria. A concept earns a pattern if it requires +\emph{protocol consistency}---multiple agents must coordinate on the exact +semantics, as with \sema{StateLock}{5602}---or if specifying it produces +\emph{structured thinking}, forcing a loosely used English concept into an +invariant-bearing mechanism. English suffices when neither criterion is met. +Pattern \emph{generality} is then checked by a broad-use test: the mechanism and +invariants must capture what every legitimate deployment context needs, while +context-specific features become descendant territory via \texttt{derived\_from} +(e.g., \sema{Lock}{051c} $\rightarrow$ \sema{Mutex}{4f92}, or +\sema{Task}{b290} $\rightarrow$ \sema{BoundedTask}{1063}). The test prevents +both overfit mechanisms that break on legitimate variants and toothless +abstractions that constrain nothing. + +After the refinement pass described in Section~\ref{sec:refinement}, the +library has no duplicate mechanisms or semantic collisions detected by the +applied audits (hash collision, canonical-form equality, and embedding-similarity +sweeps above a threshold) across \semaPatternCount{}~patterns. These audits and +the Taxonomist role maintain quality, but they do not guarantee functional +uniqueness (see §\ref{sec:limitations}, ``The Synonymy Limit''). We make no +claim to completeness or optimality. \subsection{The Civilization Stack} @@ -846,7 +890,7 @@ \section{Related Work} \item[The Living Taxonomy] The concept of a self-authoring semantic infrastructure, where AI systems mint and reference their own cognitive patterns, was sketched in earlier work~\cite{westerberg2025superintelligence}. That proposal framed the Living Taxonomy along two axes that this paper operationalizes differently. As a \emph{runtime substrate}, it called for named patterns embedded directly in natural language so agents could reference complex reasoning without re-explanation---illustrated by thinking blocks citing entries like ``Pattern \#7,832: Statistical Intimidation.'' As a \emph{training substrate}, it proposed shifting the learning objective from $P(\text{text}|\text{context})$ to $P(\text{text},\text{thinking}|\text{context})$, with the metacognitive corpus annotating text with references to those same patterns. The proposal also flagged decentralization as a necessary property, warning that centralized control over such infrastructure would lead to ``Cognitive Architecture Wars,'' and sketched on-chain governance as one architectural response. Sema provides the content-addressed implementation of the runtime half: it realizes embedded compression via hash-based references (e.g., \sema{BayesUpdate}{bbf6}) and supports decentralization by making definition identity mathematical rather than institutional. The training-substrate half remains the complementary construction proposed in that earlier work. The two fit together as a closed loop: the patterns a Sema-equipped agent references at runtime are exactly the kind of artifact a metacognitive training corpus would cite, and patterns that survive training pressure are exactly the ones a coordination protocol needs to verify at runtime. - \item[Semantic Definition Hash (SDH)] Hawke proposed content-addressing semantic definitions in 2002~\cite{sdh2002}, with format \texttt{urn:sdh::}. SDH is the most direct predecessor to Sema's core mechanism: Hawke's design principles---that use of an identifier constitutes assertion of its definition, that definitions are ``set in cryptographic stone,'' and that the result is ``a growing lattice of static documents''---anticipate Sema's architecture closely. Independently, Sayers and Eshghi~\cite{sayers2002hashing} proposed at HP Labs the same year that RDF URIs should derive from content hashes, serving both as shorthand and as integrity proof. More broadly, the distinction between SDH and Sema is not merely technical but architectural. SDH hashed \emph{whole documents} and named things inside them; Sema hashes \emph{structured behavioral contracts} with internal Merkle structure, enabling partial alignment on individual fields. SDH produced identifiers for a formal graph language (RDF) consumed by structured parsers; Sema produces \emph{Sema words}: human-readable handles anchored by cryptographic hashes that function as tokens in the natural language LLMs already think in---any medium that carries text can carry Sema words. SDH had no evolution mechanism; Sema's \texttt{derived\_from} lineage field enables vocabulary evolution without breaking existing hashes. And where SDH targeted human ontology authors in a cooperative Semantic Web, Sema targets LLM agents that hallucinate, drift, and deceive---motivating the fail-closed handshake and adversarial hardening absent from Hawke's design. RDF and the Semantic Web failed adoption not for technical reasons but because they fought human cognition~\cite{cagle2016}. Sema prioritizes human-readable handles over opaque URIs and targets LLM agents rather than human developers as the primary consumers. Gustafson's work on content-addressing semantic data~\cite{knowledgefutures2023}, combining RDF canonicalization with IPFS content-identifiers for the Underlay project, independently validates the approach for semantic datasets. + \item[Semantic Definition Hash (SDH)] Hawke proposed content-addressing semantic definitions in 2002~\cite{sdh2002}, with format \texttt{urn:sdh::}. SDH is the most direct predecessor to Sema's core mechanism: use of an identifier asserts its definition, and definitions are set in ``cryptographic stone.'' Independently, Sayers and Eshghi~\cite{sayers2002hashing} proposed that RDF URIs should derive from content hashes, serving both as shorthand and integrity proof. Sema differs architecturally: SDH hashed whole documents and named things inside them, while Sema hashes structured behavioral contracts with internal Merkle fields, producing human-readable Sema words for the natural-language stream LLM agents already use. Sema also adds lineage, fail-closed handshakes, adversarial hardening, and agent-oriented handles rather than opaque URIs. Gustafson's work on content-addressing semantic data~\cite{knowledgefutures2023}, combining RDF canonicalization with IPFS content-identifiers for the Underlay project, independently validates the approach for semantic datasets. \item[Content-Addressing in Other Domains] The canonicalize-then-hash pattern appears across domains: InChI/InChIKey for molecules (IUPAC standard), UniParc for protein sequences (70M+ entries), and Git for source code (ubiquitous). @@ -895,7 +939,7 @@ \section{Related Work} \item[The Synthesis: Code, Cognition, and Economy] Most frameworks specialize in one domain: Knowledge Graphs handle static facts (Knowing), Agent Frameworks handle tool execution (Doing), and Crypto Protocols handle value (Transacting). Sema sits at the novel intersection of these three disciplines. It treats \textbf{Thinking} (e.g., \sema{Reason}{e901}) and \textbf{Doing} (e.g., \sema{Bid}{464a}) as nodes in the same graph, allowing agents to reason about their economic commitments with the same grammar they use to reason about the world. The content-addressing principles are shared with IPFS~\cite{benet2014ipfs} and Unison~\cite{chiusano2024unison}. Unison deserves particular note: it identifies every function definition by the hash of its de-named abstract syntax tree, creating exactly the content-addressed definition DAG that Sema applies to semantic patterns. Unison's approach to the ``rename without breaking identity'' problem---separating the human-readable name from the content-derived hash---directly parallels Sema's substrate/overlay separation. Sema extends Unison's insight from code definitions to meaning definitions, adding the typed interface system and fail-closed handshake that multi-agent coordination requires. - \item[Portable Capability Definitions] The closest surface analogue to Sema's Pattern Cards is Anthropic's Agent Skills specification~\cite{anthropic2025skills}, released as an open standard in December 2025 and adopted across Claude, OpenAI, Microsoft, and Cursor. Agent Skills package a capability as a \texttt{SKILL.md} file---YAML frontmatter (\texttt{name}, \texttt{description}) plus a Markdown body---with progressive disclosure from metadata to instructions to resources. The format shape mirrors Pattern Cards: structured header over a human-readable body. Three differences are load-bearing. \textbf{Identity:} Agent Skills are identified by string \texttt{name}; two skills named \texttt{pdf-processing} authored separately may be different capabilities and no protocol-level mechanism detects the divergence. Sema identifies by hash, so divergence surfaces before it causes failure. \textbf{Contract:} Agent Skills bodies are freeform Markdown instructions; Pattern Cards carry structured, hashable contracts (invariants, preconditions, postconditions, typed I/O), making contract equivalence an $O(1)$ hash comparison rather than a runtime judgement call. \textbf{Composition:} Agent Skills are isolated artifacts---the spec provides no construct for one skill to declare a typed dependency on another. Sema patterns compose: a pattern's \texttt{mechanism} and typed-dependency fields reference other patterns by hash, so the Merkle root of any pattern captures the full transitive closure of the vocabulary it relies on. The bootstrap library is therefore not a flat dictionary but a wired executable architecture---\sema{FractalIntelligence}{c9bc} references \sema{Decompose}{dcf9}, which references \sema{MarginalValueRule}{311b}, and following the references gives an agent the full cognitive stack required to execute the top-level pattern. The same three gaps hold for adjacent industry formats: A2A Agent Cards~\cite{google2025a2a} describe capabilities in JSON keyed by human-assigned skill names with no content-addressed identity; MCP tool schemas~\cite{anthropic2024mcp} enforce JSON Schema on inputs and outputs but carry no hashable behavioral contract (a tool named \texttt{send\_email} in two different MCP servers may route, authenticate, and retry entirely differently and no protocol-level check detects the divergence). These formats share the ``portable capability'' goal with Sema; Sema adds the machinery that makes the portability cryptographically verifiable and the capabilities composable by construction. + \item[Portable Capability Definitions] The closest surface analogue to Sema's Pattern Cards is Anthropic's Agent Skills specification~\cite{anthropic2025skills}, released as an open standard in December 2025 and adopted across Claude, OpenAI, Microsoft, and Cursor. Agent Skills and Pattern Cards both place a structured header over a human-readable body, but three differences are load-bearing. \textbf{Identity:} Agent Skills are identified by string \texttt{name}; two independently authored \texttt{pdf-processing} skills may differ, while Sema identifies by hash so divergent capabilities cannot hide behind the same label. \textbf{Contract:} Agent Skills bodies are freeform Markdown instructions; Pattern Cards carry structured, hashable contracts, making contract equivalence an $O(1)$ hash comparison rather than a runtime judgement call. \textbf{Composition:} Agent Skills are isolated artifacts; Sema patterns reference other patterns by hash, so the Merkle root captures the transitive closure of the vocabulary a pattern relies on. The same gaps hold for adjacent industry formats: A2A Agent Cards~\cite{google2025a2a} use human-assigned skill names, and MCP tool schemas~\cite{anthropic2024mcp} validate inputs and outputs but carry no hashable behavioral contract. These formats share the ``portable capability'' goal with Sema; Sema adds the machinery that makes portability cryptographically verifiable and capabilities composable by construction. \item[Structured LLM Frameworks and Contract-Based Design] Sema's Pattern Cards---with preconditions, postconditions, invariants, and typed interfaces---exist within a broader lineage of structured LLM programming. DSPy~\cite{khattab2024dspy} introduced typed signatures for language model pipelines, and DSPy Assertions~\cite{singhvi2024dspy} added computational constraints (\texttt{dspy.Assert}, \texttt{dspy.Suggest}) that are structurally similar to Pattern Card contracts. White et al.~\cite{white2023prompt} provided the foundational work on formalizing prompt patterns using software design pattern structure. Leoveanu-Condrei~\cite{leoveanu2025dbc} adapted Design by Contract to LLM agents, positing that agents satisfying the same contracts are functionally equivalent---the closest published work to Sema's contract-based approach. Sema's contribution relative to this lineage is not the contract structure itself but the cryptographic binding: by hashing contracts into identifiers, Sema makes contract equivalence verifiable via hash comparison rather than requiring runtime inspection. @@ -944,7 +988,7 @@ \section{Limitations and Future Work} \paragraph{The Sema Tax.} As observed in Condition~B of the demonstration, using the vocabulary without a coordination protocol introduces cognitive overhead: agents spend additional tokens retrieving and verifying definitions. While Condition~C's protocol largely compensates for this cost, the base ``Sema Tax'' remains a real trade-off, particularly for latency-sensitive applications. Future work should quantify this overhead across model families and context window sizes. -\paragraph{Single-Agent Reasoning Scaffold.} This paper presents Sema as inter-agent coordination infrastructure, but the vocabulary's Mind layer---with executable specifications of reasoning patterns such as \sema{ChainOfThought}{dd97}, \sema{Dialectic}{856d}, and \sema{MetaCheck}{1914}---may also serve as an internal thinking tool for individual agents. An agent that hydrates these patterns into its own context gains structured reasoning primitives with explicit failure modes and invariants, effectively using the vocabulary as a cognitive scaffold rather than a coordination protocol. Early informal testing suggests that agents annotating their reasoning with Sema patterns produce more auditable and self-correcting chains of thought. This use case is not evaluated here; the theoretical foundations for composing reasoning across solvers are developed in~\cite{westerberg2026crp}. +\paragraph{Single-Agent Reasoning Scaffold.} Although this paper presents Sema as inter-agent coordination infrastructure, the vocabulary's Mind layer may also serve as an internal scaffold for individual agents: hydrated reasoning patterns carry explicit failure modes and invariants into the agent's own context. This use case is not evaluated here; the theoretical foundations for composing reasoning across solvers are developed in~\cite{westerberg2026crp}. \paragraph{Empirical Pattern Validation.} The methodology described in Section~\ref{sec:standard_library} validates patterns analytically---by review, rules, guidelines, database/DAG constraints, and broad-use tests---but not empirically. We do not currently measure whether a given pattern, once minted, actually improves downstream agent behavior: whether tasks coordinated through \sema{StateLock}{5602} succeed more reliably than those coordinated through ad-hoc prose, whether a Mind-layer reasoning pattern produces better solutions than unstructured chain-of-thought, or whether a given formulation of a mechanism is clearer to agents than its alternatives. This is a clear future-work direction. An empirical layer would treat each pattern as a testable hypothesis, A/B-comparing variants on realistic agent benchmarks and feeding the results back into the next round of mechanism refinement. The refinement pass described in Section~\ref{sec:refinement} sharpens patterns against shared formal criteria; an empirical refinement stage would sharpen them against observed task outcomes, producing a population of patterns that are not merely internally coherent but demonstrably useful.