Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 94 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

This repository already has a detailed **AGENTS.md** at the repo root — read it first for
full architecture diagrams, event-handler tables, commit/PR conventions, and common-pattern
recipes (adding a language, a marker type, a CLI command, a config option). This file only
covers what's needed to get moving quickly.

## What this project is

sphinx-codelinks is a Sphinx extension providing fast source-code traceability for
Sphinx-Needs: it scans source files (C++, Python, C#, Rust, TypeScript, Go, YAML, JSON) for
marker comments via tree-sitter, and generates Sphinx-Needs items / RST that link
documentation back to exact source locations.

## Commands

All commands run through `tox` (uses `tox-uv`).

```bash
# Run default test env (py312-sphinx8-needs5)
tox

# List all test env combinations (py{312,313,314}-sphinx{7,8,9}-needs{5,6,7,8})
tox -a

# Run a specific env / file / test
tox -e py312-sphinx8-needs5
tox -e py312-sphinx8-needs5 -- tests/test_analyse.py
tox -e py312-sphinx8-needs5 -- tests/test_analyse.py::test_function_name

# Update syrupy snapshots
tox -e py312-sphinx8-needs5 -- --snapshot-update

# Type check / lint / format
tox -e mypy
tox -e ruff-check
tox -e ruff-fmt
pre-commit run --all-files

# Docs
tox -e docs-clean
tox -e docs-update
BUILDER=linkcheck tox -e docs-clean
tox -e docs-live

# End-to-end demo (analyse -> write RST -> build docs)
tox -e demo
```

The CLI itself is installed as `codelinks` (`codelinks analyse <config.toml>`,
`codelinks write rst <input.json> --outpath <file>`).

## Architecture

Pipeline: **Source Files → Discovery → Parsing → Analysis → Results (JSON) → RST Generation**

- `source_discover/` — finds source files by include/exclude patterns, respects `.gitignore`.
- `analyse/oneline_parser.py` — tree-sitter based parser extracting comment marker nodes.
- `analyse/projects.py` — per-language analyzers, registered in a `LANGUAGE_ANALYZERS` dict.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This describes an architecture that does not exist, and points future agents at a recipe that cannot be followed.

analyse/projects.py is 78 lines containing a single AnalyseProjects class. There is no LANGUAGE_ANALYZERS dict and no per-language analyzer class anywhere in the repo:

$ grep -rn LANGUAGE_ANALYZERS src/    # no matches

Line 66 then tells the reader that adding a language "follow[s] a short recipe documented in AGENTS.md ... follow those rather than inventing a new approach" — but that recipe (AGENTS.md:400-417) instructs creating a BaseAnalyzer subclass and registering it in LANGUAGE_ANALYZERS, which is fiction. This PR itself could not follow it; TypeScript support landed via SCOPE_NODE_TYPES + init_tree_sitter in analyse/utils.py, which the new file never mentions.

Also: line 13's language list omits Bash, which shipped in 1.4.0.

Since a wrong CLAUDE.md actively misdirects, it is worth either correcting these two sections against analyse/utils.py, or dropping the architecture section and deferring to AGENTS.md. Separately, adding CLAUDE.md is unrelated to issue #69 and would be easier to review as its own PR.

- `analyse/analyse.py` — orchestrates discovery + parsing + analysis into `analyse/models.py`
Pydantic result models.
- `needextend_write.py` — turns analysis JSON into RST with Sphinx-Needs `needextend`
directives.
- `config.py` — Pydantic v2 config models (`AnalyseConfig` etc.), loadable from TOML.
- `sphinx_extension/source_tracing.py` — the Sphinx extension `setup()`; wires into Sphinx
build events (`config-inited`, `builder-inited`, `env-before-read-docs`,
`html-collect-pages`, `html-page-context`, `build-finished`) to register sphinx-needs extra
options/types, generate standalone traced-source HTML pages, and inject CSS
(`sphinx_extension/ub_sct.css`). See AGENTS.md for the full event table and mermaid diagram.

Adding a new language analyzer, marker type, CLI command, or config option each follow a
short recipe documented in AGENTS.md under "Common Patterns" — follow those rather than
inventing a new approach.

## Code style

- Ruff for lint/format (strict rule set incl. `S`, `PL`, `PTH`, `SIM`, `SLF`; see
`pyproject.toml` for per-file ignores).
- Mypy strict mode (`disallow_any_*`, `disallow_untyped_*`); relaxed for `tests/*` and
`sphinx_codelinks.*` via overrides in `pyproject.toml`.
- Full type annotations everywhere; Pydantic models (frozen where possible) for config/data.
- Sphinx-style docstrings (`:param:`, `:return:`, `:raises:`), no types in docstrings.
- Prefer pure functions and immutable data structures.

## Testing

- `pytest` with fixtures in `tests/conftest.py`; test data in `tests/data/`; Sphinx
integration tests use real minimal Sphinx projects in `tests/doc_test/`.
- `syrupy` for snapshot testing of complex outputs (JSON, doctrees) — use
`snapshot.assert_match()` and re-run with `--snapshot-update` when output intentionally
changes.
- Use `@pytest.mark.parametrize` for multi-language / multi-scenario tests.
2 changes: 1 addition & 1 deletion docs/source/components/analyse.rst
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ Limitations

**Current Limitations:**

- **Language Support**: C/C++ (``//``, ``/* */``), C# (``//``, ``/* */``, ``///``), Python (``#``), YAML (``#``), Rust (``//``, ``/* */``, ``///``), Go (``//``, ``/* */``), JSONC (``//``, ``/* */``) and Bash (``#``) comment styles are supported
- **Language Support**: C/C++ (``//``, ``/* */``), C# (``//``, ``/* */``, ``///``), TypeScript/JavaScript (``//``, ``/* */``), Python (``#``), YAML (``#``), Rust (``//``, ``/* */``, ``///``), Go (``//``, ``/* */``), JSONC (``//``, ``/* */``) and Bash (``#``) comment styles are supported
- **Single Comment Style**: Each analysis run processes only one comment style at a time

Extraction Examples
Expand Down
8 changes: 7 additions & 1 deletion docs/source/components/configuration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -271,7 +271,7 @@ Specifies the comment syntax style used in the source code files. This determine

**Type:** ``str``
**Default:** ``"cpp"``
**Supported values:** ``"cpp"``, ``"python"``, ``"cs"``, ``"yaml"``, ``"rust"``, ``"go"``, ``"jsonc"``, ``"bash"``
**Supported values:** ``"cpp"``, ``"python"``, ``"cs"``, ``"ts"``, ``"yaml"``, ``"rust"``, ``"go"``, ``"jsonc"``, ``"bash"``

.. code-block:: toml

Expand Down Expand Up @@ -304,6 +304,12 @@ Specifies the comment syntax style used in the source code files. This determine
``/* */`` (multi-line),
``///`` (XML doc comments)
- ``.cs``
* - TypeScript / JavaScript
- ``"ts"``
- ``//`` (single-line),
``/* */`` (multi-line)
- ``.ts``, ``.tsx``, ``.mts``, ``.cts``, ``.js``, ``.jsx``, ``.mjs``
and ``.cjs``
* - YAML
- ``"yaml"``
- ``#`` (single-line)
Expand Down
10 changes: 10 additions & 0 deletions docs/source/components/discover.rst
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,13 @@ Usage Examples
include = []
exclude = ["tests/**", "setup.py"]
comment_type = "python"

**TypeScript Project:**

.. code-block:: toml

[source_discover]
src_dir = "./frontend"
include = ["**/*.ts", "**/*.tsx"]
exclude = ["**/*.test.ts", "**/*.spec.ts"]
comment_type = "ts"
32 changes: 32 additions & 0 deletions docs/source/components/features.rst
Original file line number Diff line number Diff line change
Expand Up @@ -266,6 +266,38 @@ Features
.. fault:: Sphinx-codelinks hallucinates traceability objects in Bash
:id: FAULT_BASH_2

.. feature:: TypeScript Language Support
:id: FE_TS

Support for defining traceability objects in TypeScript and JavaScript source
files via one-line comment annotations.

The TypeScript language parser leverages tree-sitter to accurately identify and
extract comments from TypeScript and JavaScript sources, including single-line
(``//``) and multi-line (``/* */``) comment styles. The grammar is chosen per
file from its extension: ``.ts``, ``.mts``, and ``.cts`` — TypeScript's own
module variants — are parsed with the plain TypeScript grammar, since a legacy
angle-bracket type assertion (``<string>x``) is valid there but is JSX syntax
under the TSX grammar. Every other extension (``.tsx``, ``.jsx``, ``.js``,
``.mjs``, ``.cjs``) is parsed with the TSX grammar, which is safe for plain
JavaScript and additionally handles JSX (including JSX comments such as
``{/* ... */}``) embedded in ``.tsx`` or ``.js`` sources.

Key capabilities:

* Detection of inline and block comments
* Association of comments with function, class, and method declarations
* ``const``/``let``/``var`` declarations count as scopes only when they assign
a function or arrow function
* File extensions ``.ts``, ``.tsx``, ``.mts``, ``.cts``, ``.js``, ``.jsx``,
``.mjs`` and ``.cjs`` auto-discovered when ``comment_type = "ts"``

.. fault:: Traceability objects are not detected in TypeScript language
:id: FAULT_TS_1

.. fault:: Sphinx-codelinks hallucinates traceability objects in TypeScript
:id: FAULT_TS_2

.. feature:: Preprocessor-Aware C/C++ Extraction
:id: FE_PREPROC

Expand Down
15 changes: 15 additions & 0 deletions docs/source/development/change_log.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,21 @@
Changelog
=========

Under development

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heading deviates from the established convention.

Every prior pre-release cycle in this file used Unreleased — e.g. at adf35cc the in-progress section was:

Unreleased
----------

The PR description also says the entry was added "under Upcoming", so the actual heading matches neither. Suggest Unreleased for consistency with the release tooling and history.

Suggested change
Under development
Unreleased
----------

-----------------

New and Improved
................

- ✨ Added TypeScript comment type support for source discovery and analysis.

TypeScript and JavaScript files can now be processed using ``comment_type = "ts"``.
The tree-sitter grammar is chosen per file from its extension: ``.ts``, ``.mts``,
and ``.cts`` use the plain TypeScript grammar, and everything else (``.tsx``,
``.jsx``, ``.js``, ``.mjs``, ``.cjs``) falls back to the TSX grammar. Source
discovery supports ``.ts``, ``.tsx``, ``.mts``, ``.cts``, ``.js``, ``.jsx``,
``.mjs`` and ``.cjs`` extensions by default.

.. _`release:1.4.0`:

1.4.0
Expand Down
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ dependencies = [
# https://github.com/tree-sitter/py-tree-sitter/issues/386#issuecomment-3101430799
"tree-sitter~=0.25.1",
"tree-sitter-c-sharp>=0.23.1",
"tree-sitter-typescript>=0.23.2",
"tree-sitter-yaml>=0.7.1",
"tree-sitter-rust>=0.23.0",
"tree-sitter-go>=0.23.0",
Expand Down
23 changes: 22 additions & 1 deletion src/sphinx_codelinks/analyse/analyse.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
from typing import Any, TypedDict, cast

from tree_sitter import Node as TreeSitterNode
from tree_sitter import Parser, Query

from sphinx_codelinks.analyse import utils
from sphinx_codelinks.analyse.models import (
Expand Down Expand Up @@ -97,9 +98,29 @@ def get_src_strings(self) -> Generator[tuple[Path, bytes], Any, None]: # type:
yield src_path, text.encode("utf-8")

def create_src_objects(self) -> None:
parser, query = utils.init_tree_sitter(self.analyse_config.comment_type)
comment_type = self.analyse_config.comment_type
# One (parser, query) pair per distinct grammar actually needed, built
# lazily so a parser is never rebuilt per file. Every comment type
# except TypeScript uses a single grammar for the whole run;
# TypeScript alone varies its grammar per file (utils.ts_grammar_key)
# because a legacy TypeScript-only cast parses as JSX under the wrong
# grammar — see the CommentType.ts branch of utils.init_tree_sitter.
parser_cache: dict[str, tuple[Parser, Query]] = {}

for src_path, src_string in self.get_src_strings():
# `comment_type` is normally a CommentType member, but a few call
# sites carry it as a plain (possibly invalid) str instead — see
# SourceAnalyseConfig.comment_type — so key on `str(comment_type)`
# rather than `.value`, which only the enum has.
cache_key = (
utils.ts_grammar_key(src_path)
if comment_type == CommentType.ts
else str(comment_type)
)
if cache_key not in parser_cache:
parser_cache[cache_key] = utils.init_tree_sitter(comment_type, src_path)
parser, query = parser_cache[cache_key]

comments: list[TreeSitterNode] | None = utils.extract_comments(
src_string, parser, query
)
Expand Down
Loading
Loading