Query the in-memory graph with Cypher

[paracycle]

Goal

Give clients a flexible, future-proof way to query the in-memory graph using Cypher — the de facto standard graph query language. Instead of adding a bespoke method for every traversal, this exposes the graph through a query language clients already know, so new introspection needs become queries rather than new APIs. Queries are read-only, run directly against the existing in-memory Graph (no duplication, no embedded database).

References

• openCypher specification: https://opencypher.org/resources/
• Cypher query language (Neo4j docs): https://neo4j.com/docs/cypher-manual/current/

Architecture

The Cypher engine itself — lexer, recursive-descent parser, AST, the tree-walking executor, values, and result formatting — lives in a separate, published crate, cypher-parser. The executor is generic over a GraphProvider trait, so it has no dependency on rubydex.

rubydex depends on cypher-parser and provides only the rubydex-specific pieces:

• query::cypher::schema — impl GraphProvider for Graph, the property-graph mapping.
• query::cypher::schema_info — the static schema description.

How it's exposed

• rubydex_cli: --query "<CYPHER>", --schema, --format table|json.
• Ruby API — the whole query API lives on Rubydex::Query:
  • Rubydex::Query.parse(str) → an opaque, reusable parsed query (raises ArgumentError on syntax errors, needs no graph).
  • Rubydex::Query#render(graph, format = :table) → runs a parsed query against a graph and returns the formatted output (table or JSON).
  • Rubydex::Query.schema(format = :table) → describes the queryable schema.
• rdx command CLI:

  rdx query <CYPHER>  [--format table|json]
  rdx query --schema  [--format table|json]
  rdx console
  

Parse first, then build the graph

Both CLIs parse the query into the opaque parsed object before indexing/resolution, so a malformed query fails fast (~0.1s) instead of after a full workspace index:

parse query  ->  build graph (index + resolve)  ->  run parsed query

A parsed Rubydex::Query is reusable: parse once, run against many graphs.

Graph schema exposed to queries

rdx query --schema / rubydex_cli --schema / Rubydex::Query.schema print this model:

Node labels: Document, Definition, Declaration, the grouping label Namespace, and declaration kind sub-labels (Class, Module, SingletonClass, Method, Constant, ConstantAlias, GlobalVariable, InstanceVariable, ClassVariable).

Relationship types: DEFINES (Document→Definition), DECLARES (Definition→Declaration), CONTAINS (lexical nesting), HAS_PARENT (direct superclass; reverse-traverse for direct subclasses, * for the chain), INCLUDES/PREPENDS/EXTENDS (mixins), OWNS (members), HAS_ANCESTOR (linearized ancestor chain, incl. modules), HAS_DESCENDANT (reverse of HAS_ANCESTOR), REFERENCES (Document→Declaration).

Properties: Declaration: name, unqualified_name, kind, visibility, definition_count; Definition: kind, name, file, line; Document: uri, path (file-system path), name (base file name).

Supported syntax: MATCH (node patterns with label disjunction :A|B and inline properties; relationship patterns with direction, type lists, and variable length *min..max), WHERE (=, <>, <, <=, >, >=, CONTAINS, STARTS WITH, ENDS WITH, AND/OR/NOT), RETURN (DISTINCT, AS, aggregates count/collect/min/max/sum/avg), ORDER BY, SKIP, LIMIT. Read-only; write clauses are intentionally unsupported.

Try it

# Discover the model
rdx query --schema

# All classes or modules
rdx query "MATCH (n:Class|Module) RETURN n.name ORDER BY n.name"

# All (transitive) subclasses of a base class, as JSON
rdx query "MATCH (c:Class)-[:HAS_PARENT*1..]->(p {name: 'ApplicationRecord'}) RETURN DISTINCT c.name" --format json

# Count definitions per file
rdx query "MATCH (d:Document)-[:DEFINES]->(def:Definition) RETURN d.path, count(def) AS defs ORDER BY defs DESC"

From Ruby:

query = Rubydex::Query.parse("MATCH (n:Class|Module) RETURN n.name")  # fails fast on bad syntax
graph = Rubydex::Graph.new
graph.index_workspace
graph.resolve
puts query.render(graph, :json)

What's here

Core work:

1. A read-only Cypher engine over the in-memory graph, extracted into the standalone cypher-parser crate (generic over GraphProvider).
2. rubydex's GraphProvider mapping plus a single-source schema catalog (RelType + NODE_LABELS/NODE_PROPERTIES), rendered by schema_info.
3. --query/--schema on rubydex_cli and a rdx query / console command CLI, parsing the query up front so a syntax error fails fast before indexing.

Review refinements folded in: direction-correct HAS_PARENT/HAS_ANCESTOR/HAS_DESCENDANT edge naming, rdx query --schema as a flag, a single "build graph once, then dispatch" CLI structure, extraction of Rubydex::Query into query.c and a shared utils helper, robust file:// path handling, and clap validation that rejects nonsensical flag combinations.

[vinistock]

Still trying to wrap my head around the entire engine, but left some comments already. Excited to have a unified way of querying the graph.

I wonder if there's some IRB trick we can use to enter a "query" mode that accepts the Cypher queries directly (non-valid Ruby). Something like:

bundle exec rdx -i
Indexing...
Resolving...
> graph["Foo"]
=> <Declaration ...>
>
> query_mode!
> MATCH (n:Class|Module) RETURN n.name ORDER BY n.name
=> [Foo]

[paracycle]

I wonder if there's some IRB trick we can use to enter a "query" mode that accepts the Cypher queries directly (non-valid Ruby). Something like:

bundle exec rdx -i
Indexing...
Resolving...
> graph["Foo"]
=> <Declaration ...>
>
> query_mode!
> MATCH (n:Class|Module) RETURN n.name ORDER BY n.name
=> [Foo]

Great idea. I am prototyping what's possible here, but obviously it won't be a part of this PR.

[paracycle]

Ok, this PR has a prototype for the console mode extension: #883

[paracycle]

@vinistock If it is helpful, I had this diagram generated from the codebase for how the query engine works at a high level:

[Morriar]

Why did we need our own implementation of the Cypher parser? None of the existing Rust implementations could be reused?

Could you clean the commit history to make this PR easier to read? First commit seems to introduce a whole parser that gets removed later.

[paracycle]

Why did we need our own implementation of the Cypher parser? None of the existing Rust implementations could be reused?

The existing implementations are either unmaintained, or were doing much more than what we needed, or were pulling in a lot of dependencies. I wanted to start with a parser under our control and with no dependencies, that support only the subset of the syntax that we want to support (e.g. we will probably never need CREATE queries for building the schema).

If needed, we can swap what we have with another library down the line, but this seems to work just fine.

Could you clean the commit history to make this PR easier to read? First commit seems to introduce a whole parser that gets removed later.

Sure, I can do that tomorrow.

[vinistock]

Just some minor things, but I think we can ship and iterate on it.

Note: it seems that, like query.rs, cypher queries are going to be quite heavy on graph traversal (depending on what the user is searching for of course). If possible, it would be nice to scan the graph in parallel

[paracycle]

@vinistock and @Morriar: I think I've addressed all the comments and I have a clean commit history now. Can you both do a final review?