Skip to content

Add process-local memory backend#135

Merged
schapman1974 merged 1 commit into
masterfrom
agent/memory-backend
Jul 15, 2026
Merged

Add process-local memory backend#135
schapman1974 merged 1 commit into
masterfrom
agent/memory-backend

Conversation

@schapman1974

Copy link
Copy Markdown
Owner

Summary

This adds the roadmap's process-local in-memory backend and closes #69.

  • Add backend="memory" support to TinyMongoClient and MongoClient.
  • Give unnamed clients isolated private namespaces that are removed on close.
  • Support explicit same-process sharing through named memory://NAME namespaces.
  • Add memory to the shared MongoDB compatibility contract matrix.
  • Document the public API, lifecycle, intended uses, and limitations.

Why

Tests and temporary workloads currently need a filesystem-backed database even
when persistence is unwanted. The memory backend provides a zero-file,
zero-extra-dependency option while preserving TinyMongo's existing collection
API and JSON-compatible value behavior. It also establishes the backend needed
before the roadmap's async work can be developed and tested cleanly.

Behavior and design

Isolated clients

client = TinyMongoClient(backend="memory")

Each unnamed client receives a unique namespace. It does not create database or
lock files, and its namespace is cleared when the client closes.

Named sharing

writer = TinyMongoClient("memory://shared-test", backend="memory")
reader = TinyMongoClient("memory://shared-test", backend="memory")

Named clients share data inside the current Python process. Named data can be
reopened after a client closes and remains available until process exit.

Correctness

  • A process-local registry holds TinyDB-compatible storage entries by database
    address.
  • Every database entry has a reentrant lock so read/modify/write operations from
    multiple clients are atomic.
  • Revision tracking is recorded per retained collection handle, so one
    collection refresh cannot hide changes from another collection.
  • External writes invalidate cached queries and equality indexes before the
    next read.
  • Reads and writes use deep copies and JSON serialization rules, matching the
    default JSON backend's value behavior.
  • Anonymous cleanup verifies registry-entry identity so concurrent closes
    cannot remove a newly created replacement entry.
  • Invalid or mistyped URI schemes fail clearly instead of silently creating an
    isolated database.

Capabilities and limits

  • No disk files and no new runtime dependency.
  • Thread-safe for clients in one process.
  • Not persistent and not safe for sharing between processes.
  • Intentionally exposed through the Python API only; a one-shot CLI process
    cannot make useful use of process-local state.
  • Existing persistent backend locking behavior is unchanged.

Tests

The focused memory suite covers CRUD, isolation, named sharing, lifecycle
behavior, copying and serialization, cache refreshes, retained collection
handles, equality-index invalidation, duplicate IDs, threaded inserts,
concurrent cleanup, URI validation, capability reporting, and both client
classes.

Validation completed locally:

  • 181 passed, 47 deselected, 8 xfailed
  • 100% statement coverage and 100% branch coverage
  • Ruff, Black, mypy, and git diff --check pass
  • Python 3.9.6 with minimum supported TinyDB 3.2.1 passes the focused suite and
    contracts
  • Real MongoDB 8.0 contract matrix: 28 passed, 8 xfailed

The eight xfails are existing, explicitly documented compatibility gaps;
memory inherits the cursor pagination gap tracked by #73.

Closes #69

@schapman1974

Copy link
Copy Markdown
Owner Author

@davidlatwe, this PR adds the in-memory backend discussed in #69. It supports private test databases and named databases shared within one process, with tests for locking and stale caches. Would you be willing to review it? GitHub will let me add the formal review request after the collaborator invitation is accepted.

@schapman1974
schapman1974 merged commit e236b54 into master Jul 15, 2026
4 checks passed
@davidlatwe

Copy link
Copy Markdown
Contributor

Hi @schapman1974

Thanks for the invite , but its a bit much to read through for my limited free time (already hard for me to work on my montydb!)

Since this project is opensourced, I recommend you try https://www.coderabbit.ai/ for review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add an in-memory storage backend

4 participants