Skip to content

feat: add namespaces support#2033

Open
jfrancoa wants to merge 14 commits into
dev/1.38from
jose/namespaces
Open

feat: add namespaces support#2033
jfrancoa wants to merge 14 commits into
dev/1.38from
jose/namespaces

Conversation

@jfrancoa
Copy link
Copy Markdown
Contributor

@jfrancoa jfrancoa commented May 5, 2026

Summary

  • Adds client.namespaces with full CRUD: create, update, get, list_all, delete (requires Weaviate 1.38.0+)
  • create(name=..., home_node=...) accepts an optional home_node to pin shard placement; when omitted the cluster auto-selects
  • Adds update(name=..., home_node=...) (PUT /namespaces/{name}) to change a namespace's home node — affects future placement only, existing shards are not moved
  • Namespace model now exposes home_node and the read-only state (active/deleting), parsed from create/update/get/list responses; NamespaceState exported from weaviate.outputs.namespaces
  • Adds Permissions.namespaces(namespace=..., manage=True) RBAC factory and NamespacesAction.MANAGE enum, wired into Role parsing
  • Extends client.users.db.create(user_id=..., namespace=...) to bind a DB user to a namespace; UserDB.namespace field populated from server responses
  • Adds ci/docker-compose-namespaces.yml for integration test fixtures (ports 8094/50064, RBAC + dynamic DB users + NAMESPACES_ENABLED=true)
  • Tests: 14 unit (test/test_namespaces.py), 16 mock (mock_tests/test_namespaces.py), 9 integration (integration/test_namespaces.py), all passing

Test plan

  • pytest test/test_namespaces.py mock_tests/test_namespaces.py — unit + mock tests (no server required); mock tests pin the exact wire format for create (with/without home_node), update (PUT body + 200), get (404→None), list, and delete (202)
  • Start namespaces instance: WEAVIATE_VERSION=1.38.x docker compose -f ci/docker-compose-namespaces.yml up -d
  • pytest integration/test_namespaces.py — integration tests covering create/update/get/list/delete namespace, home_node create + update + read-back, namespaced user creation, and RBAC permission management

🤖 Generated with Claude Code

@jfrancoa jfrancoa requested a review from a team as a code owner May 5, 2026 14:15
Copy link
Copy Markdown

@orca-security-eu orca-security-eu Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orca Security Scan Summary

Status Check Issues by priority
Passed Passed Secrets high 0   medium 0   low 0   info 0 View in Orca

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@jfrancoa
Copy link
Copy Markdown
Contributor Author

jfrancoa commented May 5, 2026

@copilot review

Copy link
Copy Markdown

Copilot AI commented May 5, 2026

@copilot review

Here's a summary of the review findings and fixes applied:

CI Failure (already fixed in c636cb3):
The integration tests failed with TypeError: Role.__init__() missing 1 required positional argument: 'namespaces_permissions' because the new namespaces_permissions field was added as a required field to the Role dataclass, but integration/test_rbac.py had many direct Role(...) instantiations that were missing it.

Additional fix (f2babd2):
NamespacesPermissionOutput was not exported from weaviate/outputs/rbac.py, breaking the codebase convention that all public RBAC permission output classes are re-exported from that module. Fixed by adding it to the imports and __all__ list.

All 12 unit tests pass, and both Code Review and CodeQL Security scans are clean.

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented May 6, 2026

Codecov Report

❌ Patch coverage is 83.89831% with 76 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (dev/1.38@5a4b983). Learn more about missing BASE report.

Files with missing lines Patch % Lines
integration/test_namespaces.py 35.59% 76 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##             dev/1.38    #2033   +/-   ##
===========================================
  Coverage            ?   86.66%           
===========================================
  Files               ?      307           
  Lines               ?    23416           
  Branches            ?        0           
===========================================
  Hits                ?    20294           
  Misses              ?     3122           
  Partials            ?        0           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jfrancoa
Copy link
Copy Markdown
Contributor Author

jfrancoa commented May 8, 2026

@copilot review

Copy link
Copy Markdown

Copilot AI commented May 8, 2026

@copilot review

CI is green at HEAD (5629aee): both the Main and PR Security Lint jobs passed. The single "Main" failure at f2babd2 was caused by two pre-existing flaky tests:

  • test_authentication_user_pw — OAuth rate limit from Okta (unrelated to our changes)
  • test_embedded_create_restore_backup — gRPC DEADLINE_EXCEEDED (unrelated to our changes)

Here's a summary of all the changes in the PR:

New code

  • weaviate/namespaces/ — executor-pattern CRUD module (create/get/list_all/delete), with check_is_at_least_1_38_0 version guard on all methods
  • weaviate/outputs/namespaces.py — re-exports Namespace model
  • weaviate/outputs/rbac.py — added NamespacesPermissionOutput to imports and __all__

rbac/models.py

  • NamespacesAction.MANAGE enum, _NamespacesPermission/NamespacesPermissionOutput classes
  • Actions.Namespaces, Permissions.namespaces() factory, Role.namespaces_permissions field and parsing in _from_weaviate_role

users

  • users.db.create() — optional namespace param, omitted from body when None (prevents sending "namespace": null to older clusters)
  • users.db.get()/list_all() — populate UserDB.namespace from server response
  • UserDB.namespace: Optional[str] = None field

util.py

  • _capitalize_first_letter — handles namespace:Collection by capitalizing only the collection portion
  • check_is_at_least_1_38_0 version helper

Tests

  • 12 unit tests (test/test_namespaces.py, test/test_util.py)
  • Mock tests covering HTTP contract, version guards, and request-body shape (mock_tests/test_namespaces.py)
  • 7 integration tests (integration/test_namespaces.py)
  • Fixed integration/test_rbac.py Role() calls to include the new namespaces_permissions=[] field

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 3 comments.

Comment thread weaviate/util.py
Comment thread weaviate/users/base.py Outdated
Comment thread ci/compose.sh Outdated
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated no new comments.

@jfrancoa jfrancoa force-pushed the jose/namespaces branch 3 times, most recently from 4d9912b to 6a2b620 Compare May 28, 2026 07:48
jfrancoa and others added 5 commits May 28, 2026 10:48
Adds full CRUD support for the Weaviate namespaces feature (requires
Weaviate 1.38.0+): create/get/list/delete namespace endpoints, RBAC
manage_namespaces permission, namespace-scoped DB user creation, and
UserDB.namespace field. Includes unit and integration test suites with
a dedicated docker-compose fixture.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add namespaces_permissions=[] to all Role(...) constructions in
  integration/test_rbac.py to match the new required dataclass field
- Wait for namespaces port (8094) in ci/compose.sh
- Use _decode_json_response_list helper in namespaces.list_all
- Round-trip via roles.get(...) in namespace permission integration tests
- Use pass instead of return None in delete callback

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When a global user (operator) passes a namespaced collection name of the
form "namespace:CollectionName" (required on namespace-enabled clusters),
only the collection portion after the colon should be capitalized. The
namespace prefix must stay lowercase as it follows [a-z][a-z0-9]*.

This single-point fix covers all 30+ call sites that use
_capitalize_first_letter, including collections.delete/get/exists,
batch operations, RBAC permissions, backup/export, and filters.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds 15 mock-based tests in mock_tests/test_namespaces.py covering the
HTTP contract for the new module without requiring a Weaviate cluster.
Brings weaviate/namespaces/base.py from 33% to 100% coverage and exercises
the new code paths in weaviate/users/base.py (namespace request body and
UserDB.namespace response field).

Notable cases that catch real regressions:

- 404 on get -> None (guards ok_in=[200, 404])
- list_all on null/empty body -> [] (guards the "or []" fallback)
- Every public method asserts the 1.38.0 version guard (catches future
  contributors lowering the minimum or adding a method without the guard)
- POST /v1/users/db/{id} body shape with and without namespace
- UserDB.namespace populated from server response and defaulting to None

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
jfrancoa and others added 6 commits May 28, 2026 10:48
- Capture and assert the create() request body is empty JSON, locking the
  contract that the namespace name lives in the URL path only
- Annotate method_call parametrize argument as Callable
- Use Dict[str, Any] instead of bare dict for captured request bodies

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The server returns 202 Accepted for DELETE /namespaces/{name} because
cleanup of classes, aliases, and namespaced users completes
asynchronously. The client was declaring ok_in=[204], so successful
deletions surfaced as errors even though the server-side delete went
through. Align the status code with the spec and document the async
behaviour, and update the mock test accordingly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The server gained an optional home_node on namespace create plus a new
PUT /namespaces/{name} to modify it, and a read-only state field on the
namespace object. Bring the client to parity:

- Namespace model gains home_node and state (Literal active/deleting)
- create() accepts optional home_node; new update() backed by PUT
- create/get/list/update share a _ns_from_dict parser
- sync/async .pyi stubs updated; NamespaceState exported from outputs
- mock, unit, and integration tests for the new fields and update()

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Order the ports following alphabetical order of docker-compose files.
- Add verification of version in user creation if passing namespace.
Codecov flagged 71 missing patch lines, all in integration/test_namespaces.py.
The namespaces integration tests are version-gated to Weaviate 1.38+, but the
CI integration matrix only runs 1.36, so they are skipped and their bodies
count as uncovered. The bare `--cov` in the pytest invocation pulls test files
into the report, and .coveragerc only omitted *tests*/* (mock_tests/), missing
integration/.

Add *integration/* to the omit lists. Test files are not coverage subjects, and
the coverage they contribute to weaviate/ (via --cov=weaviate) is unaffected.
Production code (weaviate/namespaces/*) remains 100% covered by mock+unit tests.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
And add missing REPLICATION_MAXIMUM_FACTOR required
for namespaces.
jfrancoa and others added 3 commits May 28, 2026 12:45
test_delete_namespace asserted that get() returns None immediately after
delete(), but namespace deletion is asynchronous: delete() returns 202, the
server marks the namespace state="deleting", and a background sweep removes it
later (NAMESPACE_CLEANUP_INTERVAL, default 30s). Against a real 1.38 cluster
get() therefore returned Namespace(state="deleting"), failing the assertion.

Poll until the namespace is gone (asserting it stays in "deleting" while
present) with a 60s timeout that tolerates the default sweep interval. Also set
NAMESPACE_CLEANUP_INTERVAL=2s in the namespaces CI compose so the sweep runs
quickly and the test finishes in ~1s instead of ~30s.

Verified against a live 1.38.0-rc.0 cluster: all 9 namespace integration tests
pass (delete completes in ~0.6s with the shortened interval).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The server no longer accepts a separate `namespace` field when creating a DB
user; the namespace is now derived from a namespace-qualified id of the form
"<namespace>:<user>" passed in the URL path (weaviate PR #11501,
resolveUserKeyForCreate). Update the client to match:

- users.db.create: remove the `namespace` parameter; always POST an empty body.
  The (optionally qualified) id is already carried in the path. Drop the
  now-unused 1.38 version guard tied to the namespace field.
- sync/async .pyi stubs: update the create signature accordingly.
- UserDB.namespace and get/list parsing are unchanged: the server still returns
  `namespace` in the user response for global operators (handlers_db_users.go).

Tests:
- mock_tests: replace the namespace-in-body tests with
  test_users_db_create_qualified_user_id_goes_in_path (asserts the qualified id
  lands in the URL path and the body stays empty) and
  test_users_db_create_posts_empty_body. These pin the wire contract and are the
  version-independent regression guard.
- integration: create the namespaced user with the qualified id directly.
- Bump WEAVIATE_138 to 1.38.0-rc.0-b9ea106, the first build containing PR #11501.

Verified: mock + unit suites pass (34); 8/9 namespace integration tests pass
against a live cluster. The qualified-create integration test requires the
b9ea106 image, which was still publishing to Docker Hub at commit time; the
server source at b9ea106 is git-verified to implement this contract and CI runs
the live check.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First published build containing the namespace-qualified user-create change
(PR #11501). Verified end-to-end: all 9 namespace integration tests pass against
this image, including test_create_namespaced_user with a "<namespace>:<user>" id.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants