Re-split OpenSearch indices per service

docs/architecture.md

@@ -11,3 +11,51 @@ flowchart TD
     User -- HTTP --> Dashboard --> Opensearch
     Back -- REST API --> Embedding Endpoint
 ```
+
+### Per-service index topology
+
+Each registered service gets its own OpenSearch index. The index name is derived from the
+`OPENSEARCH_INDEX_PREFIX` setting and the service's name:
+
+```
+{OPENSEARCH_INDEX_PREFIX}-{service.name}
+```
+
+```
+Services (bearer token auth)          OpenSearch
++------------------+                  +------------------+
+| docs service     | -- /index/ -->   | find-docs        |
+|                  | -- /delete/ -->  |                  |
++------------------+                  +------------------+
+| drive service    | -- /index/ -->   | find-drive       |
+|                  | -- /delete/ -->  |                  |
++------------------+                  +------------------+
+| wiki service     | -- /index/ -->   | find-wiki        |
+|                  | -- /delete/ -->  |                  |
++------------------+                  +------------------+
+                                              |
+                                              | fan-out across
+                                              | all active indices
+                                              |
+Users (OIDC token auth)                       v
++------------------+                  +------------------+
+| OIDC user        | -- /search/ -->  | find-docs        |
+|                  |                  | find-drive       |
+|                  |                  | find-wiki        |
++------------------+                  +------------------+
+```
+
+**Services cannot search. Only users (OIDC tokens) can search. Services can delete from their
+own index.**
+
+- `/index/` accepts service bearer tokens (`ServiceTokenAuthentication`). Each service writes
+  exclusively to its own index.
+- `/delete/` accepts service bearer tokens (`ServiceTokenAuthentication`). Each service deletes
+  exclusively from its own index.
+- `/search/` accepts OIDC user tokens (`ResourceServerAuthentication`). Queries fan out across all
+  indices belonging to `is_active=True` services.
+
+Setting `Service.is_active=False` removes that service's index from the search fan-out. Documents
+in that index become invisible to users until the service is re-activated.
+
+See [services-cannot-search.md](./services-cannot-search.md) for the full contract.

docs/services-cannot-search.md

@@ -0,0 +1,56 @@
+# Contract: Services Cannot Search
+
+Services registered in Find can index and delete documents, but they cannot search. Only users
+authenticated via OIDC tokens can call `/search/`.
+
+## Authentication model
+
+Find exposes three endpoints, each with a distinct authentication requirement:
+
+| Endpoint    | Auth class                    | Who can call it          |
+|-------------|-------------------------------|--------------------------|
+| `/index/`   | `ServiceTokenAuthentication`  | Services (bearer token)  |
+| `/search/`  | `ResourceServerAuthentication` + `ResourceServerMixin` | OIDC users only |
+| `/delete/`  | `ServiceTokenAuthentication`  | Services (bearer token)  |
+
+`ServiceTokenAuthentication` validates the `Authorization: Bearer <token>` header against the
+`Service.token` field in the database. It is wired to `IndexDocumentView` and
+`DeleteDocumentsView`.
+
+`ResourceServerAuthentication` (from `django-lasuite`) validates OIDC access tokens issued by
+the configured identity provider. It is wired to `SearchDocumentView` via `ResourceServerMixin`.
+
+## What happens when a service tries to search
+
+A service bearer token posted to `/search/` is rejected with a 4xx status (typically
+**HTTP 400 Bad Request** from `django-lasuite`'s `SuspiciousOperation` when introspection
+fails, or **HTTP 401/403** if the token never reaches introspection).
+
+`ResourceServerAuthentication` does not recognise service bearer tokens. The request is rejected
+before any OpenSearch query runs.
+
+## Why this contract exists
+
+Services write to and delete from their own isolated index. They have no business reading across
+other services' indices. Cross-service fan-out is a user-facing feature: when a user searches,
+Find queries all active service indices and merges the results, filtered by the user's identity
+(`sub` claim).
+
+Allowing services to search would break the access-control model: a service could read documents
+belonging to users it has no relationship with.
+
+## Test coverage
+
+The following test scenarios lock this contract in:
+
+- `test_api_documents_index.py` verifies that `/index/` rejects OIDC-style bearer tokens
+  (HTTP 403 `Invalid token.`).
+- `test_api_documents_search.py` verifies that `/search/` rejects service bearer tokens and
+  anonymous requests (HTTP 4xx).
+- `test_api_documents_delete.py` verifies that `/delete/` accepts service bearer tokens and rejects
+  OIDC user tokens.
+- `test_integration_per_service_indices.py` verifies the full round-trip: services index and delete
+  documents from their own indices, users search and find their accessible documents.
+
+If you add a new endpoint that should be user-only, wire `ResourceServerAuthentication` and
+`ResourceServerMixin` to it. Do not add `ServiceTokenAuthentication` to search paths.

docs/setup-indexer.md

@@ -12,7 +12,7 @@ Add the following variables to your Django settings to configure Find
 and enable full-text search.
 
 ```python
-# Login for opensearch
+# Login for opensearch
 OPENSEARCH_USER=opensearch-user
 OPENSEARCH_PASSWORD=your-opensearch-password
 
@@ -23,10 +23,33 @@ OPENSEARCH_PORT=9200
 # Enable SSL for opensearch connection (False in dev mode)
 OPENSEARCH_USE_SSL=True
 
-# Unified index name for all services
-OPENSEARCH_INDEX=find
+# Prefix for per-service indices (each service gets its own index: {prefix}-{service.slug})
+OPENSEARCH_INDEX_PREFIX=find
 ```
 
+### Per-service indices
+
+Find creates one OpenSearch index per registered service. The index name follows the pattern:
+
+```
+{OPENSEARCH_INDEX_PREFIX}-{service.slug}
+```
+
+For example, with `OPENSEARCH_INDEX_PREFIX=find`:
+
+- A service with slug `docs` writes to `find-docs`
+- A service with slug `drive` writes to `find-drive`
+
+Indices are created lazily on the first write. You don't need to create them manually.
+
+When a service has `is_active=False`, its token can no longer index or delete, and its index is
+excluded from search fan-out. Documents in that index are effectively hidden from users until the
+service is re-activated.
+
+`Service.slug` is auto-derived from `name` on creation (lowercase, alphanumeric only) and is
+immutable thereafter. `name` is a free-form display field and can be edited freely. If you need
+a different slug, create a new service and migrate your documents.
+
 ### Language
 
 Language specific operations are applied to document titles and contents to improve search results.
@@ -64,13 +87,14 @@ Other applications can index their files through the **`/index/`** endpoint with
 For each application a new **Service** must be created through the admin interface
 (see http://localhost:9071/admin/core/service/add/)
 
-| Field                       | Description                                        |
-|-----------------------------|----------------------------------------------------|
-| Name                        | Name of the service (used as `service` field) |
-| Is active                   | Toggle service availability                        |
-| Client id                   | Calling service client_id (e.g `impress` for docs) |
-| Allowed services for search | List of sub-services. Will add the results from all these services<br>to the search results. |
-| Token (_read-only_)         | Random token for calling service authentication    |
+| Field               | Description                                                    |
+|---------------------|----------------------------------------------------------------|
+| Name                | Human-readable display name (editable)                         |
+| Slug (_read-only_)  | Stable identifier used as `service` field and index suffix; auto-derived from Name on creation (lowercase alphanumeric only); immutable thereafter |
+| Is active           | Toggle service availability (inactive services are excluded from search fan-out) |
+| Client id           | Calling service client_id (e.g `impress` for docs)            |
+| Token (_read-only_) | Random token for calling service authentication                |
+| Allowed services for search | _Currently unused._ Legacy many-to-many field retained on the model for possible future use; search now fans out across all active services regardless of this setting |
 
 And add the key in the calling application Django settings.
 

env.d/development/common.dist

@@ -15,7 +15,7 @@ FIND_BASE_URL="http://localhost:8072"
 # Opensearch
 OPENSEARCH_PASSWORD=find.PASS123
 OPENSEARCH_USE_SSL=false
-OPENSEARCH_INDEX=find
+OPENSEARCH_INDEX_PREFIX=find
 
 # OIDC
 OIDC_OP_JWKS_ENDPOINT=http://nginx:8083/realms/impress/protocol/openid-connect/certs

src/backend/core/authentication.py

@@ -31,7 +31,7 @@ def authenticate(self, request):
     def authenticate_credentials(self, token):
         """Check that the token is registered and valid."""
         try:
-            service = self.model.objects.only("name").get(token=token, is_active=True)
+            service = self.model.objects.only("slug").get(token=token, is_active=True)
         except self.model.DoesNotExist as excpt:
             raise exceptions.AuthenticationFailed("Invalid token.") from excpt
 

src/backend/core/factories.py

@@ -50,7 +50,8 @@ class ServiceFactory(factory.django.DjangoModelFactory):
     A factory for generating service instances for testing and development purposes.
     """
 
-    name = factory.Sequence(lambda n: f"test-index-{n!s}")
+    slug = factory.Sequence(lambda n: f"testidx{n!s}")
+    name = factory.LazyAttribute(lambda o: f"Test Service {o.slug}")
     created_at = factory.Faker("date_time_this_year", tzinfo=None)
     is_active = True
     client_id = "some_client_id"

src/backend/core/migrations/0003_service_slug_and_editable_name.py

@@ -0,0 +1,73 @@
+import re
+
+import django.core.validators
+from django.db import migrations, models
+
+
+def backfill_slug_from_name(apps, schema_editor):
+    Service = apps.get_model("core", "Service")  # noqa: N806
+    seen = set()
+    for service in Service.objects.all():
+        derived = re.sub(r"[^a-zA-Z0-9]", "", service.name or "").lower()
+        if not derived:
+            raise RuntimeError(
+                f"Cannot derive slug for Service id={service.pk!r} "
+                f"name={service.name!r}: name contains no alphanumeric characters."
+            )
+        if derived in seen:
+            raise RuntimeError(
+                f"Slug collision while backfilling: name={service.name!r} -> "
+                f"slug={derived!r} already used by another service."
+            )
+        seen.add(derived)
+        service.slug = derived
+        service.save(update_fields=["slug"])
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("core", "0002_service_client_id_service_services"),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name="service",
+            name="slug",
+            field=models.CharField(max_length=20, null=True),
+        ),
+        migrations.RunPython(
+            backfill_slug_from_name,
+            reverse_code=migrations.RunPython.noop,
+        ),
+        migrations.AlterField(
+            model_name="service",
+            name="slug",
+            field=models.SlugField(
+                editable=False,
+                help_text=(
+                    "Stable identifier used in the OpenSearch index name. "
+                    "Lowercase alphanumeric only. Set on creation, immutable thereafter."
+                ),
+                max_length=20,
+                unique=True,
+                validators=[
+                    django.core.validators.RegexValidator(
+                        message="Slug must contain only lowercase letters and digits.",
+                        regex="^[a-z0-9]+$",
+                    )
+                ],
+            ),
+        ),
+        migrations.AlterField(
+            model_name="service",
+            name="name",
+            field=models.CharField(max_length=255),
+        ),
+        migrations.AddConstraint(
+            model_name="service",
+            constraint=models.CheckConstraint(
+                condition=models.Q(("slug__regex", "^[a-z0-9]+$")),
+                name="slug_alphanumeric_only",
+            ),
+        ),
+    ]

src/backend/core/models.py

@@ -1,16 +1,23 @@
 """Models for find's core app"""
 
+import re
 import secrets
 import string
 
 from django.contrib.auth.models import AbstractUser
+from django.core.exceptions import ValidationError
+from django.core.validators import RegexValidator
 from django.db import models
 from django.db.models.functions import Length
-from django.utils.text import slugify
 from django.utils.translation import gettext_lazy as _
 
 models.CharField.register_lookup(Length)
 TOKEN_LENGTH = 50
+SLUG_REGEX = r"^[a-z0-9]+$"
+SLUG_VALIDATOR = RegexValidator(
+    regex=SLUG_REGEX,
+    message=_("Slug must contain only lowercase letters and digits."),
+)
 
 
 class User(AbstractUser):
@@ -20,15 +27,23 @@ class User(AbstractUser):
 class Service(models.Model):
     """Service registered to index its documents to our find"""
 
-    name = models.SlugField(max_length=20, unique=True)
+    name = models.CharField(max_length=255)
+    slug = models.SlugField(
+        max_length=20,
+        unique=True,
+        editable=False,
+        validators=[SLUG_VALIDATOR],
+        help_text=_(
+            "Stable identifier used in the OpenSearch index name. "
+            "Lowercase alphanumeric only. Set on creation, immutable thereafter."
+        ),
+    )
     token = models.CharField(max_length=TOKEN_LENGTH)
     created_at = models.DateTimeField(auto_now_add=True)
     is_active = models.BooleanField(default=True)
     client_id = models.CharField(blank=True, null=True)
     services = models.ManyToManyField(
-        "self",
-        verbose_name=_("Allowed services for search"),
-        blank=True,
+        "self", blank=True, verbose_name=_("Allowed services for search")
     )
 
     class Meta:
@@ -41,14 +56,35 @@ class Meta:
                 condition=models.Q(token__length=TOKEN_LENGTH),
                 name="token_length_exact_50",
             ),
+            models.CheckConstraint(
+                condition=models.Q(slug__regex=SLUG_REGEX),
+                name="slug_alphanumeric_only",
+            ),
         ]
 
     def __str__(self):
         return self.name
 
     def save(self, *args, **kwargs):
-        """Automatically slugify the service name and generate a token on creation"""
-        self.name = slugify(self.name)
+        """Generate token, auto-derive slug on creation, enforce slug immutability.
+
+        - ``slug`` is auto-derived from ``name`` if not provided (strip
+          non-alphanumeric, lowercase). It is immutable after creation.
+        - ``name`` is a free-form display field and can be edited.
+        - ``token`` is generated once on creation if missing.
+        """
+        if not self.slug:
+            self.slug = re.sub(r"[^a-zA-Z0-9]", "", self.name or "").lower()
+        if self.pk is not None:
+            stored_slug = (
+                Service.objects.filter(pk=self.pk)
+                .values_list("slug", flat=True)
+                .first()
+            )
+            if self.slug != stored_slug:
+                raise ValidationError(
+                    {"slug": _("Service.slug is immutable after creation.")}
+                )
         if not self.token:
             self.token = self.generate_secure_token()
         super().save(*args, **kwargs)