Discussion: defensive layer between mediatech's embedding pipeline and Qdrant

[Adelagric]

Hi @FaheemBEG,

I've been looking at mediatech as part of an effort to understand the embedding-ingestion patterns in the French public-sector RAG stack. The project's scope — vectorizing large administrative corpora (LEGI, JADE, CASS/INCA/CAPP, ANSSI guides, parliamentary debates, etc.) into Qdrant via fastembed and Airflow — makes it particularly exposed to a class of silent failures that are well-known in production RAG but rarely surfaced as issues until they cost money.

I'd like to open a discussion (not a PR yet) about whether a defensive layer between generate_embeddings_with_retry() and qdrant_client.upsert() could be useful for mediatech specifically, and what it would take to be worth integrating.

The failure modes I'm thinking about

Three patterns that the current database_manage.py does not catch:

1. Dimension drift across dataset additions. A new dataset gets added with a model variant that returns a different embedding dimension than the existing Qdrant collection. qdrant_client.upsert rejects loudly if the collection exists with a fixed size, but a recreate_collection path can silently move the collection to the new dim — and any downstream consumer relying on the old dim breaks at query time, not at ingestion.

2. NaN / Inf propagation. Provider batch endpoints occasionally return non-finite values on rare inputs (long sequences, encoding edge cases). These get upserted into the ANN index and contaminate it permanently. Detecting this post-hoc requires sampling the index and re-embedding — very expensive on a corpus the size of LEGI.

3. Normalization inconsistency. If ingestion-time embeddings are L2-normalized but search-time embeddings are not (or vice-versa), cosine similarity returns ranked results but the absolute scores are wrong. No error is raised. With Distance.COSINE configured at the collection level (as in create_qdrant_index), this can pass unnoticed for months.

None of these are mediatech bugs — they're failure modes inherent to writing vectors into a vector DB without a validation point.

What I'm proposing to discuss

I'm working on vector-router (Apache 2.0, Rust gRPC middleware) which is designed exactly for this: it sits between embedding producers and the vector DB, validates dim against a per-model registry, rejects non-finite values, applies a uniform L2 normalization policy, and exposes Prometheus metrics tagged by producer_id.

The public repo is the full Rust source — client/, server/, registry.rs, service.rs, math.rs, pool.rs, the gRPC contract under proto/, benchmarks, tests, Dockerfile. CI green on every commit. Apache 2.0, explicitly scoped: no inference, no models, no LLM logic — just the validation layer between embedding producers and the vector store. That said, this is genuinely a discussion (not "please merge my PR") because the right integration point on the mediatech side is yours to define.

In the mediatech case, the integration would look like:

   Airflow DAG
        │
        ▼
  generate_embeddings_with_retry()
        │
        ▼
  vector-router (gRPC, sidecar)  ── validates dim / NaN / norm
        │
        ▼
  qdrant_client.upsert()           ── unchanged from your side

The producer_id would be set per-dataset (e.g. mediatech-legi, mediatech-cass), which gives you per-dataset error visibility in Grafana.

Specific questions for you

1. Do these failure modes match anything you've actually observed running mediatech in production, or do they sound theoretical?
2. Would mediatech tolerate adding a gRPC dependency in the Airflow ingestion path, or is there a strong preference for keeping the pipeline pure-Python with no extra hop?
3. Is there appetite for a defensive layer at all, or is the current "trust fastembed + Qdrant" stance considered sufficient for your scale?

If the appetite is zero — totally fine to say so, no offense taken. If it's "maybe, show us numbers," I can run a benchmark on a representative mediatech dataset (e.g. a LEGI slice) and report back here with concrete latency overhead + error-catch rate.

Thanks for your time, and great work on the corpus pipeline — the vectorized administrative datasets you're publishing on HuggingFace are a genuinely useful piece of public infrastructure.

— Adel (@Adelagric)