From c9a91023fa2cb68d6a6dbdb3ffa96fb089380670 Mon Sep 17 00:00:00 2001 From: Kush <85061303+kush992@users.noreply.github.com> Date: Tue, 28 Jul 2026 12:35:04 +0200 Subject: [PATCH 1/2] docs: added opensearch-jvector plugin docs --------- Signed-off-by: Kush Bhalodi Co-authored-by: Kush Bhalodi --- .../additional-plugins/index.md | 1 + .../additional-plugins/opensearch-jvector.md | 68 +++++ .../knn-methods-engines.md | 87 ++++-- .../efficient-knn-filtering.md | 270 ++++++++++++++---- _vector-search/filter-search-knn/index.md | 3 +- _vector-search/getting-started/concepts.md | 4 +- _vector-search/getting-started/index.md | 2 + _vector-search/optimizing-storage/index.md | 13 +- .../approximate-knn.md | 2 + .../vector-search-techniques/index.md | 2 + 10 files changed, 371 insertions(+), 81 deletions(-) create mode 100644 _install-and-configure/additional-plugins/opensearch-jvector.md diff --git a/_install-and-configure/additional-plugins/index.md b/_install-and-configure/additional-plugins/index.md index 67e7ddf1302..e33a9915cdd 100644 --- a/_install-and-configure/additional-plugins/index.md +++ b/_install-and-configure/additional-plugins/index.md @@ -38,6 +38,7 @@ There are many more plugins available in addition to those provided by the stand | `store-smb` | 1.0.0 | | `transport-grpc` | 3.0.0 | | `workload-management` | 2.18.0 | +| [`opensearch-jvector`]({{site.url}}{{site.baseurl}}/install-and-configure/additional-plugins/opensearch-jvector/) | 3.5.0 | ## Related documentation diff --git a/_install-and-configure/additional-plugins/opensearch-jvector.md b/_install-and-configure/additional-plugins/opensearch-jvector.md new file mode 100644 index 00000000000..fb5faaadf51 --- /dev/null +++ b/_install-and-configure/additional-plugins/opensearch-jvector.md @@ -0,0 +1,68 @@ +--- +layout: default +title: Opensearch-jvector plugin +parent: Installing plugins +nav_order: 30 + +--- + +# Opensearch-jvector plugin + +The `opensearch-jvector` plugin enables running the nearest neighbor search on billions of documents with upto 16,000 dimensions with the same ease as running any regular OpenSearch query. Aggregations and filter clauses can be applied to further refine similarity search operations. + +## Differences between opensearch-jvector and k-NN + +The following table highlights the differences that matter most when choosing between the built-in `k-NN` plugin and `opensearch-jvector`: + +| Aspect | `k-NN` | `opensearch-jvector` | +| :----------------------- | :--------------------------------------- | :------------------------------------------------------------------------ | +| **Vector engines** | `Nmslib`, `Faiss`, `lucene` | `jvector` (primary), `lucene` | +| **Concurrent ingestion** | Supported by some vector engines | `jvector` supports concurrent inserts, enabling high-throughput ingestion | +| **Index update cost** | Usually a full rebuild required on merge | Incremental merges — no full rebuilds for updates | +| **Memory efficiency** | In-memory indexing | DiskANN-style quantization | + +## Use cases of plugin + +`opensearch-jvector` is a strong choice when any of the following apply to your workload: + +- **Your data grows continuously.** Incremental index updates and concurrent ingestion avoid the costly full rebuilds that occur with k-NN on high-update workloads. +- **Your dataset is larger than available memory.** DiskANN-style indexing with quantization-based reranking keeps recall high while memory usage stays bounded. + +Typical use cases include recommendation systems, image and video similarity search, semantic document search, and fraud detection pipelines that index millions to billions of vectors. + +## Unique features + +- **DiskANN Implementation (Pure Java)** - Based on `jvector` library, a pure Java implementation of DiskANN-style approximate nearest neighbour (ANN) search optimized for memory-constrained environments. It eliminates the need for native libraries such as `Faiss` and avoids the complexity and overhead associated with JNI, simplifying deployment and maintenance. +- **Scalable Thread-Safe Design** - The index is fully thread-safe and supports concurrent updates and insertions with near-linear scalability as CPU cores increase. The underlying Jvector library enables high-throughput ingestion without relying on costly merge operations to achieve parallelism. +- **Quantization Refinement During Merges** - The system refines quantization codebooks incrementally during merge operations. This approach improves search accuracy and recall without requiring a complete recomputation of codebooks, reducing computational overhead. +- **Incremental Index Updates** - `jvector` allows incremental insertion of vectors into existing indexes. This providing efficiency gains over full index rebuilds for workloads involving frequent updates, particularly for large graph-based indexes. +- **Quantized DiskANN with Reranking** - `jvector` supports DiskANN-style quantization combined with reranking, delivering significant performance improvements for datasets larger than available memory. This approach is particularly effective for large-scale deployments where traditional in-memory indexing is not feasible. +- **Product Quantization (PQ)** - PQ is implemented with high-performance SIMD optimizations and separate codebooks to enable fast vector searches with low memory usage. +- **Advanced Quantization Techniques** - `jvector` includes advanced capabilities such as Non-Vector Quantization (NVQ), and Anisotropic PQ, enabling more efficient and accurate similarity computations beyond standard quantization approaches. + +## Installation + +**1. Remove Existing k-NN and neural-search Plugins** (make sure no `knn` enabled indexes are created) + +```bash +bin/opensearch-plugin remove opensearch-neural-search +bin/opensearch-plugin remove opensearch-knn +``` + +**2. Install `opensearch-jvector` Plugin** + +```bash +bin/opensearch-plugin install opensearch-jvector +``` + +## OpenSearch compatible features + +- Product Quantization (since: 3.5.0) +- MMR Search (since: 3.6.0) +- Derived Sources (since: 3.6.0) + +## Limitations + +- `opensearch-jvector` plugin is not part of the default OpenSearch distribution +- `opensearch-knn` and `opensearch-jvector` cannot be installed simultaneously +- `opensearch-jvector` is not recognized by the upstream version of `neural-search` plugin. \ No newline at end of file diff --git a/_mappings/supported-field-types/knn-methods-engines.md b/_mappings/supported-field-types/knn-methods-engines.md index 1027e3744a0..123a8a3a358 100644 --- a/_mappings/supported-field-types/knn-methods-engines.md +++ b/_mappings/supported-field-types/knn-methods-engines.md @@ -23,6 +23,7 @@ OpenSearch supports the following engines: - [**Lucene**](#lucene-engine): The native search library, offering an HNSW implementation with efficient filtering capabilities - [**Faiss**](#faiss-engine) (Facebook AI Similarity Search): A comprehensive library implementing both the HNSW and IVF methods, with additional vector compression options - [**NMSLIB**](#nmslib-engine-deprecated) (Non-Metric Space Library): A legacy implementation of HNSW (now deprecated) +- [**jvector**]({{site.url}}{{site.baseurl}}/install-and-configure/additional-plugins/opensearch-jvector/): Available through the `opensearch-jvector` plugin (not included in the default distribution). Implements DiskANN-style approximate nearest neighbor search using the `disk_ann` method. Supports concurrent inserts, incremental merges, and native Product Quantization (PQ). Cannot be installed alongside `opensearch-knn`. ## Method definition example @@ -76,7 +77,7 @@ Mapping parameter | Required | Default | Updatable | Description :--- | :--- | :--- | :--- | :--- `name` | Yes | N/A | No | The nearest neighbor method. Valid values are `hnsw`, `ivf`, and `flat`. Not every engine combination supports each of the methods. For a list of supported methods, see the section for a specific engine. `space_type` | No | `l2` | No | The vector space used to calculate the distance between vectors. Valid values are `l1`, `l2`, `linf`, `cosinesimil`, `innerproduct`, `hamming`, and `hammingbit`. Not every method/engine combination supports each of the spaces. For a list of supported spaces, see the section for a specific engine. Note: This value can also be specified at the top level of the mapping. For more information, see [Spaces]({{site.url}}{{site.baseurl}}/mappings/supported-field-types/knn-spaces/). -`engine` | No | `faiss` | No | The approximate k-NN library to use for indexing and search. Valid values are `faiss`, `lucene`, and `nmslib` (deprecated). +`engine` | No | `faiss` | No | The approximate k-NN library to use for indexing and search. Valid values are `faiss`, `lucene`, `nmslib` (deprecated), and `jvector` (requires the [`opensearch-jvector` plugin]({{site.url}}{{site.baseurl}}/install-and-configure/additional-plugins/opensearch-jvector/)). `parameters` | No | `null` | No | The parameters used for the nearest neighbor method. For more information, see the section for a specific engine. ## Lucene engine @@ -396,7 +397,58 @@ An index created in OpenSearch version 2.11 or earlier will still use the previo } ``` -## Choosing the right method +## `jvector` engine + +The `jvector` engine is available through the [`opensearch-jvector` plugin]({{site.url}}{{site.baseurl}}/install-and-configure/additional-plugins/opensearch-jvector/) and is not included in the default OpenSearch distribution. It cannot be installed alongside `opensearch-knn`. The `jvector` engine implements DiskANN-style approximate nearest neighbor search in pure Java, with no JNI dependency. + +### Supported methods + +The `jvector` engine supports the following method. + +Method name | Requires training | Supported spaces +:--- | :--- | :--- +`disk_ann` | No | `l2`, `cosinesimil`, `innerproduct` + +### Method parameters + +| Parameter | Default | Description | +|---|---|---| +| `m` | `16` | Bi-directional link count per node. Higher values improve recall, increase index size. | +| `ef_construction` | `100` | Candidate list size during graph construction. Higher values improve recall, slow ingestion. | +| `advanced.alpha` | `1.2` | Diversity factor for neighbor selection | +| `advanced.neighbor_overflow` | `1.2` | Overflow factor for neighbor lists | +| `advanced.hierarchy_enabled` | `false` | Enable hierarchical graph structure | +| `advanced.num_pq_subspaces` | — | Number of PQ subspaces. Must be ≤ dimension. | +| `advanced.min_batch_size_for_quantization` | `1024` | Documents needed before quantization is trained | +| `advanced.leading_segment_merge_disabled` | `false` | Prevent leading segment from being rebuilt during force merge | + +### Space Types + +The `space_type` field controls which distance metric is used. + +| Value | Distance metric | Use case | +|---|---|---| +| `l2` | Euclidean distance (L2 norm) | General-purpose; raw coordinates | +| `cosinesimil` | Cosine similarity | Text embeddings; direction matters more than magnitude | +| `innerproduct` | Dot product (inner product) | Embeddings where magnitude carries meaning (e.g., biencoder models) | + +The default is `l2` when `space_type` is omitted. + +### Example configuration + +```json +"method": { + "name": "disk_ann", + "engine": "jvector", + "space_type": "l2", + "parameters": { + "m": 16, + "ef_construction": 100 + } +} +``` + +## Choosing the right method and engine There are several options to choose from when building your `knn_vector` field. To select the correct method and parameters, you should first understand the requirements of your workload and what trade-offs you are willing to make. Factors to consider are (1) query latency, (2) query quality, (3) memory limits, and (4) indexing latency. @@ -406,23 +458,26 @@ If you want to use less memory and increase indexing speed as compared to HNSW w If memory is a concern, consider adding a PQ encoder to your HNSW or IVF index. Because PQ is a lossy encoding, query quality will drop. -You can reduce the memory footprint by a factor of 2, with a minimal loss in search quality, by using the [`fp_16` encoder]({{site.url}}{{site.baseurl}}/vector-search/optimizing-storage/faiss-scalar-quantization/#16-bit-quantization). If your vector dimensions are within the [-128, 127] byte range, we recommend using the [byte quantizer]({{site.url}}{{site.baseurl}}/mappings/supported-field-types/knn-memory-optimized/#byte-vectors) to reduce the memory footprint by a factor of 4. To learn more about vector quantization options, see [k-NN vector quantization]({{site.url}}{{site.baseurl}}/vector-search/optimizing-storage/knn-vector-quantization/). +If your dataset grows continuously or exceeds available memory, consider the [jvector engine](#jvector-engine) (via the [`opensearch-jvector` plugin]({{site.url}}{{site.baseurl}}/install-and-configure/additional-plugins/opensearch-jvector/)). It supports concurrent inserts, incremental merges without full graph rebuilds, and native PQ with better recall at higher compression ratios. + +You can reduce the memory footprint by a factor of 2, with a minimal loss in search quality, by using the [`fp_16` encoder]({{site.url}}{{site.baseurl}}/vector-search/optimizing-storage/faiss-scalar-quantization/#16-bit-quantization). If your vector dimensions are within the [-128, 127] byte range, we recommend using the [byte quantizer]({{site.url}}{{site.baseurl}}/mappings/supported-field-types/knn-memory-optimized/#byte-vectors) to reduce the memory footprint by a factor of 4. To learn more about vector quantization options, see [k-NN vector quantization]({{site.url}}{{site.baseurl}}/vector-search/optimizing-storage/knn-vector-quantization/). ## Engine recommendations -In general, select Faiss for large-scale use cases. Lucene is a good option for smaller deployments and offers benefits like smart filtering, where the optimal filtering strategy—pre-filtering, post-filtering, or exact k-NN—is automatically applied depending on the situation. The following table summarizes the differences between each option. - -| | Faiss/HNSW | Faiss/IVF | Lucene/HNSW | -|:---|:---|:---|:---| -| Max dimensions | 16,000 | 16,000 | 16,000 | -| Filter | Post-filter | Post-filter | Filter during search | -| Training required | No (Yes for PQ) | Yes | No | -| Similarity metrics | `l2`, `innerproduct`, `cosinesimil` | `l2`, `innerproduct`, `cosinesimil` | `l2`, `cosinesimil` | -| Number of vectors | Tens of billions | Tens of billions | Less than 10 million | -| Indexing latency | Low | Lowest | Low | -| Query latency and quality | Low latency and high quality | Low latency and low quality | High latency and high quality | -| Vector compression | Flat

PQ | Flat

PQ | Flat | -| Memory consumption | High

Low with PQ | Medium

Low with PQ | High | +In general, select Faiss for large-scale use cases. Lucene is a good option for smaller deployments and offers benefits like smart filtering, where the optimal filtering strategy—pre-filtering, post-filtering, or exact k-NN—is automatically applied depending on the situation. Consider `jvector` for high-update workloads or memory-constrained large-scale deployments. The following table summarizes the differences between each option. + +| | Faiss/HNSW | Faiss/IVF | Lucene/HNSW | jvector/disk_ann | +|:---|:---|:---|:---|:---| +| Max dimensions | 16,000 | 16,000 | 16,000 | 16,000 | +| Filter | Post-filter | Post-filter | Filter during search | Filter during search

Post-filter | +| Training required | No (Yes for PQ) | Yes | No | No | +| Similarity metrics | `l2`, `innerproduct`, `cosinesimil` | `l2`, `innerproduct`, `cosinesimil` | `l2`, `cosinesimil` | `l2`, `cosinesimil`, `innerproduct`, `l1`, `linf` | +| Number of vectors | Tens of billions | Tens of billions | Less than 10 million | Billions | +| Indexing latency | Low | Lowest | Low | Low (concurrent inserts) | +| Query latency and quality | Low latency and high quality | Low latency and low quality | High latency and high quality | Low latency and high quality | +| Vector compression | Flat

PQ | Flat

PQ | Flat | Flat

PQ (native, no training) | +| Memory consumption | High

Low with PQ | Medium

Low with PQ | High | Low (DiskANN)

Very low with PQ | +| Plugin required | No | No | No | Yes ([opensearch-jvector]({{site.url}}{{site.baseurl}}/install-and-configure/additional-plugins/opensearch-jvector/)) | ## Memory estimation diff --git a/_vector-search/filter-search-knn/efficient-knn-filtering.md b/_vector-search/filter-search-knn/efficient-knn-filtering.md index a16a1a29035..d9c8405b5c1 100644 --- a/_vector-search/filter-search-knn/efficient-knn-filtering.md +++ b/_vector-search/filter-search-knn/efficient-knn-filtering.md @@ -7,7 +7,7 @@ nav_order: 10 # Efficient k-NN filtering -You can perform efficient k-NN filtering with the `lucene` or `faiss` engines. +You can perform efficient k-NN filtering with the `lucene`, `faiss` or `jvector` engines. ## Lucene k-NN filter implementation @@ -69,13 +69,14 @@ PUT /hotels-index } } ``` + {% include copy-curl.html %} ### Step 2: Add data to your index Next, add data to your index. -The following request adds 12 documents that contain hotel location, rating, and parking information: +The following request adds 12 documents that contain hotel location, rating, and parking information: ```json POST /_bulk @@ -104,6 +105,7 @@ POST /_bulk { "index": { "_index": "hotels-index", "_id": "12" } } { "location": [5.0, 1.0], "parking" : "true", "rating" : 3 } ``` + {% include copy-curl.html %} ### Step 3: Search your data with a filter @@ -148,64 +150,56 @@ POST /hotels-index/_search } } ``` + {% include copy-curl.html %} The response returns the three hotels that are nearest to the search point and have met the filter criteria: ```json { - "took" : 47, - "timed_out" : false, - "_shards" : { - "total" : 1, - "successful" : 1, - "skipped" : 0, - "failed" : 0 + "took": 47, + "timed_out": false, + "_shards": { + "total": 1, + "successful": 1, + "skipped": 0, + "failed": 0 }, - "hits" : { - "total" : { - "value" : 3, - "relation" : "eq" + "hits": { + "total": { + "value": 3, + "relation": "eq" }, - "max_score" : 0.72992706, - "hits" : [ + "max_score": 0.72992706, + "hits": [ { - "_index" : "hotels-index", - "_id" : "3", - "_score" : 0.72992706, - "_source" : { - "location" : [ - 4.9, - 3.4 - ], - "parking" : "true", - "rating" : 9 + "_index": "hotels-index", + "_id": "3", + "_score": 0.72992706, + "_source": { + "location": [4.9, 3.4], + "parking": "true", + "rating": 9 } }, { - "_index" : "hotels-index", - "_id" : "6", - "_score" : 0.3012048, - "_source" : { - "location" : [ - 6.4, - 3.4 - ], - "parking" : "true", - "rating" : 9 + "_index": "hotels-index", + "_id": "6", + "_score": 0.3012048, + "_source": { + "location": [6.4, 3.4], + "parking": "true", + "rating": 9 } }, { - "_index" : "hotels-index", - "_id" : "5", - "_score" : 0.24154587, - "_source" : { - "location" : [ - 3.3, - 4.5 - ], - "parking" : "true", - "rating" : 8 + "_index": "hotels-index", + "_id": "5", + "_score": 0.24154587, + "_source": { + "location": [3.3, 4.5], + "parking": "true", + "rating": 8 } } ] @@ -215,7 +209,7 @@ The response returns the three hotels that are nearest to the search point and h For more ways to construct a filter, see [Constructing a filter](#constructing-a-filter). -## Faiss k-NN filter implementation +## Faiss k-NN filter implementation For k-NN searches, you can use `faiss` filters with an HNSW algorithm (OpenSearch version 2.9 and later) or IVF algorithm (OpenSearch version 2.10 and later). @@ -233,13 +227,14 @@ The following flow chart outlines the Faiss algorithm. ![Faiss algorithm for filtering]({{site.url}}{{site.baseurl}}/images/faiss-algorithm.jpg) ### Disabling the exact search fallback + **Introduced 3.5** {: .label .label-purple } When a Faiss efficient-filtered ANN search returns fewer than `k` results (R < k) even though more than `k` documents match the filter (P ≥ k), the algorithm falls back to an exact search over the filtered document IDs to ensure that `k` results are returned. For latency-sensitive workloads in which fewer than `k` results are acceptable, you can disable this fallback by setting the `index.knn.faiss.efficient_filter.disable_exact_search` index setting to `true`. When this setting is enabled, the search returns only approximate results and skips the additional exact search. - For more information about this setting, see [Vector search settings]({{site.url}}{{site.baseurl}}/vector-search/settings/). +For more information about this setting, see [Vector search settings]({{site.url}}{{site.baseurl}}/vector-search/settings/). ## Using a Faiss efficient filter @@ -276,13 +271,14 @@ PUT /products-shirts } } ``` + {% include copy-curl.html %} ### Step 2: Add data to your index Next, add data to your index. -The following request adds 12 documents that contain information about shirts, including their vector representation, size, and rating: +The following request adds 12 documents that contain information about shirts, including their vector representation, size, and rating: ```json POST /_bulk?refresh @@ -312,6 +308,7 @@ POST /_bulk?refresh { "item_vector": [5.0, 1.0, 4.0], "size" : "large", "rating" : 3 } ``` + {% include copy-curl.html %} ### Step 3: Search your data with a filter @@ -355,6 +352,7 @@ POST /products-shirts/_search } } ``` + {% include copy-curl.html %} The response returns the two matching documents: @@ -381,11 +379,7 @@ The response returns the two matching documents: "_id": "8", "_score": 0.8620689, "_source": { - "item_vector": [ - 2.4, - 4, - 3 - ], + "item_vector": [2.4, 4, 3], "size": "small", "rating": 8 } @@ -395,11 +389,7 @@ The response returns the two matching documents: "_id": "6", "_score": 0.029691212, "_source": { - "item_vector": [ - 6.4, - 3.4, - 6.6 - ], + "item_vector": [6.4, 3.4, 6.6], "size": "small", "rating": 9 } @@ -412,14 +402,173 @@ The response returns the two matching documents: For more ways to construct a filter, see [Constructing a filter](#constructing-a-filter). ### ACORN filtering optimization + Introduced 3.1 {: .label .label-purple } The ACORN filtering optimization modifies the baseline algorithm to score and explore only vectors that match the filtering criteria. When filtering increases graph sparsity, the search expands to include neighbors of neighbors. The extent of this additional exploration depends on the percentage of neighbors filtered out, with more restrictive filters resulting in a wider search. -The algorithm bypasses these optimizations entirely when filtering is minimal. By default, this threshold is 60%. Extended neighbor exploration occurs only if fewer than 90% of the current neighbors match the filter. +The algorithm bypasses these optimizations entirely when filtering is minimal. By default, this threshold is 60%. Extended neighbor exploration occurs only if fewer than 90% of the current neighbors match the filter. When [memory-optimized search]({{site.url}}{{site.baseurl}}/vector-search/optimizing-storage/memory-optimized-search/) is enabled, the efficient filter framework continues to apply filtering within HNSW. The ACORN filtering optimization is applied only when the number of filtered documents is 60% or fewer of the total number of documents in the current search space being considered by the HNSW algorithm. +## Using a JVector efficient filter + +The [`opensearch-jvector`]({{site.url}}{{site.baseurl}}/install-and-configure/additional-plugins/opensearch-jvector/) plugin supports filtering using the `jvector` engine with the `disk_ann` method. An inline `filter` inside the `knn` query clause restricts candidates during graph traversal. Post-filtering via `post_filter` is also supported but may return fewer than `k` results when the filter is restrictive. + +In this example, you will create an index using the `jvector` engine and search for the three closest hotels that have high ratings and provide parking. + +### Step 1: Create a new index + +Create an index with a `knn_vector` field, specifying `jvector` as the engine and `disk_ann` as the method: + +```json +PUT /hotels-index +{ + "settings": { + "index": { + "knn": true, + "number_of_shards": 1, + "number_of_replicas": 0 + } + }, + "mappings": { + "properties": { + "location": { + "type": "knn_vector", + "dimension": 2, + "method": { + "name": "disk_ann", + "space_type": "l2", + "engine": "jvector", + "parameters": { + "ef_construction": 100, + "m": 16 + } + } + } + } + } +} +``` + +{% include copy-curl.html %} + +### Step 2: Add data to your index + +Add 12 documents containing hotel location, rating, and parking information: + +```json +POST /_bulk +{ "index": { "_index": "hotels-index", "_id": "1" } } +{ "location": [5.2, 4.4], "parking" : "true", "rating" : 5 } +{ "index": { "_index": "hotels-index", "_id": "2" } } +{ "location": [5.2, 3.9], "parking" : "false", "rating" : 4 } +{ "index": { "_index": "hotels-index", "_id": "3" } } +{ "location": [4.9, 3.4], "parking" : "true", "rating" : 9 } +{ "index": { "_index": "hotels-index", "_id": "4" } } +{ "location": [4.2, 4.6], "parking" : "false", "rating" : 6} +{ "index": { "_index": "hotels-index", "_id": "5" } } +{ "location": [3.3, 4.5], "parking" : "true", "rating" : 8 } +{ "index": { "_index": "hotels-index", "_id": "6" } } +{ "location": [6.4, 3.4], "parking" : "true", "rating" : 9 } +{ "index": { "_index": "hotels-index", "_id": "7" } } +{ "location": [4.2, 6.2], "parking" : "true", "rating" : 5 } +{ "index": { "_index": "hotels-index", "_id": "8" } } +{ "location": [2.4, 4.0], "parking" : "true", "rating" : 8 } +{ "index": { "_index": "hotels-index", "_id": "9" } } +{ "location": [1.4, 3.2], "parking" : "false", "rating" : 5 } +{ "index": { "_index": "hotels-index", "_id": "10" } } +{ "location": [7.0, 9.9], "parking" : "true", "rating" : 9 } +{ "index": { "_index": "hotels-index", "_id": "11" } } +{ "location": [3.0, 2.3], "parking" : "false", "rating" : 6 } +{ "index": { "_index": "hotels-index", "_id": "12" } } +{ "location": [5.0, 1.0], "parking" : "true", "rating" : 3 } +``` + +{% include copy-curl.html %} + +### Step 3: Search your data with a filter + +Place the `filter` inside the `knn` query clause to restrict candidates during graph traversal. The following request searches for the top three hotels near `[5, 4]` that are rated between 8 and 10 and provide parking: + +```json +POST /hotels-index/_search +{ + "size": 3, + "query": { + "knn": { + "location": { + "vector": [5, 4], + "k": 3, + "filter": { + "bool": { + "must": [ + { + "range": { + "rating": { + "gte": 8, + "lte": 10 + } + } + }, + { + "term": { + "parking": "true" + } + } + ] + } + } + } + } + } +} +``` + +{% include copy-curl.html %} + +### Step 4: Search your data with a post_filter + +You can also apply filtering after the ANN search using `post_filter`. Because `post_filter` runs after ANN retrieval, the final result count may be fewer than `k` when the filter is restrictive. To compensate, set `k` to a value larger than the number of results you need. + +The following request retrieves the 20 nearest hotels and then restricts the results to those rated 8 or above with parking: + +```json +POST /hotels-index/_search +{ + "size": 3, + "query": { + "knn": { + "location": { + "vector": [5, 4], + "k": 20 + } + } + }, + "post_filter": { + "bool": { + "must": [ + { + "range": { + "rating": { + "gte": 8, + "lte": 10 + } + } + }, + { + "term": { + "parking": "true" + } + } + ] + } + } +} +``` + +{% include copy-curl.html %} + ## Constructing a filter There are multiple ways to construct a filter for the same condition. For example, you can use the following constructs to create a filter that returns hotels that provide parking: @@ -482,6 +631,7 @@ POST /hotels-index/_search } } } -} +} ``` + {% include copy-curl.html %} diff --git a/_vector-search/filter-search-knn/index.md b/_vector-search/filter-search-knn/index.md index c230d8566c8..e08e6654344 100644 --- a/_vector-search/filter-search-knn/index.md +++ b/_vector-search/filter-search-knn/index.md @@ -13,8 +13,9 @@ redirect_from: To refine vector search results, you can filter a vector search using one of the following methods: - [Efficient k-nearest neighbors (k-NN) filtering]({{site.url}}{{site.baseurl}}/vector-search/filter-search-knn/efficient-knn-filtering/): This approach applies filtering _during_ the vector search, as opposed to before or after the vector search, which ensures that `k` results are returned (if there are at least `k` results in total). This approach is supported by the following engines: - - Lucene engine with a Hierarchical Navigable Small World (HNSW) algorithm (OpenSearch version 2.4 and later) + - Lucene engine with a Hierarchical Navigable Small World (HNSW) algorithm (OpenSearch version 2.4 and later) - Faiss engine with an HNSW algorithm (OpenSearch version 2.9 and later) or IVF algorithm (OpenSearch version 2.10 and later). In OpenSearch version 3.1 and later, when using the Faiss engine and HNSW, the [Lucene ACORN filtering optimization](https://github.com/apache/lucene/pull/14160) is applied during HNSW traversal when [memory-optimized search]({{site.url}}{{site.baseurl}}/vector-search/optimizing-storage/memory-optimized-search/) is enabled. + - `jvector` engine (via the [`opensearch-jvector`]({{site.url}}{{site.baseurl}}/install-and-configure/additional-plugins/opensearch-jvector/) plugin): Supports inline filters inside the `knn` query clause using the same syntax as the Lucene and Faiss efficient filters. - [Post-filtering]({{site.url}}{{site.baseurl}}/vector-search/filter-search-knn/post-filtering/): Because it is performed after the vector search, this approach may return significantly fewer than `k` results for a restrictive filter. You can use the following two filtering strategies for this approach: - [Boolean post-filter]({{site.url}}{{site.baseurl}}/vector-search/filter-search-knn/post-filtering/#boolean-filter-with-ann-search): This approach runs an [approximate nearest neighbor (ANN)]({{site.url}}{{site.baseurl}}/search-plugins/knn/approximate-knn/) search and then applies a filter to the results. The two query parts are executed independently, and then the results are combined based on the query operator (`should`, `must`, and so on) provided in the query. diff --git a/_vector-search/getting-started/concepts.md b/_vector-search/getting-started/concepts.md index 9df4476a068..bc27a18c4af 100644 --- a/_vector-search/getting-started/concepts.md +++ b/_vector-search/getting-started/concepts.md @@ -23,9 +23,9 @@ This page defines key terms and techniques related to vector search in OpenSearc - A [**_space_**]({{site.url}}{{site.baseurl}}/vector-search/getting-started/vector-search-basics/#calculating-similarity) defines how similarity or distance between two vectors is measured. Different spaces use different distance metrics, such as Euclidean distance or cosine similarity, to determine how closely vectors resemble each other. -- A [**_method_**]({{site.url}}{{site.baseurl}}/mappings/supported-field-types/knn-methods-engines/) refers to the algorithm used to organize vector data during indexing and retrieve relevant results during search in approximate k-NN search. Different methods balance trade-offs between accuracy, speed, and memory usage. +- A [**_method_**]({{site.url}}{{site.baseurl}}/mappings/supported-field-types/knn-methods-engines/) refers to the algorithm used to organize vector data during indexing and retrieve relevant results during search in approximate k-NN search. Different methods balance trade-offs between accuracy, speed, and memory usage. -- An [**_engine_**]({{site.url}}{{site.baseurl}}/mappings/supported-field-types/knn-methods-engines/) is the underlying library that implements vector search methods. It determines how vectors are indexed, stored, and retrieved during similarity search operations. +- An [**_engine_**]({{site.url}}{{site.baseurl}}/mappings/supported-field-types/knn-methods-engines/) is the underlying library that implements vector search methods. It determines how vectors are indexed, stored, and retrieved during similarity search operations. OpenSearch supports multiple engines through the built-in `k-NN` plugin. Alternatively, the [`opensearch-jvector`]({{site.url}}{{site.baseurl}}/install-and-configure/additional-plugins/opensearch-jvector/) plugin provides an additional `jvector` engine, which implements `DiskANN` method. ## k-NN search diff --git a/_vector-search/getting-started/index.md b/_vector-search/getting-started/index.md index 4b1fde17e8c..36b76454d7f 100644 --- a/_vector-search/getting-started/index.md +++ b/_vector-search/getting-started/index.md @@ -95,6 +95,8 @@ PUT /hotels-index ``` {% include copy-curl.html %} +To use a different engine, see [Methods and engines]({{site.url}}{{site.baseurl}}/mappings/supported-field-types/knn-methods-engines/). + Vector queries usually have a `size` > 0, so by default they don't enter the request cache. In OpenSearch 2.19 or later, if your workload mostly consists of vector queries, consider increasing the dynamic `indices.requests.cache.maximum_cacheable_size` cluster setting to a larger value, such as `256`. This allows queries with a `size` of up to 256 to enter the request cache, improving performance. For more information, see [Request cache]({{site.url}}{{site.baseurl}}/search-plugins/caching/request-cache/). ## Step 2: Add data to your index diff --git a/_vector-search/optimizing-storage/index.md b/_vector-search/optimizing-storage/index.md index 4b04024e710..f3e5055e644 100644 --- a/_vector-search/optimizing-storage/index.md +++ b/_vector-search/optimizing-storage/index.md @@ -17,6 +17,15 @@ storage_cards: # Optimizing vector storage -Vector search operations can be resource intensive, especially when dealing with large-scale vector datasets. OpenSearch provides several optimization techniques for reducing memory usage. +Vector search operations can be resource intensive, especially when dealing with large-scale vector datasets. OpenSearch provides several optimization techniques for reducing memory usage. -{% include cards.html cards=page.storage_cards %} \ No newline at end of file +{% include cards.html cards=page.storage_cards %} + +## Opensearch-jvector: Built-in disk-friendly quantization + +The `jvector` engine provided by [`opensearch-jvector`]({{site.url}}{{site.baseurl}}/install-and-configure/additional-plugins/opensearch-jvector/) plugin implements DiskANN-style indexing, which stores vectors on disk rather than in memory, and builds indexes directly from quantized vectors. This approach provides substantial memory savings without requiring separate quantization configuration steps. + +Key storage advantages of `jvector` compared to the built-in engines: + +- **Quantized index construction**: Builds indexes from quantized vectors, reducing memory required during indexing. +- **Incremental merges**: Refines quantization codebooks incrementally during merges instead of requiring full rebuilds, reducing computational overhead. diff --git a/_vector-search/vector-search-techniques/approximate-knn.md b/_vector-search/vector-search-techniques/approximate-knn.md index 6af216feaaa..00cdf1d971d 100644 --- a/_vector-search/vector-search-techniques/approximate-knn.md +++ b/_vector-search/vector-search-techniques/approximate-knn.md @@ -15,6 +15,8 @@ Standard k-nearest neighbors (k-NN) search methods compute similarity using a br The approximate k-NN search methods in OpenSearch use approximate nearest neighbor (ANN) algorithms from the [NMSLIB](https://github.com/nmslib/nmslib), [Faiss](https://github.com/facebookresearch/faiss), and [Lucene](https://lucene.apache.org/) libraries to power k-NN search. These search methods employ ANN to improve search latency for large datasets. Of the three search methods OpenSearch provides, this method offers the best search scalability for large datasets. This approach is the preferred method when a dataset reaches hundreds of thousands of vectors. +The [`opensearch-jvector`]({{site.url}}{{site.baseurl}}/install-and-configure/additional-plugins/opensearch-jvector/) plugin provides an additional `jvector` engine for approximate k-NN search, implementing DiskANN-style indexing in pure Java. Key advantages over the built-in engines include thread-safe concurrent ingestion, incremental index updates without full graph rebuilds, and native Product Quantization (PQ). Consider `jvector` when your dataset grows continuously or when you need high-recall compression at datasets larger than available memory. + For information about the algorithms OpenSearch supports, see [Methods and engines]({{site.url}}{{site.baseurl}}/mappings/supported-field-types/knn-methods-engines/). {: .note} diff --git a/_vector-search/vector-search-techniques/index.md b/_vector-search/vector-search-techniques/index.md index 10b7cd55ae5..ed407d92d61 100644 --- a/_vector-search/vector-search-techniques/index.md +++ b/_vector-search/vector-search-techniques/index.md @@ -29,6 +29,8 @@ In general, you should choose the ANN method for larger datasets because it scal OpenSearch supports multiple backend algorithms (_methods_) and libraries for implementing these algorithms (_engines_). It automatically selects the optimal configuration based on the chosen mode and available memory. For more information, see [Methods and engines]({{site.url}}{{site.baseurl}}/mappings/supported-field-types/knn-methods-engines/). +The [`opensearch-jvector`]({{site.url}}{{site.baseurl}}/install-and-configure/additional-plugins/opensearch-jvector/) plugin provides an additional `jvector` engine with the `disk_ann` method. Unlike the built-in engines, `jvector` is not part of the default OpenSearch distribution and cannot be installed alongside `opensearch-knn`. + ## Using sparse vectors _Neural sparse search_ offers an efficient alternative to dense vector search by using sparse embedding models and inverted indexes, providing performance similar to BM25. Unlike dense vector methods that require significant memory and CPU resources, sparse search creates a list of token-weight pairs and stores them in a rank features index. This approach combines the efficiency of traditional search with the semantic understanding of neural networks. OpenSearch supports both automatic embedding generation through ingest pipelines and direct sparse vector ingestion. For more information, see [Neural sparse search]({{site.url}}{{site.baseurl}}/vector-search/ai-search/neural-sparse-search/). From 9621e6659e6e9a9142ce197a7863c9b18a048124 Mon Sep 17 00:00:00 2001 From: Kush Bhalodi Date: Tue, 28 Jul 2026 17:46:29 +0200 Subject: [PATCH 2/2] chore: corrected name in opensearch-jvector docs Signed-off-by: Kush Bhalodi --- _install-and-configure/additional-plugins/opensearch-jvector.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_install-and-configure/additional-plugins/opensearch-jvector.md b/_install-and-configure/additional-plugins/opensearch-jvector.md index fb5faaadf51..27470e535d0 100644 --- a/_install-and-configure/additional-plugins/opensearch-jvector.md +++ b/_install-and-configure/additional-plugins/opensearch-jvector.md @@ -65,4 +65,4 @@ bin/opensearch-plugin install opensearch-jvector - `opensearch-jvector` plugin is not part of the default OpenSearch distribution - `opensearch-knn` and `opensearch-jvector` cannot be installed simultaneously -- `opensearch-jvector` is not recognized by the upstream version of `neural-search` plugin. \ No newline at end of file +- `opensearch-jvector` is not recognized by the upstream version of `opensearch-neural-search` plugin. \ No newline at end of file