[v1.38] HFresh GA docs#436
Open
g-despot wants to merge 1 commit into
Open
Conversation
![orca-security-eu[bot]](https://avatars.githubusercontent.com/in/276244?s=60&v=4){kind=small}
There was a problem hiding this comment.
Orca Security Scan Summary
| Status | Check | Issues by priority | |
|---|---|---|---|
 Passed |
Secrets |  0  0  0  0 |
View in Orca |
Contributor
There was a problem hiding this comment.
Pull request overview
Updates Weaviate documentation to reflect the HFresh vector index being generally available in v1.38 and positions it as a low-memory, disk-backed alternative to HNSW for very large datasets.
Changes:
- Expands HFresh explanations and adds recall-tuning guidance (
searchProbe,rescoreLimit) in starter guides and index reference docs. - Cross-links HFresh from memory/quantization/index-choice sections to help readers discover it when optimizing RAM usage.
- Updates the HFresh feature note/status include to remove “Preview” language and present it as “Added in v1.36”.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/weaviate/starter-guides/managing-resources/indexing.mdx | Expands HFresh overview and adds a tuning tip for recall. |
| docs/weaviate/starter-guides/managing-resources/compression.mdx | Adds HFresh as an index-level alternative when RAM is the main constraint. |
| docs/weaviate/config-refs/indexing/vector-index.mdx | Updates HFresh parameter table (mutability/details) and adds HFresh tuning guidance. |
| docs/weaviate/concepts/vector-quantization.md | Cross-references HFresh as a disk-backed alternative to compression for memory limits. |
| docs/weaviate/concepts/resources.md | Adds HFresh as an option for very large, memory-constrained datasets. |
| docs/weaviate/concepts/indexing/vector-index.md | Expands HFresh “how it works” + tradeoffs and updates the index comparison table. |
| docs/weaviate/concepts/filtering.md | Notes that HFresh benefits from ACORN via its centroid HNSW routing. |
| docs/weaviate/concepts/cluster.md | Mentions HFresh can reduce the need to shard purely for RAM reasons. |
| docs/weaviate/best-practices/index.md | Adds HFresh as an alternative to quantization for lowering memory footprint. |
| _includes/feature-notes/hfresh_status.mdx | Updates the feature status note to indicate availability since v1.36. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+49
to
51
| The HNSW index must be stored in memory. The memory required is directly related to the size of your dataset. There is no correlation between the size of your dataset and the current query load. You can use [`product quantization (PQ)`](/weaviate/concepts/vector-quantization#product-quantization) to compress the vectors in your dataset in increase the number of vectors your can hold in memory. For very large datasets, the disk-based [HFresh index](/weaviate/concepts/vector-index#hfresh-index) is another option. | ||
|
|
||
| Weaviate let's you configure a limit to the number of vectors held in memory in order to prevent unexpected Out-of-Memory ("OOM") situations. The default value is one trillion (`1e12`) objects. per collection. To adjust the number of objects, update the value of [`vectorCacheMaxObjects`](../config-refs/indexing/vector-index.mdx) in your index settings. |
| - BQ compressed vectors use 97% less memory than uncompressed vectors. | ||
|
|
||
| An HNSW index comprises a connection graph as well as the vectors. Quantization methods reduce the size of the vectors, but does not affect the size of the graph. As a result the overall reduction in memory usage is less than the reduction in vector size, but still significant. | ||
| An HNSW index comprises a connection graph as well as the vectors. Quantization methods reduce the size of the vectors, but does not affect the size of the graph. As a result the overall reduction in memory usage is less than the reduction in vector size, but still significant. If you need to reduce memory further for a very large dataset, the disk-based [HFresh index](/weaviate/concepts/vector-index#hfresh-index) avoids keeping the full graph in memory altogether. |
| | `replicas` | integer | `4` | No | Number of posting lists in which a vector is added. Min: `1`, Max: `10`. | | ||
| | `searchProbe` | integer | `64` | Yes | Number of posting lists to search during a query. | | ||
| | `rq` | object | -- | No | Rotational quantization (RQ) compression configuration. RQ is enabled by default and immutable. | | ||
| | `rq` | object | -- | Partial | Rotational quantization (RQ) compression configuration. RQ is mandatory for HFresh and cannot be turned off. Its `rescoreLimit` (default `350`) — the number of candidates rescored against uncompressed vectors — is mutable at runtime. | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documents the HFresh vector index for its v1.38 GA and surfaces it as a low-memory alternative to HNSW.
v1.36")maxPostingSizeKBmutability🤖 Generated with Claude Code