Skip to content

Add embedder auto-truncation and settings serialization#78

Merged
AsphodelRem merged 8 commits into
mainfrom
icl_emb
Jun 26, 2026
Merged

Add embedder auto-truncation and settings serialization#78
AsphodelRem merged 8 commits into
mainfrom
icl_emb

Conversation

@bond005

@bond005 bond005 commented Jun 23, 2026

Copy link
Copy Markdown
Collaborator
  • EmbedderOpenAI now auto-truncates input to Settings.embedder_token_limit tokens
    before every API call (per-instance overrides via constructor). The context_truncator
    plumbing was removed from Index / KnowledgeGraph — truncation now lives inside the embedder.

  • Search engines no longer accept max_context_length / tokenizer_backend /
    tokenizer_model; BaseEngine reads these from Settings.

  • TokenTruncation (local backend) now accounts for special tokens ([CLS]/[SEP]),
    fixing silent over-limit requests against BGE/E5-style models.

  • Settings.save(path) / Settings.load(path) — JSON serialization of the config
    (Literal/int/str validation; storage_folder deliberately excluded) for reproducible runs.

  • Centralized tokenizer & token-limit configuration in the Settings singleton:
    new typed fields tokenizer_embedder_backend, tokenizer_llm_backend,
    tokenizer_embedder_name, tokenizer_llm_name, embedder_token_limit,
    llm_token_limit, llm_context_token_limit (with Literal["tiktoken","local"] validation).

  • Bugfix: SQLite database disk image is malformed corruption resolved via sqlite_synchronous=2.

  • New example examples/local_embedder_with_short_context.py (local vLLM embedder, 512-token context);
    README, AGENTS.md and subpackage READMEs updated.

@bond005 bond005 requested a review from AsphodelRem June 23, 2026 04:36
@AsphodelRem AsphodelRem merged commit 80f26b9 into main Jun 26, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants