Skip to content

Add tier-1 BEI-torch example templates - #584

Open
bohanyao-baseten wants to merge 1 commit into
mainfrom
encoder-torch-templates
Open

Add tier-1 BEI-torch example templates#584
bohanyao-baseten wants to merge 1 commit into
mainfrom
encoder-torch-templates

Conversation

@bohanyao-baseten

Copy link
Copy Markdown

Summary

Three tier-1 truss templates for base_model: encoder_torch (BEI-torch backend), in a new 11-embeddings-reranker-classification-torch/ directory mirroring the existing -tensorrt/ layout:

  • BEI-torch-nvidia-Nemotron-3-Embed-8B-bf16 (H100)
  • BEI-torch-nvidia-Nemotron-3-Embed-1B-bf16 (L4)
  • BEI-torch-nvidia-llama-nemotron-rerank-1b-v2 (L4)

Paired with basetenlabs/truss#2592 and basetenlabs/baseten#25084.

Plan: https://app.notion.com/p/3ae91d24727381d5902de1d02944ead4

Notes

  • Templates are hand-written, not yet wired into ../11-embeddings-reranker-classification-tensorrt/templating/generate_templates.py. A BEI_TORCH Solution class needs to land there before tier-2/3 templates get added. Flagged as follow-up.
  • Each README calls out the three-JSON-patch workaround needed on the stock NVIDIA repos (transformers-incompatible tokenizer_class: "TokenizersBackend", list-form extra_special_tokens, unregistered model_type: "llama_bidirec") until BEI-torch lands the serde alias + tokenizer-config fallback upstream. Customers point at a forked repo with those patches for now.

Test plan

  • Wait for Add base_model=encoder_torch for BEI-torch backend truss#2592 and basetenlabs/baseten#25084 to merge and the BEI_TORCH_IMAGE_URI constance to be set to a real tag.
  • For each template: truss push --publish against a patched HF fork, confirm MODEL_READY, check /v1/embeddings (or /rerank) returns valid outputs, run a cosine-similarity spot-check against a vLLM reference.

🤖 Generated with Claude Code

Three hand-written templates for `base_model: encoder_torch`
(BEI-torch backend). Lives in a new `11-embeddings-reranker-
classification-torch/` directory mirroring the existing `-tensorrt/`
layout.

- BEI-torch-nvidia-Nemotron-3-Embed-8B-bf16 (H100)
- BEI-torch-nvidia-Nemotron-3-Embed-1B-bf16 (L4)
- BEI-torch-nvidia-llama-nemotron-rerank-1b-v2 (L4)

Each README documents the three-JSON-patch workaround for the
stock NVIDIA repos (transformers-incompatible tokenizer_class,
extra_special_tokens list, unregistered `llama_bidirec` model_type)
until BEI-torch lands the serde alias and tokenizer-config fallback.

Not yet wired into `generate_templates.py` — follow-up work to add a
`BEI_TORCH` `Solution` class before tier-2/3.

Paired with:
- basetenlabs/truss#2592
- basetenlabs/baseten#25084

Plan: https://app.notion.com/p/3ae91d24727381d5902de1d02944ead4

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@bohanyao-baseten
bohanyao-baseten force-pushed the encoder-torch-templates branch from 828fb03 to 7be48e4 Compare August 4, 2026 21:58
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@bohanyao-baseten
bohanyao-baseten force-pushed the encoder-torch-templates branch from 7be48e4 to 828fb03 Compare August 4, 2026 22:19
@bohanyao-baseten
bohanyao-baseten marked this pull request as ready for review August 5, 2026 21:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants