Skip to content

Support multiple competing LLMs #9

Description

@gbenson

e.g.

  • main/primary (x86+GPU, larger model)
  • backup/secondary/fallback (arm64, some smollm2, in case network down or ollama not running)

How to prioritize?

  • chat-router knows & invokes each directly (easy with current rpc-invocation style, easy to experiment (don't need to change hive-ollama), prob only good for main+fallback)
  • hive-ollamas consume fanout & indicate their model's "goodness" (probably a better solution definitely harder to experiment since both chat-router and hive-ollama need synchronous changes, could add external (paid) options maybe a later enhancement)

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions