Limit concurrent request

[Faylixe]

Hello and first of all thanks for your project. I have installed LLMKube on a k3s cluster made of 4 raspberry pi 5 to run multiple Gemma 4 models and it works great ! I will probably take some time to document it and / or try to contribute by submitting a k3s installation guide when I get some free time :).

I have a question regarding the deployment of multiple replicas. I will start by bring more context to my use case to make it more clear: I am developing my own AI harness to run against my homelab, which is built with following requirements at this time:

• 1 Raspberry PI 5 w/ 4 Go of RAM, acting as my cluster control plane (k3s server, llmkube controller, prometheus and grafana).
• 3 Raspberry PI 5 w/ 16 Go of RAM, acting as worker nodes in my cluster (k3s agents, and InferenceService)

I plan to have 1 Gemma 4 model hosted per agent node so 1 InferenceService scaled w/ 3 replica in total. I will add an ingress on top of that so I can have a dedicated endpoint my harness will hit. So far so good.

Since I am running on raw Raspberry PI with limited resources I would like to limit the concurrency of the traffic hitting my InferenceService so that only 1 inference request per replica can be executed at a time.

I assume I can work around by adding a nginx load balancer to enforce that in front of my inference service (but I feel like it won't be the best pattern as I would require to probably connect my load balancer to the pod instead of the svc), but I was wondering if InferenceService can handle that natively (maybe it already is ?) and if not, what you feel like would be the ideal solution to achieve that :).

[Defilan]

Thanks for the great writeup, and welcome! A 4-node Raspberry Pi 5 cluster running multiple Gemma 4 models on LLMKube is exactly the edge use case I had in mind when starting this project. Seeing it actually working is huge.

To your question: llama.cpp's server has a built-in --parallel flag that controls how many parallel inference slots a single process serves. Setting it to 1 makes each replica serialize requests at the runtime level, so subsequent requests queue inside llama-server until the active one finishes. Combined with the Kubernetes Service round-robining across your 3 replicas, that should give you exactly what you want without any external load balancer.

Today you'd configure it via extraArgs:

spec:
  runtime: llamacpp
  extraArgs:
    - "--parallel"
    - "1"

That said, this is one of the most-tuned llama.cpp flags for resource-constrained deployments, and it's a feature gap that it lives in extraArgs rather than being a first-class field on InferenceService. I've filed #339 to promote it to spec.maxParallelRequests. That would also unlock a future direction: HPA-aware autoscaling based on queue depth, since the operator would then know the per-replica capacity.

And yes, please do consider that k3s install guide if you find the time. The Pi 5 cluster pattern has been a long-standing curiosity in the homelab community, and a real-world walkthrough would help a lot of people. Happy to review docs PRs and integrate them under docs/deployment/k3s/ or similar.

[Faylixe]

Awesome ! That is all I need fo now ! And yes it works very smoothly honestly this is really perfect !

[Defilan]

Quick correction on what I told you above, courtesy of @Faylixe doing the work I should have done first.

InferenceService already has a first-class spec.parallelSlots field for llama.cpp (api/v1alpha1/inferenceservice_types.go#L92-L98), so the right shape for your use case is:

spec:
  runtime: llamacpp
  parallelSlots: 1

…with one catch we found while verifying: the controller currently only injects --parallel when parallelSlots > 1, so parallelSlots: 1 silently does nothing today. Until that lands (tracked in #339, going out tonight), the extraArgs: ["--parallel", "1"] workaround from my earlier reply is still the way to actually serialize on a Pi 5 replica.

After #339 ships, parallelSlots: 1 will be the canonical way to express it, the field will be documented, and we'll add the same knob for vLLM (--max-num-seqs) and Ollama (OLLAMA_NUM_PARALLEL) so the spec reads the same regardless of runtime.

Sorry for the detour, and thanks again for surfacing the use case.

[Defilan]

I realized I just tagged you in your own post, it's been a long day hehe. I need more coffee :)