vLLM Tool Calling

Welcome to the vLLM Function Calling Quickstart!

Use this to quickly get a vLLM runtime with Function Calling enabled in your OpenShift AI environment, loading models directly from ModelCar containers.

To see how it's done, jump straight to installation.

Table of Contents

• vLLM Tool Calling
• Detailed description
  • See it in action
  • Architecture diagrams
  • References
• Requirements
  • Minimum hardware requirements
  • Required software
  • Required permissions
• Install
  • Clone the repository
  • Create the project
  • Choose your LLM to be deployed
  • Check the deployment

Detailed description

The vLLM Function Calling Quickstart is a template for deploying vLLM with Function Calling enabled, integrated with ModelCar containerized models, within Red Hat OpenShift AI.

It’s designed for environments where you want to:

• Enable LLMs to call external tools (Tool/Function Calling).
• Serve LLMs (like Granite3, Llama3) directly from a container.
• Easily customize your model deployments without needing cluster admin privileges.

Use this project to quickly spin up a powerful vLLM instance ready for function-enabled Agents or AI applications.

Architecture diagrams

References

• The runtime is out of the box in RHOAI called vLLM ServingRuntime for KServe
• Detailed guide and documentations is available in this article.
• Code for testing the Function Calling in OpenShift AI is in github.com/rh-aiservices-bu/llm-on-openshift

NOTE: To find more patterns and pre-built ModelCar images, take a look at the Red Hat AI Services ModelCar Catalog repo on GitHub and the ModelCar Catalog registry on Quay.

Requirements

Minimum hardware requirements

• 8+ vCPUs
• 24+ GiB RAM
• Storage: 30Gi minimum in PVC (larger models may require more)

Optional, depending on selected hardware platform

• 1 GPU (NVIDIA L40, A10, or similar)
• 1 Intel® Gaudi® AI Accelerator
• For optimal performance on Xeon CPUs, refer to the Intel AI Performance Advisor for cloud instances and on-prem servers for a sizing guide, as well as vLLM Recipes for the model recipes.

Required software

• Red Hat OpenShift
• Red Hat OpenShift AI 2.16+
• Dependencies for Single-model server:
  • Red Hat OpenShift Service Mesh
  • Red Hat OpenShift Serverless

Required permissions

• Standard user. No elevated cluster permissions required

Install

Please note before you start

This example was tested on Red Hat OpenShift 4.16.24 & Red Hat OpenShift AI v2.16.2.

Clone the repository

git clone https://github.com/rh-ai-quickstart/vllm-tool-calling.git && \
    cd vllm-tool-calling/  

Create the project

PROJECT can be set to any value. This will also be used as the namespace.

export PROJECT="vllm-tool-calling-demo"

oc new-project $PROJECT

Specify your LLM and device:

• MODEL: select from [granite3.2-8b, llama3.2-1b, llama3.2-3b]
• DEVICE: select from [cpu, gpu, hpu]

Set variables to the selected options. Example is shown below.

export MODEL="granite3.2-8b"
export DEVICE="gpu"

Deploy the LLM on the target hardware:

oc apply -n $PROJECT -k vllm-tool-calling/$MODEL/$DEVICE

Check the deployment

• From the OpenShift Console, go to the App Switcher / Waffle in the upper right and go to the Red Hat OpenShift AI Dashboard.

• Once inside the dashboard, navigate to Data Science Projects or Projects -> vllm-tool-calling-demo (or the values of ${PROJECT} if changed from the default):

• Check the models deployed, and wait for the green tick in the Status, meaning that the model is deployed successfully:

• Important: Note down the URL of the inference endpoint and resource name of the model deployment. They will be used in the function calling section.

Alternatively, the model status can be checked using OC client:

oc get pods
oc logs -f <model-name-device-predictor-pod>

Look for Application startup complete.

Function Calling in OpenShift AI

Now that the model is deployed, it can be used with function calling. The notebook from github.com/rh-aiservices-bu/llm-on-openshift is used for this example.

Inside the OpenShift AI Dashboard, go to Data Science Projects or Projects -> vllm-tool-calling-demo (or the values of ${PROJECT} if changed from the default) -> Workbenches. Create a workbench based on the hardware platform used. When it is in the Running state, click on the name of the workbench to open the Jupyter environment.

In the Launcher, open a Terminal. Clone the rh-aiservices-bu repo:

git clone https://github.com/rh-aiservices-bu/llm-on-openshift.git

On the left-hand side, navigate to rh-aiservices-bu/llm-on-openshift/examples/notebooks/langchain. Open the Langchain-FunctionCalling.ipynb notebook.

Execute the first few cells. In section 3 Model Configuration, set the following variables depending on the model deployed:

• INFERENCE_SERVER_URL: the inference endpoint noted down AND append :8080 since that is the default port for vLLM
• MODEL_NAME: the model deployment name noted down
• API_KEY: set to "EMPTY"

Continue running the rest of the notebook to invoke tools. The query can also be customized.

Cleanup

To remove all deployed components:

oc delete -n $PROJECT -f vllm-tool-calling/$MODEL/$DEVICE

Delete the project:

oc delete project $PROJECT