Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/spelling_wordlist.txt
Original file line number Diff line number Diff line change
Expand Up @@ -318,6 +318,7 @@ contrib
convertToIri
cooldown
copyable
coreutils
CoreV
coroutine
cosmosdb
Expand Down Expand Up @@ -879,6 +880,7 @@ irreproducible
IRSA
isfile
ish
islo
isn
isort
istio
Expand Down Expand Up @@ -1069,6 +1071,7 @@ mgmt
microservice
microservices
microsoft
microVM
middleware
middlewares
midnights
Expand Down Expand Up @@ -1481,6 +1484,7 @@ sanitization
sas
Sasl
sasl
sbx
scala
scalability
scalable
Expand Down Expand Up @@ -1753,6 +1757,7 @@ tis
TLS
tls
tmp
tmpfs
tnsnames
todo
tokenization
Expand Down
47 changes: 24 additions & 23 deletions providers/common/ai/docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -239,29 +239,30 @@ Install them when installing from PyPI. For example:
pip install apache-airflow-providers-common-ai[anthropic]


============== =======================================================================================================================================
Extra Dependencies
============== =======================================================================================================================================
``anthropic`` ``pydantic-ai-slim[anthropic]``
``bedrock`` ``pydantic-ai-slim[bedrock]``
``google`` ``pydantic-ai-slim[google]``
``openai`` ``pydantic-ai-slim[openai]``
``mcp`` ``pydantic-ai-slim[mcp]``
``code-mode`` ``pydantic-ai-harness[codemode]>=0.3.0``
``shields`` ``pydantic-ai-shields>=0.3.4``
``skills`` ``apache-airflow-providers-git>=0.4.0``, ``pydantic-ai-skills>=1.2.0``
``avro`` ``fastavro>=1.10.0; python_version < "3.14"``, ``fastavro>=1.12.1; python_version >= "3.14"``
``parquet`` ``pyarrow>=18.0.0; python_version < '3.14'``, ``pyarrow>=22.0.0; python_version >= '3.14'``
``sql`` ``apache-airflow-providers-common-sql>=1.33.0``, ``sqlglot>=30.0.0``
``aws`` ``apache-airflow-providers-amazon>=9.0.0``
``common.sql`` ``apache-airflow-providers-common-sql>=1.33.0``
``langchain`` ``langchain>=1.0.0``
``llamaindex`` ``dataclasses-json>=0.6.7``, ``llama-index-core>=0.13.0``, ``llama-index-embeddings-openai>=0.6.0``, ``llama-index-llms-openai>=0.6.0``
``pdf`` ``pypdf>=4.0.0``
``docx`` ``python-docx>=1.0.0``
``git`` ``apache-airflow-providers-git``
``amazon`` ``apache-airflow-providers-amazon``
============== =======================================================================================================================================
================ =======================================================================================================================================
Extra Dependencies
================ =======================================================================================================================================
``anthropic`` ``pydantic-ai-slim[anthropic]``
``bedrock`` ``pydantic-ai-slim[bedrock]``
``google`` ``pydantic-ai-slim[google]``
``openai`` ``pydantic-ai-slim[openai]``
``mcp`` ``pydantic-ai-slim[mcp]``
``code-mode`` ``pydantic-ai-harness[codemode]>=0.3.0``
``shields`` ``pydantic-ai-shields>=0.3.4``
``skills`` ``apache-airflow-providers-git>=0.4.0``, ``pydantic-ai-skills>=1.2.0``
``sandbox-islo`` ``islo>=0.3.9``
``avro`` ``fastavro>=1.10.0; python_version < "3.14"``, ``fastavro>=1.12.1; python_version >= "3.14"``
``parquet`` ``pyarrow>=18.0.0; python_version < '3.14'``, ``pyarrow>=22.0.0; python_version >= '3.14'``
``sql`` ``apache-airflow-providers-common-sql>=1.33.0``, ``sqlglot>=30.0.0``
``aws`` ``apache-airflow-providers-amazon>=9.0.0``
``common.sql`` ``apache-airflow-providers-common-sql>=1.33.0``
``langchain`` ``langchain>=1.0.0``
``llamaindex`` ``dataclasses-json>=0.6.7``, ``llama-index-core>=0.13.0``, ``llama-index-embeddings-openai>=0.6.0``, ``llama-index-llms-openai>=0.6.0``
``pdf`` ``pypdf>=4.0.0``
``docx`` ``python-docx>=1.0.0``
``git`` ``apache-airflow-providers-git``
``amazon`` ``apache-airflow-providers-amazon``
================ =======================================================================================================================================

Downloading official packages
-----------------------------
Expand Down
176 changes: 174 additions & 2 deletions providers/common/ai/docs/toolsets.rst
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Airflow's 350+ provider hooks already have typed methods, rich docstrings,
and managed credentials. Toolsets expose them as pydantic-ai tools so that
LLM agents can call them during multi-turn reasoning.

Four toolsets are included:
The toolsets include:

- :class:`~airflow.providers.common.ai.toolsets.aws.AWSToolset` — configured
AWS services toolset for agent access to AWS APIs through Airflow-managed
Expand All @@ -36,8 +36,10 @@ Four toolsets are included:
connections.
- :class:`~airflow.providers.common.ai.toolsets.sql.SQLToolset` — curated
4-tool database toolset.
- :class:`~airflow.providers.common.ai.toolsets.sandbox.SandboxToolset` —
execute agent-written Python in an isolated sandbox, off the Airflow worker.

All four implement pydantic-ai's
All of them implement pydantic-ai's
`AbstractToolset <https://ai.pydantic.dev/toolsets/>`__ interface and can be
passed to any pydantic-ai ``Agent``, including via
:class:`~airflow.providers.common.ai.operators.agent.AgentOperator`.
Expand Down Expand Up @@ -543,6 +545,166 @@ resolves sources to local ``SKILL.md`` directories that any loader accepts:
``resolve_skills`` needs the Git provider (for ``GitSkills``) but not pydantic-ai,
and removes any cloned directories when the ``with`` block exits.


``SandboxToolset``
------------------

:class:`~airflow.providers.common.ai.toolsets.sandbox.SandboxToolset` gives the
agent one tool, ``run_python_in_sandbox``, that executes agent-written Python in a
disposable sandbox provisioned by a
:class:`~airflow.providers.common.ai.sandbox.SandboxBackend` — off the Airflow
worker process. Airflow does not inject its context, connections, credentials,
or worker environment into the sandbox. Custom images can still contain
credentials, and hosted backends can provide their own identity, so image and
backend configuration remain the Deployment Manager's responsibility.

It is the counterpart to :ref:`code mode <code-mode>`. Code mode's ``run_code``
executes in-process on the worker via the Monty interpreter: it is glue between
tool calls, restricted to a subset of Python with no third-party imports.
``SandboxToolset`` instead ships the code to a real Python interpreter in an
isolated environment, so the agent can crunch data with the full language and
whatever the sandbox image provides — at the cost of provisioning a microVM
per run.

The sandbox is created lazily on the first ``run_python_in_sandbox`` call, shared by every
call within the agent run, and destroyed when the run ends. Each call runs a
fresh interpreter (``python3 -c``), so files written by earlier calls persist
in the sandbox while in-memory variables do not. A nonzero exit code or a
timeout is normal tool output — the model reads ``stderr`` and fixes its own
code. If a backend cannot confirm that timed-out code stopped, it destroys the
sandbox and returns ``sandbox_terminated: true``; the next call receives a
fresh sandbox and files from earlier calls are no longer available. Only
infrastructure errors (the ``sbx`` CLI missing, the islo API unreachable) fail
the task.

sbx backend (Docker Sandboxes)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

:class:`~airflow.providers.common.ai.sandbox.SbxSandboxBackend` runs each
sandbox in a `Docker Sandboxes <https://docs.docker.com/ai/sandboxes/>`__
microVM by driving the ``sbx`` CLI (``create`` / ``exec`` / ``rm``). Each
sandbox is a genuine microVM with its own kernel, so agent code is isolated by
a hardware boundary rather than a shared kernel. Command output is capped at
the first 64 KiB of each stream, and the microVM is torn down when the run
ends.

It is a Deployment Manager prerequisite to install the ``sbx`` binary
(``brew install docker/tap/sbx`` / ``winget install Docker.sbx``) and run
``sbx policy init`` once on the worker host; the backend needs no Python
dependency. Outbound network egress is governed by that host ``sbx policy``
rather than by the backend — use ``sbx policy init deny-all`` for a no-egress
default. The template image must provide the GNU ``timeout`` utility, which
enforces the per-command timeout (any Debian/Ubuntu-based image, including
``python:*-slim``, does).

.. code-block:: python

from airflow.providers.common.ai.operators.agent import AgentOperator
from airflow.providers.common.ai.sandbox import SbxSandboxBackend
from airflow.providers.common.ai.toolsets import SandboxToolset

AgentOperator(
task_id="sandboxed_analyst",
prompt="Estimate pi with a Monte Carlo simulation of one million points.",
llm_conn_id="pydanticai_default",
toolsets=[SandboxToolset(SbxSandboxBackend())],
)

Constructor parameters:

- ``image``: Container image for the sandbox (``sbx --template``).
Default ``"python:3.12-slim"``.
- ``memory``: Memory limit in binary units (e.g. ``"2g"``). ``sbx`` enforces a
1 GiB minimum. Default ``"2g"``.
- ``cpus``: Number of CPUs to allocate. ``None`` (default) uses the ``sbx``
default.
- ``sbx_path``: Path to the ``sbx`` binary. Default ``"sbx"``.
- ``create_timeout``: Seconds allowed for provisioning (first-run microVM boot
plus image pull can be slow). Default ``600``.

islo backend
^^^^^^^^^^^^

:class:`~airflow.providers.common.ai.sandbox.IsloSandboxBackend` runs each
sandbox in an `islo.dev <https://islo.dev>`__ microVM — hardware-level
isolation with no local Docker daemon required.

Requires the ``sandbox-islo`` extra:
``pip install "apache-airflow-providers-common-ai[sandbox-islo]"``

.. code-block:: python

from airflow.providers.common.ai.sandbox import IsloSandboxBackend

SandboxToolset(IsloSandboxBackend(islo_conn_id="islo_default"))

Credentials come from a generic Airflow connection (no custom connection type
is needed):

- ``password``: the islo API key. Required.
- ``host``: the compute URL. Optional.
- Extra: optional ``base_url`` and ``timeout`` (request timeout in seconds)
keys.

Constructor parameters:

- ``islo_conn_id``: Airflow connection ID for islo. Default ``"islo_default"``.
``None`` lets the SDK resolve credentials from its environment variables
(``ISLO_API_KEY`` etc.).
- ``image``, ``vcpus``, ``memory_mb``: sandbox image and sizing. ``None``
(default) uses the server default for each.
- ``internet_enabled``: outbound internet access. Default ``False`` (no egress);
``None`` defers to the server default.
- ``delete_after``: Server-side TTL in seconds after which the sandbox is
deleted even if the worker never got to destroy it (killed mid-run).
Default ``3600``.

The command timeout is enforced by the backend, not the islo API (the API's
``timeout_secs`` is only a compatibility hint). If no terminal command state
arrives within the timeout plus a short grace period, the backend deletes the
microVM, marks the sandbox terminated, and returns a timeout.

Parameters
^^^^^^^^^^

- ``backend``: The :class:`~airflow.providers.common.ai.sandbox.SandboxBackend`
that provisions and runs the sandbox.
- ``timeout``: Timeout in seconds for a single ``run_python_in_sandbox`` call.
Default ``300``.
- ``python_command``: Python executable inside the sandbox.
Default ``"python3"``.

Bringing your own backend
^^^^^^^^^^^^^^^^^^^^^^^^^

Any vendor that can create a sandbox, run a command in it, and destroy it can
plug in: subclass
:class:`~airflow.providers.common.ai.sandbox.SandboxBackend` in your own
package and pass an instance to ``SandboxToolset``:

.. code-block:: python

from airflow.providers.common.ai.sandbox import SandboxBackend, SandboxResult


class AcmeSandboxBackend(SandboxBackend):
name = "acme"

def create(self) -> str:
return acme_sdk.create_sandbox().id

def run(self, sandbox: str, command: list[str], *, timeout: float) -> SandboxResult:
r = acme_sdk.exec(sandbox, command, timeout=timeout)
return SandboxResult(exit_code=r.exit_code, stdout=r.stdout, stderr=r.stderr)

def destroy(self, sandbox: str) -> None:
acme_sdk.delete_sandbox(sandbox)

Constructors run at Dag-parse time, so resolve credentials lazily, on first
use, and make ``destroy`` idempotent (destroying an already-gone sandbox must
not raise).


Working with LangChain
----------------------

Expand Down Expand Up @@ -692,6 +854,16 @@ No single layer is sufficient — they work together.
- Does **not** constrain what those tools do. An MCP server can expose
shell, filesystem, or network access. Run only trusted servers and
audit the tools they expose.
* - **SandboxToolset: off-worker execution**
- Executes agent-written code in a disposable microVM, never in the
worker process. Airflow does not inject its context, connections,
credentials, or worker environment. The backends enforce command
timeouts, cap each output stream at 64 KiB, and tear down the sandbox
after the run; the islo backend also sets a server-side cleanup TTL.
- Does not sanitize what the code computes or returns. Custom images can
contain secrets, hosted backends can expose their own identity, and
Docker Sandboxes network access follows the host ``sbx policy``. Its
CPU allocation defaults to all host CPUs unless explicitly limited.
* - **pydantic-ai: tool call budget**
- pydantic-ai's ``max_result_retries`` and ``model_settings`` control
how many tool-call rounds the agent can make before stopping.
Expand Down
7 changes: 7 additions & 0 deletions providers/common/ai/provider.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,12 @@ integrations:
- /docs/apache-airflow-providers-common-ai/operators/llamaindex_embedding.rst
- /docs/apache-airflow-providers-common-ai/operators/llamaindex_retrieval.rst
tags: [ai]
- integration-name: Docker Sandboxes
external-doc-url: https://docs.docker.com/ai/sandboxes/
tags: [software]
- integration-name: Islo
external-doc-url: https://docs.islo.dev/
tags: [service]

hooks:
- integration-name: Pydantic AI
Expand Down Expand Up @@ -460,6 +466,7 @@ toolsets:
- airflow.providers.common.ai.toolsets.datafusion
- airflow.providers.common.ai.toolsets.logging
- airflow.providers.common.ai.toolsets.mcp
- airflow.providers.common.ai.toolsets.sandbox
- airflow.providers.common.ai.toolsets.skills
- airflow.providers.common.ai.toolsets.langchain_bridge

Expand Down
5 changes: 5 additions & 0 deletions providers/common/ai/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,10 @@ dependencies = [
"apache-airflow-providers-git>=0.4.0",
"pydantic-ai-skills>=1.2.0",
]
# Isolated sandbox code execution for agents (SandboxToolset). The islo backend
# needs its SDK; the sbx backend only shells out to the `sbx` CLI, so it needs
# no Python dependency and has no extra.
"sandbox-islo" = ["islo>=0.3.9"]
"avro" = [
'fastavro>=1.10.0; python_version < "3.14"',
'fastavro>=1.12.1; python_version >= "3.14"',
Expand Down Expand Up @@ -153,6 +157,7 @@ dev = [
"llama-index-core>=0.13.0",
"llama-index-embeddings-openai>=0.6.0",
"llama-index-llms-openai>=0.6.0",
"islo>=0.3.9",
]

# To build docs:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,12 @@ def get_provider_info():
],
"tags": ["ai"],
},
{
"integration-name": "Docker Sandboxes",
"external-doc-url": "https://docs.docker.com/ai/sandboxes/",
"tags": ["software"],
},
{"integration-name": "Islo", "external-doc-url": "https://docs.islo.dev/", "tags": ["service"]},
],
"hooks": [
{
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""Isolated sandbox backends for the SandboxToolset."""

from __future__ import annotations

from airflow.providers.common.ai.sandbox.base import SandboxBackend, SandboxResult

# SbxSandboxBackend only shells out to the `sbx` CLI (stdlib imports), so it is
# always importable; IsloSandboxBackend needs the optional `islo` SDK.
from airflow.providers.common.ai.sandbox.sbx import SbxSandboxBackend

__all__ = ["IsloSandboxBackend", "SandboxBackend", "SandboxResult", "SbxSandboxBackend"]


def __getattr__(name: str):
if name == "IsloSandboxBackend":
try:
from airflow.providers.common.ai.sandbox.islo import IsloSandboxBackend
except ImportError as e:
from airflow.providers.common.compat.sdk import AirflowOptionalProviderFeatureException

raise AirflowOptionalProviderFeatureException(e)
return IsloSandboxBackend
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
Loading