Feature request: allow tools to measure output in tokens instead of lines/bytes

[hugosenari]

Describe the problem

Currently, vix's file-reading tools (read_file, read_minified_file) use lines as their unit for both input parameters (offset, limit) and output. The output is also capped at 20,000 characters.

For LLM-driven agents, the more relevant unit is tokens — models reason about and are constrained by token budgets, not line counts or byte counts. A 20,000-character cap might translate to wildly different token counts depending on the language (dense code vs. prose vs. data files). Similarly, a limit: 100 lines could be 200 tokens or 2000 tokens depending on line length.

Use case example

An agent wants to read "as much of this file as fits in 2000 tokens" — currently it has to guess based on lines. If the guess is wrong, it either wastes tokens on a too-large read or wastes a round-trip on a too-small read.

Proposed solution

Add an optional max_tokens parameter to read_file and read_minified_file as an alternative to limit (lines). When max_tokens is specified:

1. The tool estimates the token count of the file content (using a fast heuristic like len(text) / 4 or a proper tokenizer if available).
2. It returns only as many lines as fit within the token budget.
3. The tool result also reports the actual token count of what was returned.

For tool results across the board, consider adding a token_count field alongside line/byte info, so the agent can make informed decisions.

Additional context

This came up while reviewing the tool catalog in the vix manual:

• read_file — cap is "20,000 chars", params are offset and limit (lines)
• read_minified_file — described as "20–50% fewer tokens" but still uses line-based params
• MCP tool results already compact to row counts — a similar principle applied to token-aware limits would help everywhere

Related

• https://getvix.dev/docs#tool-catalog