Skip to content

Fix chunking file_format placement: Markdown-aware chunking was never enabled - #557

Open
claudespice wants to merge 1 commit into
spiceai:trunkfrom
claudespice:claude/fix-chunking-file-format-placement
Open

Fix chunking file_format placement: Markdown-aware chunking was never enabled#557
claudespice wants to merge 1 commit into
spiceai:trunkfrom
claudespice:claude/fix-chunking-file-format-placement

Conversation

@claudespice

Copy link
Copy Markdown
Contributor

Summary

Three recipes set file_format: md inside the chunking: block of a column embedding:

chunking:
  enabled: true
  target_chunk_size: 256
  overlap_size: 64
  file_format: md      # <-- silently ignored here

chunking: deserializes into EmbeddingChunkConfig, which has exactly four fields — enabled, target_chunk_size, overlap_size, trim_whitespace — and no file_format. The struct does not carry #[serde(deny_unknown_fields)], so the key is silently dropped rather than reported: the recipes look like they enable Markdown-aware chunking, but every one of them has been chunking with the plain text splitter.

The runtime reads file_format from the dataset's params, not from the chunking block. This PR moves the key up into params:, which is also the placement used by the file recipe (file/README.md) and by spiceai's own test spicepods (test/models/spicepod_openai.yml).

Recipes fixed:

  • models/openai (spicepod.yaml + the README's copy of the same block)
  • azure_openai (spicepod.yaml)
  • search_github_files (spicepod.yaml, commented-out example)

Verified against

spiceai/spiceai at v2.1.1:

git log -S file_format confirms the misplacement predates #508 and #136, which only flipped chunking.enabled from false to true — so this is not reverting a deliberate choice.

One thing for reviewers to weigh

The github connector does not list file_format in its PARAMETERS spec, so it will log one line at startup:

Ignoring parameter file_format: not supported for github.

This is cosmetic — the chunker reads the raw params map, so Markdown chunking does take effect — but if you would rather not have the warning, the alternative is to drop the file_format line entirely and accept plain-text chunking (which is what these recipes have actually been doing all along). Happy to switch to that if preferred.

Evidence

Static review only — these recipes need a GITHUB_TOKEN plus OpenAI/Azure credentials, so they were not run. The config path is traced end-to-end in the source links above, and all three files were re-parsed as valid YAML after the change.

file_format belongs in the dataset's params, not inside the chunking
block. Inside chunking: it is silently dropped, so Markdown-aware
chunking was never enabled.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant