Skip to content

feat: support multiple images in conversations and add PDF-to-image utility#313

Draft
fcarli wants to merge 8 commits into
stagingfrom
dev-images
Draft

feat: support multiple images in conversations and add PDF-to-image utility#313
fcarli wants to merge 8 commits into
stagingfrom
dev-images

Conversation

@fcarli

@fcarli fcarli commented Jun 18, 2025

Copy link
Copy Markdown
Contributor

Summary

This PR enhances BioChatter's multimodal capabilities by adding support for multiple images in conversations and introducing PDF-to-image conversion functionality.

Key Changes:

  1. Multiple Image Support:

    • Updated append_image_message() and query() methods to accept either a single image URL or a list of image URLs
    • Maintains backward compatibility with existing single-image functionality
    • Added local parameter to the query() method for proper handling of local vs remote images
  2. PDF-to-Image Conversion:

    • Added new pdf_pages_to_images() utility function in biochatter/utils.py
    • Converts PDF documents into individual page images using PyMuPDF and PIL
    • Supports configurable DPI, image format, and output naming
    • Enables document question-answering by passing PDF pages as images to multimodal models
  3. Documentation & Testing:

    • Updated multimodal documentation with examples for both features
    • Added comprehensive test coverage for single/multiple image handling
    • Tests cover local files, remote URLs, and mixed scenarios

Benefits:

  • Enables analysis of multiple images simultaneously (e.g., comparing figures, processing document pages)
  • Provides fast document Q&A by leveraging multimodal models' image processing capabilities
  • Offers an alternative to traditional PDF parsing for noisy documents with embedded images

The changes are fully backward compatible and extend BioChatter's multimodal functionality for more complex use cases involving multiple images and PDF documents.

@fcarli fcarli self-assigned this Jun 18, 2025
@fcarli fcarli added the enhancement New feature or request label Jun 18, 2025
@fcarli fcarli moved this to In Progress in BioCypher Development Jun 18, 2025
@fcarli fcarli moved this to In Progress in OTAR3088 Jun 18, 2025
text: str,
image_url: str | None = None,
image_url: str | list[str] | None = None,
local: bool = False,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe the param name should refer to the image-related function, otherwise could be unclear (this is the generic query function, and params typically refer to the purpose of the method). Call it image_local or similar?

@slobentanzer

Copy link
Copy Markdown
Contributor

@fcarli looks good; only one comment on the param name. As it seems a minor update, do you suppose we bump patch?

@slobentanzer

Copy link
Copy Markdown
Contributor

@fcarli bump

@slobentanzer slobentanzer assigned slobentanzer and unassigned fcarli Nov 6, 2025
@slobentanzer slobentanzer changed the base branch from main to staging June 24, 2026 21:55
@VladSam42

Copy link
Copy Markdown
Collaborator

@fcarli, the architecture can be improved a bit because the current approach with the local flag can break integration with frontends later. A better approach would be:

  • enforce accepting only image_urls: str | list[str] (either web urls or base64 encoded0, and also raise errors if incorrect. The whole concept of "local files" shouldn't exist at this level and providing the correct url should be handled by the UI layer.
  • defer conversion to modules outside; the same way you also kept pdf conversion outside

@VladSam42 VladSam42 self-requested a review June 26, 2026 08:10
@VladSam42 VladSam42 marked this pull request as draft June 26, 2026 08:11
self,
message: str,
image_url: str,
image_url: str | list[str],

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

enforce accepting only image_urls: str | list[str] (either web urls or base64 encoded0, and also raise errors if incorrect. The whole concept of "local files" shouldn't exist at this level and providing the correct url should be handled by the UI layer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

Status: In Progress
Status: In Progress

Development

Successfully merging this pull request may close these issues.

3 participants