Add multi-model support infrastructure for SAFE framework#37
Open
Robbyswimmer wants to merge 17 commits into
Open
Add multi-model support infrastructure for SAFE framework#37Robbyswimmer wants to merge 17 commits into
Robbyswimmer wants to merge 17 commits into
Conversation
- Log evaluation entry point with GPU memory stats - Log batch processing progress (every 10 batches) - Log CIDEr/SPICE computation steps - Free GPU cache before SPICE (Java subprocess) - This will pinpoint where the OOM crash occurs
- Detect if captions are already in list format (audiocaps_val.json) - Skip re-grouping if data is pre-grouped - Add logging to show which file is loaded - This should fix avg=1.0 references issue
Owner
Author
|
@codex review for issues |
|
To use Codex here, create a Codex account and connect to github. |
Owner
Author
|
@claude review for issues |
Adds comprehensive support for different backbone models beyond LLaVA 1.5: Model Registry (safe/models/model_registry.py): - Centralized registry for all supported models with ModelSpec dataclass - Support for Vision-Language, Text-Only, and API model types - Pre-registered: LLaVA (7B/13B), BLIP2, Llama 3.1/3.2, Mistral, Qwen, Gemma Base Model Abstraction (safe/models/base_model.py): - Abstract BaseModel class with unified interface - TextOnlyModel for Llama, Mistral, Qwen, Phi, Gemma - VisionLanguageModel for LLaVA, BLIP2 - APIModel for Gemini, GPT-4, Claude - Flash Attention 2 and quantization (8-bit/4-bit) support Updated BaseVLModel (safe/models/base_vl.py): - Auto model type detection via _detect_model_type() - Text-only model support without vision encoder - New properties: has_vision, hidden_size, num_layers, num_attention_heads - get_decoder_layers() method for layer hook injection New Model Configurations (configs/model_configs.py): - llava-7b: LLaVA 1.5 7B (18GB VRAM) - llama-3.1-8b: Llama 3.1 8B (20GB VRAM) - llama-3.2-3b: Llama 3.2 3B (10GB VRAM) - mistral-7b: Mistral 7B v0.3 (18GB VRAM) - qwen-2.5-7b: Qwen 2.5 7B (18GB VRAM) - gemma-2-9b: Gemma 2 9B (22GB VRAM) - gemini-flash: Gemini 1.5 Flash (API) This enables using modern text-only LLMs like Llama 3.1 8B for audio fusion experiments without requiring vision capabilities.
Robbyswimmer
force-pushed
the
claude/multi-model-support-bVa0D
branch
from
51b582e to
c6b6477
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This commit introduces comprehensive support for using different backbone
models beyond LLaVA 1.5 13B, including:
Model Registry (safe/models/model_registry.py):
Mistral 7B, Qwen 2.5, Phi-3, Gemma 2, and API models (Gemini, GPT-4)
Base Model Abstraction (safe/models/base_model.py):
Updated BaseVLModel (safe/models/base_vl.py):
Model Configurations (configs/model_configs.py):
This enables experimenting with modern LLMs like Llama 3.1 and smaller
efficient models for faster iteration and reduced memory requirements.