feat: cross‑platform Florence/YOLO startup via flash_attn guard by Santosh69 · Pull Request #22 · ruxailab/ai-heuristic-evaluation

Santosh69 · 2026-02-05T16:28:03Z

Summary

Add cross‑platform dev script start-dev.py and OmniParser setup script setup_omniparser.py.
Clarify setup steps and model weight requirements in README.
Prevent Florence startup failures on CPU/MPS by skipping flash_attn checks.
Pin/update dependencies required by the patch and setup tooling.

Changes

Add start-dev.py and scripts/setup_omniparser.py.
Add app/utils/transformers_patches.py and apply it in omniparser_client.py.
Update OmniParser init (device selection, YOLO path, logging, metadata).
Updated requirements.txt (torch/transformers pins + einops, huggingface-hub, accelerate).
Updated .gitignore to ignore local weights and Firebase config.
Refresh README quick start + manual setup flow.

Testing

- Update code to use model.pt (matches OmniParser release) - Add .gitignore patterns for model weight files - Remove *.rst from .gitignore - Add Prerequisites section to README with download instructions - Fix verify_model_load.py to use model.pt Sub-issue of MVP demo preparation. Fixes FileNotFoundError for contributors.

Copilot

Pull request overview

This pull request adds cross-platform development tooling and fixes Florence model initialization failures on non-CUDA systems by patching the flash_attn dependency check. The changes enable the project to run on CPU and MPS devices.

Changes:

Added cross-platform development launcher (start-dev.py) and OmniParser weight setup script (scripts/setup_omniparser.py)
Implemented transformers patch to skip flash_attn checks on non-CUDA systems
Updated YOLO model path from best.pt to model.pt and enhanced Florence model initialization with device selection
Pinned critical dependencies and added new requirements for the patch functionality

Reviewed changes

Copilot reviewed 5 out of 7 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
start-dev.py	New cross-platform development launcher that verifies weights and starts backend/Firebase services
scripts/setup_omniparser.py	New script that downloads and organizes OmniParser model weights from HuggingFace
app/utils/transformers_patches.py	New module that patches transformers to prevent false-positive flash_attn errors on CPU/MPS
app/services/omniparser_client.py	Updated to apply transformers patch, change YOLO path, add device selection, and enhance logging
requirements.txt	Pinned torch/transformers versions and added einops, huggingface-hub, and accelerate dependencies
.gitignore	Removed markdown/rst exclusions, added firebase config and weights folder to ignore list
README.md	Restructured with new quick start instructions and detailed model weight setup options

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-05T16:38:09Z

+        ).to(device)
+
+        self.logger.info("✅ Florence model loaded")
+        self.processor = AutoProcessor.from_pretrained("microsoft/Florence-2-base-ft", trust_remote_code=True)


The processor is being loaded from the remote repository "microsoft/Florence-2-base-ft" while the model is loaded from local weights at "weights/icon_caption_florence". This creates an inconsistency where the processor and model might be from different versions. The processor should be loaded from the same local path as the model to ensure compatibility. Consider changing this to load from "weights/icon_caption_florence" to match the model loading on line 229.

Suggested change

self.processor = AutoProcessor.from_pretrained("microsoft/Florence-2-base-ft", trust_remote_code=True)

self.processor = AutoProcessor.from_pretrained("weights/icon_caption_florence", trust_remote_code=True)

Copilot · 2026-02-05T16:38:09Z

+
+if setup_script.exists():
+    print("Verifying OmniParser V2 weights...")
+    subprocess.run([sys.executable, str(setup_script)], check=True)


The script checks subprocess.run() with check=True but doesn't handle the CalledProcessError that could be raised if the setup script fails. This will result in an unhandled exception. Consider wrapping this in a try-except block to provide a better error message to users when setup fails.

Suggested change

subprocess.run([sys.executable, str(setup_script)], check=True)

try:

subprocess.run([sys.executable, str(setup_script)], check=True)

except subprocess.CalledProcessError as e:

print(

f"Error: scripts/setup_omniparser.py failed with exit code {e.returncode}.",

file=sys.stderr,

)

sys.exit(e.returncode or 1)

Copilot · 2026-02-05T16:38:10Z

+    try:
+        snapshot_download(
+            repo_id="microsoft/OmniParser-v2.0",
+            revision=None,


Setting revision=None explicitly might cause confusion. The default behavior of snapshot_download is to use the main/master branch when revision is not specified. Consider either removing this parameter entirely (to use the default) or setting it to a specific commit hash or tag for reproducibility. Using None here is equivalent to omitting the parameter.

Suggested change

revision=None,

Copilot · 2026-02-05T16:38:10Z

+    sys.exit(0)
+
+signal.signal(signal.SIGINT, shutdown_handler)
+signal.signal(signal.SIGTERM, shutdown_handler)


On Windows, SIGTERM is not available. This will cause an AttributeError when the script tries to register the signal handler. Consider wrapping this in a try-except block or checking for the signal's availability with hasattr(signal, 'SIGTERM') before registering.

Suggested change

signal.signal(signal.SIGTERM, shutdown_handler)

if hasattr(signal, "SIGTERM"):

signal.signal(signal.SIGTERM, shutdown_handler)

Copilot · 2026-02-05T16:38:10Z

+    # if model exists, do nothing
+    safetensors_path = os.path.join(target_caption_folder, "model.safetensors")
+    if os.path.exists(safetensors_path):
+        print(f"✅ Models found in {target_caption_folder}. Skipping download.")


The script only checks for the existence of "model.safetensors" to determine if models are present, but according to the README and the code, it should also verify that the YOLO model at "weights/icon_detect/model.pt" exists. Currently, if the Florence model exists but the YOLO model is missing, the script will skip the download entirely, leading to runtime errors later.

Suggested change

# if model exists, do nothing

safetensors_path = os.path.join(target_caption_folder, "model.safetensors")

if os.path.exists(safetensors_path):

print(f"✅ Models found in {target_caption_folder}. Skipping download.")

# if both caption and detection models exist, do nothing

safetensors_path = os.path.join(target_caption_folder, "model.safetensors")

yolo_model_path = os.path.join(weights_dir, "icon_detect", "model.pt")

if os.path.exists(safetensors_path) and os.path.exists(yolo_model_path):

print(f"✅ Models found in {target_caption_folder} and icon_detect. Skipping download.")

Copilot · 2026-02-05T16:38:11Z

+
+try:
+    from huggingface_hub import snapshot_download
+except Exception:


The exception handling uses a bare "except Exception:" which is too broad and might catch unexpected errors like KeyboardInterrupt (though Python 3 handles this differently). Consider catching more specific exceptions like ImportError, ModuleNotFoundError to provide more accurate error messages to users.

Suggested change

except Exception:

except (ImportError, ModuleNotFoundError):

Copilot · 2026-02-05T16:38:11Z

+            cwd=firebase_dir,
+            shell=(os.name == "nt")


Setting shell=True only on Windows (os.name == "nt") may cause inconsistent behavior. The "npm run serve" command should work without shell=True on all platforms since "npm" is a regular executable. Consider removing the shell parameter entirely, or if it's needed on Windows for some reason, add a comment explaining why.

Suggested change

cwd=firebase_dir,

shell=(os.name == "nt")

cwd=firebase_dir

Copilot · 2026-02-05T16:38:11Z

 ## Quick Start

-### Using the Development Script
+### Quick Start (Recommended)


The README title shows "Quick Start" twice - first as "## Quick Start" (line 65) and then immediately as "### Quick Start (Recommended)" (line 67). This creates redundancy and an inconsistent heading hierarchy. Consider removing the "## Quick Start" header and keeping only the "### Quick Start (Recommended)" subsection, or restructure to have "## Quick Start" with multiple subsections beneath it.

Suggested change

### Quick Start (Recommended)

### Recommended

Santosh69 added 9 commits January 18, 2026 16:06

docs: improve onboarding and model weights instructions in README

67bb208

docs: add curl alternative for model download in README

d8c545f

feat: add onboarding/start-dev onboarding scripts

81c2f74

docs: switch dev script to start-dev.py

c14b0f4

docs: clarify setup and ignore local weights

40d4191

chore: skip firebase-functions when npm is missing

f307912

fix(transformers): skip flash_attn on CPU/MPS

80def70

fix(omniparser): apply flash_attn patch before Florence init

970774e

Copilot AI review requested due to automatic review settings February 5, 2026 16:28

Copilot started reviewing on behalf of Santosh69 February 5, 2026 16:28 View session

Navigation Menu

Uh oh!

feat: cross‑platform Florence/YOLO startup via flash_attn guard - #22

feat: cross‑platform Florence/YOLO startup via flash_attn guard#22
Santosh69 wants to merge 9 commits into
ruxailab:mainfrom
Santosh69:fix/transformers-flash-attn

Santosh69 commented Feb 5, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 5, 2026

Uh oh!

Copilot AI Feb 5, 2026

Uh oh!

Copilot AI Feb 5, 2026

Uh oh!

Copilot AI Feb 5, 2026

Uh oh!

Copilot AI Feb 5, 2026

Uh oh!

Copilot AI Feb 5, 2026

Uh oh!

Copilot AI Feb 5, 2026

Uh oh!

Copilot AI Feb 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	self.processor = AutoProcessor.from_pretrained("microsoft/Florence-2-base-ft", trust_remote_code=True)
	self.processor = AutoProcessor.from_pretrained("weights/icon_caption_florence", trust_remote_code=True)

-    subprocess.run([sys.executable, str(setup_script)], check=True)
+    try:
+        subprocess.run([sys.executable, str(setup_script)], check=True)
+    except subprocess.CalledProcessError as e:
+        print(
+            f"Error: scripts/setup_omniparser.py failed with exit code {e.returncode}.",
+            file=sys.stderr,
+        )
+        sys.exit(e.returncode or 1)

	signal.signal(signal.SIGTERM, shutdown_handler)
	if hasattr(signal, "SIGTERM"):
	signal.signal(signal.SIGTERM, shutdown_handler)

Uh oh!

Conversation

Santosh69 commented Feb 5, 2026

Summary

Changes

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants