Optimize entropy scanner: skip dirs, limit size, and stream files

[Sejusahu011]

Before opening: make sure there is an issue tracking this work, and link it below. PRs without a linked issue may be closed without review.

Linked issue

Closes #226

What this PR does

This PR optimizes the entropy.py scanner to improve performance and prevent memory crashes on large repositories. It introduces directory filtering, a 1MB file size limit, and implements line-by-line file streaming.

Type of change

• Bug fix
• New feature
• ML model / training pipeline
• Refactor (no behaviour change)
• Documentation
• Tests only

ML tier (if applicable)

• Tier 1 — Triage
• Tier 2 — Predictive
• Tier 3 — Autonomous
• Not ML-related

Stack affected

• Backend
• Frontend
• Both

---

Changes

Backend

• Added ignored_dirs to skip unnecessary folders.
  Implemented MAX_FILE_SIZE = 1MB limit.
  Switched to line-by-line file streaming for memory efficiency.

Frontend

-none

New dependencies

-none

Database / schema changes

-none

---

Testing

How did you test this?
Tested locally by scanning repositories with large files and ignored directories. Verified that the scanner correctly skips ignored paths and processes only allowed files without memory spikes.

Checklist

• Tested locally end-to-end (upload ZIP or GitHub URL → scan → findings returned correctly)
• New ML model falls back gracefully when model file is absent
• No new console.error or unhandled Python exceptions introduced
• Added or updated tests where applicable
• requirements.txt / package.json updated if new dependencies added
• New model files (.pkl, .pt, etc.) are gitignored, not committed

---

Anything reviewers should focus on

The ignored_dirs list is currently set to common patterns. Please verify if any other directory types should be added.

Screenshots (if UI changed)

None

[github-actions]

🎉 Thank you @Sejusahu011 for submitting a Pull Request!

We're excited to review your contribution.

Before Review

✅ Ensure all CI checks pass
✅ Complete the PR template
✅ Link the related issue

⚡ Want faster reviews and contributor support?

Join our Discord community:

🔗 https://discord.gg/FcXuyw2Rs

Maintainers and mentors are active there and can help resolve blockers quickly.

Happy Contributing! 🚀

[github-actions]

✅ PR template check passed!

@arpit2006 this PR is ready for your review. 🚀

[arpit2006]

@Sejusahu011 , Please resolve CI issue

[arpit2006]

CI Issue

[github-actions]

✅ PR template check passed!

@arpit2006 this PR is ready for your review. 🚀

[github-actions]

✅ PR template check passed!

@arpit2006 this PR is ready for your review. 🚀

[github-actions]

✅ PR template check passed!

@arpit2006 this PR is ready for your review. 🚀

[arpit2006]

In backend/app/scanners/entropy.py, the PR author needs to make two specific code changes. You can give them these exact snippets:

1. Expand the ignored_dirs list

They need to add the missing IDE, cache, and build folders to the existing set:

    ignored_dirs = {
        "node_modules", "venv", ".venv", "build", "dist", "target", ".next",
        ".cache", "coverage", "vendor", "__pycache__", ".git",
        # --- Add these new directories below ---
        ".idea", ".vscode", ".tox", ".pytest_cache", ".mypy_cache", 
        ".ruff_cache", "bin", "obj", "out", "tmp"
    }

2. Rewrite the file traversal loop

They need to remove the rglob("*") approach and replace it with os.walk() (making sure to import os at the top of the file).

Change this (Current PR code):

    for file_path in repo_dir.rglob("*"):
        if any(part in ignored_dirs for part in file_path.parts):
            continue

        if not file_path.is_file() or file_path.suffix.lower() in ignored_extensions:
            continue

To this (Optimized code):

    import os

    # ... inside run_entropy function ...

    for root, dirs, files in os.walk(repo_dir):
        # Prune the ignored directories in-place so os.walk skips them entirely
        dirs[:] = [d for d in dirs if d not in ignored_dirs]
        
        for file_name in files:
            file_path = Path(root) / file_name
            
            if file_path.suffix.lower() in ignored_extensions:
                continue

            if file_path.stat().st_size > MAX_FILE_SIZE:
                continue
                
            # ... keep the existing file reading (with open(...)) logic here ...

By assigning to dirs[:], Python will drop the ignored folders from the traversal list, which is what actually prevents the memory crashes and slowdowns on large repositories.

[arpit2006]

@Sejusahu011 , Any Update?