⚡ Bolt: [performance improvement] Optimize nested loops in material extraction and RAG indexing#135
⚡ Bolt: [performance improvement] Optimize nested loops in material extraction and RAG indexing#135glacy wants to merge 1 commit into
Conversation
…dict lookups - Convert nested O(N*M) list searches in `get_all_exercises` and `index_materials` to O(1) hash map lookups. - Preserves the first-match logic of the original `break` statement by validating `if key not in dict`. - Cleans up unused variables in `rag_indexer.py` and `material_extractor.py` caught by Ruff. Co-authored-by: glacy <1131951+glacy@users.noreply.github.com>
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
There was a problem hiding this comment.
Pull request overview
Este PR busca mejorar el rendimiento al eliminar búsquedas O(N*M) al asociar ejercicios con sus soluciones durante la extracción de materiales y el indexado RAG, reemplazándolas por lookups O(1) mediante diccionarios precomputados, preservando la semántica de “primera coincidencia”.
Changes:
- Optimiza el matching ejercicio→solución en
MaterialExtractor.get_all_exercisesusando unsolution_mappor material. - Optimiza el matching ejercicio→solución en
RAGIndexer.index_materialsusando unsolution_mappor material. - Reformat/normaliza estilo (comillas, wrapping, etc.) en los módulos afectados.
Reviewed changes
Copilot reviewed 2 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
tests/test_exercise_analyzer.py |
Sin cambios funcionales relevantes; principalmente normalización de diff/format. |
evolutia/rag/rag_indexer.py |
Introduce solution_map para evitar nested loops en indexación; incluye refactors/formatting. |
evolutia/material_extractor.py |
Introduce solution_map para evitar nested loops al consolidar ejercicios; incluye refactors/formatting. |
Comments suppressed due to low confidence (1)
evolutia/material_extractor.py:336
- La validación de caché usa
_last_scan_timestamp(global) en lugar del timestamp del entry para este archivo, por lo que puede devolver datos obsoletos si un archivo cambia conmtimemenor al de otro archivo escaneado. Además, el acceso_= self._file_cache[file_path]no aporta nada. Conviene validar pormtimedel propio archivo y, si elstat()falla, usar el TTL para mantener el caché de errores y evitar reintentos constantes.
if file_path not in self._file_cache:
return False
# Verificar si el archivo fue modificado
try:
_ = self._file_cache[file_path]
file_mtime = file_path.stat().st_mtime
# Usar el timestamp de escaneo más reciente para verificar
if file_mtime > self._last_scan_timestamp:
return False
return True
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # Generar embeddings | ||
| embeddings = self._generate_embeddings_batch(chunks) | ||
|
|
||
| # Sincronizar chunks con embeddings (por si se filtraron vacíos en _generate_embeddings_batch) | ||
| # Aunque aquí preferimos filtrar antes para mantener consistencia | ||
| valid_indices = [i for i, chunk in enumerate(chunks) if chunk and chunk.strip()] | ||
| chunks = [chunks[i] for i in valid_indices] | ||
|
|
| # Generar embeddings | ||
| embeddings = self._generate_embeddings_batch(chunks) | ||
|
|
||
| # Sincronizar chunks con embeddings | ||
| valid_indices = [i for i, chunk in enumerate(chunks) if chunk and chunk.strip()] | ||
| chunks = [chunks[i] for i in valid_indices] | ||
|
|
💡 What:
Converted O(N*M) nested loop searches in
evolutia/material_extractor.py(get_all_exercises) andevolutia/rag/rag_indexer.py(index_materials) into O(N) hash map lookups by pre-computingsolution_mapdictionaries.🎯 Why:
When iterating over exercises and repeatedly searching through a list of solutions for a matching label, the time complexity scales quadratically (O(N*M)). For repositories with large amounts of exercises and solutions, this becomes a noticeable bottleneck during extraction and RAG indexing.
📊 Impact:
Reduces the time complexity of exercise-to-solution matching from quadratic O(N*M) to linear O(N), significantly speeding up extraction and index generation pipelines.
🔬 Measurement:
To verify the improvement, generate materials or index using a large subset of exercises. Performance profiles before and after will show a stark decrease in cumulative execution time within
get_all_exercisesandindex_materialsfunctions. Testing verified the exact same output generation correctly preserving first-match semantics.PR created automatically by Jules for task 14618264829651679475 started by @glacy