Skip to content

⚡ Performance: Add parallel processing for batch extractions #26

Description

@ZeroSumQuant

Performance Enhancement

Component: extract_claude_logs.py

Feature Request

Implement parallel processing for extracting multiple conversations simultaneously.

Proposed Implementation

def extract_multiple_parallel(self, sessions: List[Path], indices: List[int]) -> Tuple[int, int]:
    """Extract multiple sessions in parallel for better performance"""
    from concurrent.futures import ThreadPoolExecutor, as_completed
    
    success = 0
    total = len(indices)
    
    def extract_single(idx):
        if 0 <= idx < len(sessions):
            session_path = sessions[idx]
            conversation = self.extract_conversation(session_path)
            if conversation:
                output_path = self.save_as_markdown(conversation, session_path.stem)
                return True, len(conversation), output_path.name if output_path else None
        return False, 0, None
    
    with ThreadPoolExecutor(max_workers=4) as executor:
        futures = {executor.submit(extract_single, idx): idx for idx in indices}
        
        for future in as_completed(futures):
            idx = futures[future]
            try:
                is_success, msg_count, filename = future.result()
                if is_success:
                    success += 1
                    print(f"✅ {success}/{total}: {filename} ({msg_count} messages)")
                else:
                    print(f"⏭️  Skipped session {idx + 1} (no conversation)")
            except Exception as e:
                print(f"❌ Error processing session {idx + 1}: {e}")
    
    return success, total

Benefits

  • 3-4x faster batch extraction operations
  • Better CPU utilization
  • Progress reporting during extraction
  • Graceful error handling per file

Use Cases

  • --all flag operations
  • --recent N extractions
  • Batch exports from interactive UI

Priority

MEDIUM - Significant speedup for bulk operations

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions