A Java processing engine designed to aggregate and analyze large-scale datasets of news articles. The system utilizes advanced concurrency patterns to handle intensive I/O operations and complex data filtering in parallel.
This project implements a staged pipeline where multiple worker threads collaborate to parse, clean, and categorize data, significantly reducing execution time compared to serial processing.
The engine follows a multithreaded architecture built for maximum CPU utilization:
- Workload Partitioning: Uses a start/end index formula to ensure every thread processes an equal number of input files.
-
Staged Execution: Uses a
$CyclicBarrier$ to synchronize threads between the ingestion, processing, and final reporting phases. - Memory-Efficient Ingestion: Threads operate on local data buffers to minimize synchronization overhead and prevent global lock contention.
-
Race Condition Prevention: Resolved initial synchronization errors by transitioning from standard collections to
$ConcurrentHashMap$ and$AtomicInteger$ for global statistics. - Lock Contention Optimization: Minimized bottlenecks by using a staged parallel architecture, allowing threads to work on local buffers before synchronized global aggregation.
- Testing Hardware: Apple M1 (8 cores), 8GB RAM.
- Speedup Achieved: Up to 2.16x on 4 performance cores, reducing processing time for ~8,300 articles from 12.5s to 5.7s.
-
Efficiency Analysis: Identified the optimal thread count (
$P=4$ ) for the M1 architecture to balance speed and resource utilization.
cd src
make
# Usage: java NewsAggregator <thread_count> <input_articles_path> <config_files>
java NewsAggregator 4 your/path/to/the/articles your/path/to/the/config/file
./test/result_validator.sh