A robust, ultra-fast asynchronous data extraction engine for the National Examinations Council of Tanzania (NECTA) CSEE (O-Level) results.
Engineered strictly for high scalability, it seamlessly processes datasets ranging from hundreds of thousands to millions of student indices across varying years—outputting structured CSV data with zero memory bottlenecks.
This package uses advanced networking strategies to dramatically minimize HTTP footprint and maximize execution speed:
NECTA publishes one holistic HTML page per school containing all its students. Instead of querying the database for every single student (e.g. 1 million index numbers = 1 million requests), this engine intelligently groups the students by school. Result: Millions of student lookups across ~4,000 schools generate exactly ~4,000 HTTP requests.
The scraper utilizes an intelligent two-pass pipeline targeting both the official (onlinesys.necta.go.tz) and archive (maktaba.tetea.org) servers.
- Pass 1: Sweeps the primary endpoints mapping over 99% of students securely.
- Pass 2 (The Fallback): Sweeps the exact opposite secondary mirror only for the specific subset of students that were corrupted, rate-limited, or entirely missing from the first pass. Result: Absolutely minimal redundant requests.
- Auto-Resume: Power cut at 85%? Restart with the
--resumeflag. The system auto-verifies your CSV and picks up on the exact missing indices instantly without duplicating HTTP fetches. - Smart Backoff: Safely handles HTTP
429 Too Many Requestsor server500downtime by scaling thread timeouts and dynamically parsingRetry-Afterheaders—immune to global crashes. - Memory Safety: Processed rows are flushed sequentially. A $5 remote server can run billions of rows without triggering an Out-Of-Memory (OOM) killer.
# Clone the repository
git clone https://github.com/issahakimu/necta-scraper.git
cd necta-scraper
# Ensure you have a virtual environment (Optional but Recommended)
python3 -m venv venv
source venv/bin/activate
# Install dependencies
pip install -r requirements.txtFeed standard student index numbers to the engine. Both slash format (S1832/0036/2019) and dot format (S1832.0036.2019) are natively supported.
# Basic Usage — Automatically detects your index column
python main.py students.csv
# Resume a stopped/crashed run without losing progress
python main.py students.csv --resume
# Specify exact output file and specific index column
python main.py students.csv -o processed.csv --column indexnumber
# JSON Output — Collapse 9 subjects into a single compact JSON column
python main.py students.csv --subjects-as-json
# Network Tuning — Max concurrent HTTP pipelines (Default is safely 20)
python main.py students.csv --concurrency 35At the end of processing, you will receive newly appended columns such as status, gender, division, points, school_name, and individual parsed passing grades specifically sorted for standard Tanzanian curriculum requirements (e.g. CIV, HIST, GEO).
Included in this repository is an isolated post-processing algorithm capable of strictly verifying NECTA grades against complex college admission matrices.
Features:
- Filters out non-qualifying Religious/Islamic subject points automatically.
- Maps A-D grades against passing requirement gates.
- Outputs individual
Yes/Noboolean columns per course.
# Feed the output CSV directly to the qualifier mapping engine
python qualify.py results.csv -o results_with_courses.csvCurrently Checks For:
- Ordinary Diploma in Clinical Medicine
- Ordinary Diploma in Community Development
- Ordinary Diploma in Diagnostic Radiography
- Ordinary Diploma in Environmental Health Sciences
- Ordinary Diploma in Health Records and Information Technology
- Ordinary Diploma in Medical Laboratory Sciences
- Ordinary Diploma in Pharmaceutical Sciences
- Ordinary Diploma in Social Work
Limits have been natively disabled. Future NECTA results are seamlessly integrated out-of-the-box dynamically shifting API schemas based strictly around positive chronological boundaries. (2015 to Infinity).