- Packetpulse - Real Time Delivery Tracking
PacketPulse is a comprehensive real-time delivery tracking platform that simulates, processes, and visualizes package delivery data through a modern data engineering architecture. The system provides end-to-end visibility into delivery operations with real-time analytics and interactive dashboards.
The platform follows a modern data pipeline architecture:
Mars (Simulation) → Venus (FastAPI WebSocket) → Kafka ↔ Neptune (Bronze Layer) / Jupiter (Silver Layer) → Uranus (dbt Gold Layer) → ClickHouse → Mercury (Full Stack Platform)
Data Flow:
- Bronze Layer: Raw data stored in Iceberg tables on MinIO S3
- Silver Layer: Processed data stored in ClickHouse
- Gold Layer: Business logic and aggregations via dbt
- Analytics: SQLPad and Trino for querying MinIO/Iceberg data
Simulates multiple delivery vehicles with real-time GPS tracking, generating realistic delivery scenarios with configurable routes, speeds, and status updates. Sends data via WebSocket to Venus API.
FastAPI-based WebSocket server that receives delivery data from Mars simulators, validates incoming data, and publishes it to Kafka for downstream processing. Handles real-time data ingestion with authentication.
Consumes raw delivery data from Kafka and stores it unchanged in Iceberg tables on MinIO S3. Implements the bronze layer pattern for raw data preservation and basic validation.
Real-time stream processing service using PyFlink that transforms delivery data from Kafka and stores it in ClickHouse. Handles data cleaning, enrichment, and structured storage for analytics.
dbt-based data transformation layer that creates business-ready models and aggregations. Processes data from ClickHouse to generate KPIs, session summaries, and analytical views.
Next.js-based real-time dashboard with interactive maps, analytics, and user authentication. Provides comprehensive delivery tracking visualization and fleet management capabilities.
- Docker and Docker Compose
- Node.js 18+ (for local Mercury development)
- Start all services:
cd Infrastructure
docker-compose up -d- Access the applications:
- Mercury Dashboard: http://localhost:3000 (admin/password)
- SQLPad Analytics: http://localhost:3002 (admin@example.com/password)
- Trino Query Engine: http://localhost:8080
- MinIO Console: http://localhost:9001 (admin/password)
- Start simulation:
docker-compose up marsEach component can be run independently. See individual README files for detailed setup instructions:
- Mars Documentation
- Venus Documentation
- Neptune Documentation
- Jupiter Documentation
- Uranus Documentation
- Mercury Documentation
- Infrastructure Documentation
- Data Processing: Apache Kafka, Apache Flink, dbt
- Storage: ClickHouse, Apache Iceberg, MinIO S3
- Analytics: Trino, SQLPad
- Frontend: Next.js, TypeScript, Material-UI
- Backend: FastAPI, WebSockets
- Infrastructure: Docker, Docker Compose
We welcome contributions to PacketPulse! Here's how you can help:
- Fork the repository and clone it locally
- Create a feature branch:
git checkout -b feature/your-feature-name
- Make your changes and commit them
- Push to your fork and create a pull request
- Search existing issues before creating a new one
- Include details about your environment and steps to reproduce
- Add logs and error messages when applicable
- Describe your changes clearly in the PR description
- Add tests for new functionality
- Update documentation if needed
- Ensure all checks pass before submitting
- Python: Follow PEP 8 standards
- TypeScript: Use ESLint and Prettier
- SQL: Follow dbt style guide
- Documentation: Update README files for new features
- Check the individual component READMEs for detailed setup instructions
- Open an issue for bugs or feature requests
- Ask questions in GitHub Discussions
By contributing, you agree that your contributions will be licensed under the MIT License.