Rebranding ,restructure & Feature Update for BlueQuery Migration Tool (v1.1.0)

[essiebx]

---

Description:

This PR includes a rebranding update and feature enhancements for the BlueQuery Migration Tool. The tool is now fully production-ready, with several key improvements to the overall functionality and user experience:

• Rebranding Update: The project name has been changed to BlueQuery for better alignment with its core functionality.

• UI Enhancements: The new Web UI has been enhanced to support an easy-to-use Migration Wizard for both technical and non-technical users.

• Core Features Added:

  • High-Performance Engine: Optimized for large-scale migrations with chunked extraction, cost estimation, and automatic checkpointing.
  • Reliability Improvements: Added retry logic, dead-letter queue (DLQ), and circuit breaker to ensure robust operation.
  • Data Handling: Low memory footprint (<500MB) for handling "massive" data, with automatic checkpoints every 100,000 rows.
  • Security Enhancements: Integrated rate-limiting, size-limiting, and log masking for better data protection.

• Updated Documentation: The README has been restructured and enhanced with detailed descriptions, new features, and setup instructions.

---

Key Changes:

1. Rebranding:

   • The project is now officially named BlueQuery (formerly BQ2PG).
   • Updated project structure and related files to reflect the new branding.

2. Migration Wizard (Web UI):

   • Added a user-friendly interface to manage ETL workflows (credentials, source/destination config, execution status).

3. Engine Enhancements:

   • Optimized data extraction to minimize memory usage.
   • Implemented cost estimation and dry-run features for better migration planning.

4. Reliability Patterns:

   • Introduced exponential backoff retry policy and dead-letter queue for failed records.
   • Added a circuit breaker to avoid resource exhaustion during migrations.

5. Security:

   • Implemented strict input validation, rate-limiting, and response scrubbing to protect sensitive data.

6. Documentation:

   • Reworked the README with:
     • Detailed installation and setup instructions.
     • Enhanced security and troubleshooting sections.
     • Success metrics for the tool’s performance.

---

Testing:

• Unit Tests: All unit tests have been updated and are passing.
• Manual Tests:
  • Validated the migration process with datasets exceeding 1 million rows.
  • Verified UI responsiveness and stability.
  • Tested error handling for network failures and data-type mismatches.

---

How to Test:

1. Setup:

   • Clone the repository and checkout the dev branch.
   • Run the following commands to install dependencies and set up the environment:

   python3 -m venv .venv
   ./.venv/bin/pip install -r requirements.txt

2. Running the Application:

   • Start the API server with the command:

   nohup ./.venv/bin/python api/server.py > api.log 2>&1 & echo $! > api.pid

   • Access the Web UI at http://localhost:5000.

3. Test Migration:

   • Upload valid Google Cloud credentials (service account key) and configure the BigQuery and PostgreSQL sources.
   • Start a migration task and monitor progress through the UI.

---

Changelog:

• v1.1.0:
  • Rebranding and UI enhancements.
  • Added support for large-scale migrations with improved memory efficiency.
  • Integrated better error handling and retry mechanisms.
  • Enhanced security features, including rate-limiting and log masking.

---

Future Improvements:

• Automated Testing: Integrate additional automated tests for edge cases and large datasets.
• User Roles & Permissions: Implement role-based access control for enterprise environments.

---

Screenshots (if applicable):

• Web UI Screenshot:
  

---

Additional Notes:

• Ensure PostgreSQL is up and running locally before initiating migrations.
• Migration tasks can be paused and resumed at any point using the built-in checkpointing.

---

Checklist:

• Code follows the project’s coding style.
• Documentation has been updated.
• All tests pass.
• PR is based on the latest dev branch.

---

Please review and feel free to provide feedback or suggest further improvements!