Refactor: centralize configuration and convert to JUnit testing#4
Merged
Conversation
Major Changes: - Centralized all dependency versions in gradle/libs.versions.toml - Converted TripsTableExample to JUnit test (TripsTableTest) - Replaced System.out.println with SLF4J logging - Externalized MinIO configuration with environment variable support - Created reusable HudiTestSteps for given/when/then test pattern - Resolved SLF4J multiple bindings warning Configuration Centralization: - All dependency versions now in libs.versions.toml (Spark, Hadoop, Hudi, etc.) - MinIO configuration centralized with env var overrides - Java version centralized across all modules - Created SECRETS_MIGRATION.md documenting future Git secrets strategy Testing Improvements: - Converted example to proper JUnit test structure - Added custom assertions support in HudiTestSteps.then() - Improved logging throughout test execution - Tests pass successfully with centralized configuration Next Steps: - Add test coverage checks (starting at 5%, goal: 95%) - Implement JaCoCo for coverage reporting - Set up coverage thresholds in CI/CD
- Fixed all indentation issues to use consistent 4-space indentation - Removed unused import (datahub.core.metadata.Field) - All checkstyle checks now pass cleanly
- Tests need hudi-common classes at runtime to initialize HudiType - Added testImplementation(libs.hudi.common) to hudi/build.gradle.kts - All 13 hudi tests now pass successfully
- Run MinIO as a GitHub Actions service for integration tests - Add early Docker image pull verification with detailed error messages - Detect CI environment and skip local Docker tasks when running in CI - Add comprehensive health checks and wait logic for MinIO readiness - Create MinIO bucket in CI before running tests - Set MinIO environment variables for test execution - Add emoji indicators and clear error messages for debugging This allows the example module tests to run in CI without requiring local Docker setup, while maintaining backward compatibility for local development.
- Start MinIO as a Docker container (not service) with proper server command - Add early Docker image pull verification with detailed error messages - Detect CI environment and skip local Docker tasks when running in CI - Add comprehensive health checks and wait logic for MinIO readiness - Create MinIO bucket in CI before running tests - Set MinIO environment variables for test execution - Add emoji indicators and clear error messages for debugging - Ensure MinIO container cleanup with 'if: always()' step This allows the example module tests to run in CI without requiring local Docker setup, while maintaining backward compatibility for local development. Fixed: MinIO now starts with 'server /data' command instead of failing as a GitHub Actions service which doesn't support custom commands.
rkwagner
force-pushed
the
refactor/centralize-config-and-testing
branch
from
258afe5 to
a852ca8
Compare
- Split CI into parallel jobs (checkstyle + build/test run concurrently) - Enable Gradle build cache in gradle.properties - Add --parallel and --build-cache flags to checkstyle execution - This should reduce checkstyle time from ~2m37s significantly Benefits: - Checkstyle runs in parallel with build/test - Gradle executes checkstyle tasks across modules in parallel - Build cache speeds up subsequent runs - Overall CI time should be dominated by the slower job (likely build/test)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Take this with a grain of salt, it's AI generated.
Major Changes:
Configuration Centralization:
Testing Improvements:
Next Steps: