Add national teams acquiring pipeline and dataset assets#353
Merged
Conversation
- Add countries, national_teams assets to the acquiring script and dbt pipeline - Add national_team_players acquisition (reuses players crawler, merges into players.json.gz) - Add 5 national team competitions (World Cup, UEFA Euro, Copa América, AFCON, AFC Asian Cup) - Add international_caps, international_goals, current_national_team_id fields to players - Add GitHub Actions workflow for scheduled national teams acquisition - Update README with new tables and ER diagram Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…on.gz Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…-2022 - Add data/tournament_editions.json with 5 WC editions (2006/2010/2014/2018/2022) - Add acquire_tournament_games() to scraper script, driven by tournament_editions.json - Add tournament_games asset to argparse choices - Fix VARCHAR_CASTS for games: add home_club_position/away_club_position to prevent type mismatch when merging tournament (JSON-inferred) with league (VARCHAR) game data - Add .github/workflows/acquire-tournament-games.yml (workflow_dispatch only) - Update poetry.lock with latest transfermarkt-scraper from add-countries-national-teams Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ed stats Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Changes are now merged into main, no longer need the feature branch. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
countries(200+ rows) andnational_teams(200+ rows) with full dbt base + curated models and testsplayers:international_caps,international_goals,current_national_team_id— populated from national team player scrapesnational_team_competitionincompetitions— covers World Cup, UEFA Euro, Copa América, AFCON, AFC Asian Cupacquire-national-teams.yml— scheduled Tue/Fri at 06:00 UTC, jobs: countries → national_teams → national_team_players → dvc-pushTest plan
dbt build -s base_countries base_national_teams countries national_teams— 20/20 tests passdbt build -s base_players players— 11/11 tests pass (height regex fix for U+201A character,current_club_nametolerance raised to 15% for uncovered-league national team players)dbt build -s base_games games club_games game_events game_lineups— 33/33 tests pass🤖 Generated with Claude Code