Add countries and national teams crawlers#99
Merged
Conversation
Introduce a new parallel pipeline branch from confederations: confederations → countries → national_teams → players → appearances The countries crawler discovers countries from confederation pages (deduped by country_id). The national_teams crawler visits country pages, finds the senior national team, and scrapes team details (squad size, market value, FIFA ranking, confederation, coach, etc.). The players crawler now also accepts national_team parents via a fallback table selector. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…rally - Add /wettbewerbe/fifa as a 5th confederation so World Cup and global qualifiers are discovered alongside confederation-level competitions - Extract national team competitions (Euro, Nations League, AFCON, Copa América, WC qualifiers, etc.) from the headerless div.box blocks on each confederation page; deduplicated by href, emitted on page 1 only - Skip country-page visits for the FIFA rankings page (not a league listing) - Add defensive guards on country row iteration in competitions.py - Remove per-team competition extraction from national_teams.py (was causing the same competition to appear once per participating team) - Add samples/national_team_competitions.json with 130 competitions - Update README with dual-hierarchy diagram, crawler table, and usage examples for national team workflows - Update tests: confederations count 4→5, competitions asserts national team items present, new test_competitions_fifa Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The parse handler previously crashed on None footer link text and had no fallback for competitions (like UEFA Euro) whose pages don't use the standard "All fixtures & results" footer link. Add a gesamtspielplan href fallback and a test confirming Euro 2024 scrapes correctly via season=2023 (Transfermarkt's convention for that tournament). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
5 tasks
Scrapes the /erfolge/ page to get all historical editions of a tournament competition, emitting year, season (saison_id), winner, winner image, and victory coach per edition. The season field maps directly to the --season argument needed by the games crawler (e.g. season=2021 for Qatar 2022). Also adds test_tournament_editions (World Cup) and test_games_world_cup (Qatar 2022, season=2021), and bumps crawlee to ^1.6.0 to pick up the parsel extra. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
countriescrawler — discovers all countries from confederation pages, outputting country items with metadata (country_id, country_name, code, total_clubs, total_players, market values)national_teamscrawler — takes country parents and scrapes the senior national team per country (squad size, market value, coach, ranking)playerscrawler to acceptnational_teamparents (national team squad pages use the same URL pattern as clubs)competitionssupport for national team competitions (World Cup, Euros, etc.) — these are now emitted alongside domestic competitions from the confederation pagegamescrawler crash on None footer link and add fallback gesamtspielplan link discovery for tournament competitions (UEFA Euro, World Cup)tournament_editionscrawler — takes any cup competition and scrapes all historical editions from the/erfolge/page (year, season id, winner, coach)saison_id = tournament_year - 1(e.g. Qatar 2022 →--season 2021, Euro 2024 →--season 2023)New pipeline branches:
Test plan
test_confederations— 5 confederations including FIFAtest_competitions— domestic + national team competitions from Afrika confederationtest_competitions_fifa— World Cup and qualifier competitions from FIFA confederationtest_clubs— clubs from Croatian 1.HNLtest_players— players from HNK Sibeniktest_appearances— appearances for Ayoze Pereztest_games— DFL Supercup (domestic super cup, 1 game)test_games_euro— UEFA Euro 2024 (season=2023), asserts ≥ 48 gamestest_games_world_cup— FIFA World Cup Qatar 2022 (season=2021), asserts ≥ 60 gamestest_game_lineups— DFL Supercup 2024 lineupstest_countries— countries from Afrika confederationtest_national_teams— Wales national teamtest_national_team_players— players from Wales national team squadtest_tournament_editions— all historical World Cup editions (≥ 20), verifies 2022 edition hasseason="2021"andwinner="Argentina"🤖 Generated with Claude Code