Skip to content

feat: add transfers crawler#101

Open
cesarpmorais wants to merge 1 commit into
dcaribou:mainfrom
cesarpmorais:transfer-crawler
Open

feat: add transfers crawler#101
cesarpmorais wants to merge 1 commit into
dcaribou:mainfrom
cesarpmorais:transfer-crawler

Conversation

@cesarpmorais

Copy link
Copy Markdown

Transfers Crawler

This PR adds a transfers crawler that scrapes player arrivals and departures from Transfermarkt club and competition transfer pages.

About #89 — the original approach proposed using Scrapy, which was lateconsidered overkill for this use case by the maintainer. This implementation uses Crawlee instead, matching the framework adopted by the project in #97.

Usage

# scrape transfers for a competition
echo '{"href":"/premier-league/transfers/wettbewerb/GB1/plus/?saison_id=2025&s_w=&leihe=1&intern=0&intern=1"}' \
    | python -m tfmkt transfers > transfers.json

# scrape transfers for a single club
echo '{"href":"/ec-cruzeiro-belo-horizonte/transfers/verein/609/saison_id/2025"}' \
    | python -m tfmkt transfers > transfers.json

# chain with the clubs crawler
cat competitions.json | head -1 \
    | python -m tfmkt clubs \
    | python -m tfmkt transfers > transfers.json

@dcaribou

Copy link
Copy Markdown
Owner

Hey @cesarpmorais, thanks for your PR!

One question. What'd be the advantage of scraping this data from the html vs querying the public transfers API like here for example?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants