You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add PlayerTransfersSpider for scraping transfer data
Description
This pull request introduces a new spider, PlayerTransfersSpider, designed to scrape transfer history data for football players from transfermarkt.co.uk. The spider utilizes Scrapy framework and follows the structure of our BaseSpider class.
Features
Parses player profile pages to extract transfer history API URL
Makes API calls to retrieve detailed transfer data
Extracts key information including transfer dates, clubs involved, market values, and transfer fees
Implementation Details
Uses JSON parsing to handle API responses
Implements two main parsing methods: parse and parse_transfers
Includes Scrapy contracts for testing and documentation purposes
Please review and provide feedback, especially regarding the structure and any potential optimizations.
First of all, thanks a lot for your PR. Transfer history would be a really nice addition to the project, and what you've done here is smart use of scrapy to fetch it. Here's my feedback.
scrapy main purpose is to support scraping data from a website HTML, where you'd need to use css or xpath queries to extract parts of the HTML, follow links that you find there etc. See for example, the games spider.
For extracting data from a REST API I feel it's a bit of an overkill to use scrapy and you're probably better off by simply using a library like requests. I actually did something very similar to this for the marketValue endpoint (you can have a look at the code here) where I just used requests to simply query the endpoint and saved the results in a file. You are very welcome to contribute this functionality there.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add PlayerTransfersSpider for scraping transfer data
Description
This pull request introduces a new spider,
PlayerTransfersSpider, designed to scrape transfer history data for football players from transfermarkt.co.uk. The spider utilizes Scrapy framework and follows the structure of our BaseSpider class.Features
Implementation Details
parseandparse_transfersPlease review and provide feedback, especially regarding the structure and any potential optimizations.