Getting around the 50,000 listing limit #65
|
I built something similar but I ran into issues around page 50. After some investigating I found that Ebay won't show the requester any more than 10,000 listings, even through the browser. How did you get around this roadblock? I was able to get around the captcha issue with proxies but I can't find anything related to getting around the listing limit. I thought using listing date as an 'anchor' and then picking up and moving the date every 10,000 listings could work, but I can't find any search parameter for listing date! Any help offered is much appreciated. |
Replies: 1 comment 7 replies
|
There is no way to filter by date on ebay, however what own can do is filter by price. Suppose you have 20,000 listings, and 10,000 are below $100 and 10,000 are above $100. You can do a search for $0-$100 and then a second for $100.01-$999999 and you'll get everything. Rather than try to figure that out on my own for each listing, if you look at my code I basically do a binary search, splitting my input ranges in half until the results within that range are less than 800, then I iterate over those ranges I determined. |
|
Thanks, that helped a lot. |
|
Unfortunately there's no way around that. Ebay only shows the last 90 days when you search sold listings. Terapeaks shows more but my tool can't scrape that. You could either try to scrape Terapeaks, which is very much against the ToS, mock up some data, or live with just 90 days of data. What are you trying to do with the data if I can ask? |
|
There's a lot of conversation surrounding the link between crypto currency and the price of graphics cards. I wanted to look at historic selling prices, ideally over the last few years to capture both 'peaks' of crypto and compare to the prices of graphics cards. These cards are used for mining crypto, and many would paint them as the main factor responsible for the massive spike in selling price. |
|
Ah yes, I'm actually doing some investigation into that right now. I can show graphically that that's basically the case, see below. You can pretty much see that with the exception of the 3080 Ti and 3090, GPU prices on eBay tend to all sell for about the same price/hashrate, further it seems that an increase in in ETH price leads to an increase in eBay sale price, though a decrease in ETH doesn't necessarily lead to a decrease in eBay sale price. At the moment I'm working on cleaning up these plots (they need it), and also determining some more statistically significant methods of proving correlation other than looking at graphs. Current two metrics are autocorrelation and granger causality
|
|
How did you manage to get sale data as far back as Dec 2020? Have you just been gathering data over the last few months? |
There is no way to filter by date on ebay, however what own can do is filter by price. Suppose you have 20,000 listings, and 10,000 are below $100 and 10,000 are above $100. You can do a search for $0-$100 and then a second for $100.01-$999999 and you'll get everything. Rather than try to figure that out on my own for each listing, if you look at my code I basically do a binary search, splitting my input ranges in half until the results within that range are less than 800, then I iterate over those ranges I determined.