feat(scraping): Implement Real-Time Market Prices (StockX/GOAT)

[rushikesh-bobade]

Problem Statement

Currently, the runAllScrapers function in app/services/scrapers/index.ts is just a stub generating random fake prices. To provide real value to users (and compete with apps like Pricely), FlipTrack needs to fetch real-time market data so resellers know the actual market value of their inventory.

Proposed Solution

Implement a robust scraping architecture or integrate an unofficial API to fetch live ask/bid prices for sneakers and streetwear based on SKU. This data must be stored in the database and updated periodically.

Acceptance Criteria

1. Scraping Engine: Implement a Puppeteer/Playwright scraper OR integrate a reliable unofficial API for StockX or GOAT in app/services/scrapers/index.ts.
2. Data Model: Ensure the MarketPrice Prisma model accurately stores askPrice, bidPrice, and lastSold.
3. Resilience: Add try/catch blocks and exponential backoff to handle rate limits from the target marketplaces.
4. UI Integration: Ensure the market-prices.tsx route accurately displays this newly scraped data.

Technical Hints

• Focus on StockX first, as it is the most common pricing benchmark.
• Use a headless browser pool if scraping to avoid memory leaks.
• Ensure the cron job triggers this scraper function securely.

[mrunmayeekokitkar]

Hi! I would like to work on this. Can you please assign this to me?

[github-actions]

Hi @mrunmayeekokitkar! 👋

We love the enthusiasm! However, to keep things fair and manageable, our policy is that contributors may only take on one active issue at a time.

It looks like you currently have unfinished work (#93). Please focus on completing that one first!

Once your current work is merged, if this issue is still open, you are more than welcome to claim it. Happy coding! 🚀

[mrunmayeekokitkar]

#93 has been merged, can i work this now?

[rushikesh-bobade]

Hi @mrunmayeekokitkar,

Yes, absolutely! Since your OCR feature is merged, you are all clear. Consider this issue officially assigned to you!

A quick technical heads-up before you start:
Scraping StockX and GOAT is notoriously difficult because they use intense anti-bot protection (Datadome/Cloudflare/PerimeterX).

• Standard fetch or axios requests will get blocked immediately with a 403 Forbidden.
• If you decide to build a scraper, I highly recommend looking into puppeteer-extra-plugin-stealth or Playwright to bypass the initial bot checks.
• Alternatively, if you can find a reliable, unofficial API wrapper on GitHub or NPM, feel free to integrate that instead!

Focus on getting the data model (MarketPrice) set up first, and build a basic integration. Don't worry if you get rate-limited or blocked locally—we can figure out proxy rotation later.

Let me know if you run into any roadblocks. Good luck!