feat(scraping): Implement real-time StockX market price scraping

[mrunmayeekokitkar]

Pull Request

Description

Implements real-time StockX market price scraping by replacing placeholder market data with live pricing collected through a headless browser. The scraper includes anti-bot protections, automatic retries with exponential backoff, and database upsert support for reliable market price updates.

Related Issues

Fixes #122

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

Changes

Real-Time StockX Scraper

• Added puppeteer-extra with the stealth plugin for improved anti-bot detection bypass.
• Implemented a new scrapeStockX function using a headless browser.
• Replaced the previous mock/random market data with real StockX pricing.
• Added proper browser lifecycle management to prevent resource leaks.
• Added comprehensive error handling throughout the scraping workflow.

Retry & Resilience

• Added fetchWithRetry helper.
• Implemented exponential backoff retry logic:
  • Up to 3 retry attempts
  • 1-second base delay
• Improves reliability during temporary network failures or scraping issues.

Market Price Updates

• Updated runAllScrapers to use the new real-time StockX scraper.
• Scrapes and stores:
  • Ask Price
  • Bid Price
  • Last Sold Price
• Persists market data using database upsert operations.

Database

Updated prisma/schema.prisma:

• Replaced the existing index with a unique constraint on:

(sku, size, marketplace)

This enables reliable upsert operations without duplicate market price records.

Dependencies

Added the following packages:

• puppeteer
• puppeteer-extra
• puppeteer-extra-plugin-stealth

---

Implementation Summary

app/services/scrapers/index.ts

• Added Puppeteer Extra with the Stealth plugin.
• Implemented the scrapeStockX function using a headless browser.
• Added fetchWithRetry with exponential backoff.
• Updated runAllScrapers to use live StockX data instead of randomly generated values.
• Added error handling using try/catch.
• Ensured browser instances are properly closed after scraping.

prisma/schema.prisma

• Replaced the existing index with a unique constraint on:
  • sku
  • size
  • marketplace

to support database upsert operations.

package.json

Added dependencies:

• puppeteer
• puppeteer-extra
• puppeteer-extra-plugin-stealth

---

Scraper Workflow

The scraper now:

1. Launches a headless browser with stealth mode enabled.
2. Navigates to the StockX search page for the requested SKU.
3. Extracts:
   • Ask Price
   • Bid Price
   • Last Sold Price
4. Stores the market data using database upsert operations.
5. Retries failed requests using exponential backoff.
6. Closes browser instances to prevent memory leaks.

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings

Screenshots (if applicable)

N/A

[vercel]

@mrunmayeekokitkar is attempting to deploy a commit to the participationcorner2025-8967's projects Team on Vercel.

A member of the Team first needs to authorize it.

[github-actions]

Codesense Ai: This PR is too large to review automatically. A human maintainer will take a look!