About

Lurker planned as complex server app to parse steam data and store into DB inspired by SteamDB and SteamSpy. Project was created in educational purposes. Lurker is pretty shallow now, but I have vision to work on.

Architecture

• PICS-driven update pipeline - proposed SteamKit listener, PostgreSQL queue, and C++ worker architecture
• Online player-count scheduler - polling gates, compacted history, and the PostgreSQL/ClickHouse validation plan
• Tag and follower history - PICS-triggered tag snapshots, follower polling, and ClickHouse change points
• Online scheduler simulator - standalone million-app scheduling simulation
• Steam app timeline example - interactive example combining online, review candles, event markers, followers, and price

Current stage

• Common libs
  • cURLing - cURL returning buffer with page data
  • Logger - simple class to store logs to DB
  • Postgre - DB connection holder (/Credentials/credentials.cpp reqiured)
• Common apps
  • InitAllGames - Get gamelist feom steam web api and insert appid and game title to DB
  • UpdateGameInfo - Get list of games in DB (after InitAllGames) and push to DB info from steam web api

Dependencies

vcpkg install curl

vcpkg install libpqxx

vcpkg install rapidjson

Working on:

• After initial UpdateGameInfo complete - make upper-analytics for efficiency decision
• Learn SteamKit and try to realise usefull methods in Lurker
• Pricing for each region
• Tags for games
• Live Online
• Comments for games

Diagnostics

Use the standalone online probe to measure current-player endpoint behaviour from the machine that will run Lurker:

node tools/online-probe.mjs --app-list=appids.csv --limit=500 --rps=8 --concurrency=8

Set --limit=0 only for an intentionally paced full app-list pass. The probe does not retry failures and stops after three HTTP 429 responses by default so that the measurement does not hide or continuously hammer a rate limit. It also stops after ten transport-level failures because Steam may throttle by refusing connections instead of returning an HTTP status. Without an input file, it can use the current paginated IStoreService/GetAppList when STEAM_WEB_API_KEY is set. The legacy ISteamApps/GetAppList/v2 endpoint used by the prototype has been deprecated and currently returns HTTP 404.

Longterms

• Somithing like SteamKit listner to Steam changes. Need to research steam callbacs and
• Web interface for "Steam Pulse" (upper level analytics of platform e.g. games/year, games/languages e.t.c)
• Steamworks csv analytics (Steam traffic, Pricing, wishlists/cohort)

Known issues

• Language incorrect regex, so bad DB results.
• too many 429. Proxy holder needed or SteamKitResearch.