Skip to content

data: discover RSS feeds for products (32 found, 354 checked) - #209

Merged
514sid merged 40 commits into
mainfrom
data/rss-discovery
Jul 29, 2026
Merged

data: discover RSS feeds for products (32 found, 354 checked)#209
514sid merged 40 commits into
mainfrom
data/rss-discovery

Conversation

@514sid

@514sid 514sid commented Jul 28, 2026

Copy link
Copy Markdown
Member

What

Swept every active product that had a website but no rss_feed_url (386 in total) and tried to discover a real RSS/Atom feed for each.

  • 32 feeds found and recorded as rss_feed_url.
  • 354 marked with a dated YAML comment # 2026-07-27 - RSS feed not found, so we know they were checked and do not re-scan them.

How feeds were discovered

For each product: fetch the homepage and read any <link rel="alternate" type="application/rss+xml|atom+xml">, then probe common feed paths (/feed/, /rss.xml, /atom.xml, etc.). Every candidate is validated by fetching it and confirming it is actually a feed (contains <rss>, <feed>, <channel>, or <rdf>), not just any XML. First validated hit wins, preferring the declared <link> feed.

Notes

  • The not-found marker is a YAML comment, so it does not render on the product page; it is an internal "checked on this date" marker.
  • A few feeds live on the vendor's parent or company domain rather than the product's own (e.g. Media4Display -> telelogos.com, Switchboard -> coatesgroup.com, frontface -> mirabyte's blog). These are the correct company behind the product and each validated as a real feed.
  • Hit rate was ~8%; RSS feeds are simply uncommon on these vendor sites.

Validation

bun run check passes on all 625 products. hugo --minify builds clean.

514sid added 30 commits July 28, 2026 18:35
@514sid
514sid merged commit ccf7194 into main Jul 29, 2026
3 checks passed
@514sid
514sid deleted the data/rss-discovery branch July 29, 2026 12:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant