Website Link Graph Crawler — Extract Every Internal & Outbound Link (No Browser)

▶️ Run Website Link Graph Crawler on Apify

Crawl an entire website and export every internal and outbound link as a graph edge. The Website Link Graph Crawler walks a site from one start URL, follows internal links to discover pages, and records every <a href> it finds as a single row — a directed edge from the page it was found on (sourceUrl) to the URL it points to (targetUrl). Each edge carries the target domain, the visible anchor text, a link-type classification (internal / subdomain / external), the raw rel attribute, and boolean flags for nofollow, sponsored, ugc, and target=_blank. The result is a complete link graph of the site: every internal link, every outbound link, and the anchor text and rel flags attached to each one.

It's built for internal-linking SEO audits, outbound / external link analysis, and site-structure mapping. Because it works purely over HTTP against the server-rendered HTML — no login, no API key, and no headless browser — it is fast and cheap, and a single run can produce thousands of link edges from a modest crawl. Point it at your own site to optimize internal link equity and find orphan pages, or crawl a competitor to see exactly who they link out to and how.

---

What you get

Every row in the output dataset is one link edge (sourceUrl → targetUrl). For each edge you get:

• sourceUrl — the page the link was found on. Group by this to see every link out of a given page.
• targetUrl — the absolute URL the link points to (relative hrefs are resolved to absolute).
• targetDomain — the hostname of the target URL. Group by this to see every external site you link to, ranked by frequency.
• anchorText — the visible link text, trimmed to 200 characters, for anchor-text analysis.
• linkType — classification of the edge: internal (same host), subdomain (same base domain, different host), or external (outbound to another domain).
• rel — the raw rel attribute string, if any (e.g. nofollow sponsored).
• isNofollow / isSponsored / isUgc — boolean rel flags parsed from the rel attribute, so you can filter for nofollow outbound links or find sponsored / ugc links that need attention.
• opensNewTab — true when the link has target="_blank".
• crawledAt — ISO timestamp of when the edge was recorded.

Together these columns let you reconstruct the full internal-linking structure of a site, tally outbound links by domain, and audit rel-attribute hygiene across every page.

Use cases

• Internal-linking SEO audits. Map which pages link to which, with the anchor text used, so you can optimize link equity, strengthen key pages, fix thin internal linking, and spot orphan pages that nothing links to.
• Outbound / external link analysis. Extract every external site your pages link to, see which links are nofollow vs followed, and group by targetDomain to understand where your outbound links (and link equity) go.
• Site-structure mapping. Build a directed link graph of the whole site for visualization or crawl-budget analysis — every edge from source page to target URL in one dataset.
• Link cleanup & compliance. Find sponsored and ugc links that are missing the correct rel value, and audit nofollow usage across paid or user-generated links to stay compliant.
• Anchor-text analysis. Study the distribution of anchor text across the entire site — over-optimized exact-match anchors, empty anchors, or generic "click here" text.
• Competitor link mapping. Crawl a competitor's site to see who they link out to, how much they interlink internally, and which outbound links they mark nofollow or sponsored.

Quick start

You can run the actor with zero configuration — an empty input crawls a default demo site so you can see the output shape immediately. Then point it at any URL.

Apify Console

1. Open the Website Link Graph Crawler actor page.
2. Click Try for free.
3. Leave the input empty to crawl a built-in demo site, or paste a URL into Start URLs.
4. (Optional) Pick a Links to export value — All links, Internal only, or External / outbound only — and a Crawl scope.
5. Click Start, then open the Dataset tab to browse, filter, or export the link edges.

Apify CLI

# Install the Apify CLI and log in
npm i -g apify-cli
apify login

# Run with defaults (empty input crawls the demo site)
apify call logiover/website-link-graph-crawler

Crawl a specific site, cap the pages, and export only outbound links:

apify call logiover/website-link-graph-crawler --input '{
  "startUrls": [{ "url": "https://example.com" }],
  "maxPagesToCrawl": 100,
  "linkScope": "external"
}'

API / cURL

Run the actor and get the link edges back in one synchronous call. Replace YOUR_TOKEN with your Apify API token.

curl -X POST "https://api.apify.com/v2/acts/logiover~website-link-graph-crawler/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{ "url": "https://example.com" }],
    "maxPagesToCrawl": 50,
    "linkScope": "external"
  }'

The response body is a JSON array of link-edge objects — ready to pipe into jq, a file, or a database.

JavaScript & Python (apify-client)

JavaScript (npm install apify-client):

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_TOKEN' });

const run = await client.actor('logiover/website-link-graph-crawler').call({
  startUrls: [{ url: 'https://example.com' }],
  maxPagesToCrawl: 50,
  linkScope: 'external',
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(`Extracted ${items.length} link edges`);
console.log(items[0]);

Python (pip install apify-client):

from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")

run = client.actor("logiover/website-link-graph-crawler").call(run_input={
    "startUrls": [{"url": "https://example.com"}],
    "maxPagesToCrawl": 50,
    "linkScope": "external",
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["sourceUrl"], "->", item["targetUrl"], item["linkType"])

Input

All fields are optional. An empty input crawls a default demo site.

Field	Type	Default	Description
startUrls	array of { url }	demo site	Websites to crawl. The actor follows internal links from each start URL and records every link (edge) it finds.
linkScope	string (enum)	all	Which links to export. Dropdown: All links (all), Internal only (internal), or External / outbound only (external).
crawlScope	string (enum)	same-domain	How far to follow links when discovering pages to visit. Dropdown: Same domain only (same-domain), Include subdomains (include-subdomains), or Single page only (no crawl) (single-page).
maxPagesToCrawl	integer	30	Max pages visited per run (min 0). Each page yields many edges, so the default is modest for a fast, cheap run. Set to 0 for no limit / whole site.
maxConcurrency	integer	10	Number of parallel requests (min 1). Lower it if the target site rate-limits.
proxyConfiguration	object	{ "useApifyProxy": true }	Proxy settings for the crawl.

Tips:

• Use linkScope: "external" for outbound-link analysis, or linkScope: "internal" for internal-linking audits.
• Use crawlScope: "single-page" to extract every link from one page only (no crawling).
• Set maxPagesToCrawl: 0 to map an entire site; keep it low for a quick sample.

Output

One row per link edge.

Field	Type	Description
sourceUrl	string	Page the link was found on.
targetUrl	string	Absolute URL the link points to.
targetDomain	string | null	Hostname of the target URL.
anchorText	string | null	Visible link text (trimmed to 200 chars).
linkType	string	internal (same host), subdomain (same base domain), or external.
rel	string | null	Raw rel attribute, if any.
isNofollow	boolean	true if rel contains nofollow.
isUgc	boolean	true if rel contains ugc.
isSponsored	boolean	true if rel contains sponsored.
opensNewTab	boolean	true if the link has target="_blank".
crawledAt	string	ISO timestamp when the edge was recorded.

The default Links view shows: sourceUrl, targetUrl, anchorText, linkType, isNofollow.

Sample output item (an external, nofollow + sponsored outbound link):

{
  "sourceUrl": "https://example.com/blog/best-tools",
  "targetUrl": "https://partner-site.com/product?ref=example",
  "targetDomain": "partner-site.com",
  "anchorText": "Check out this tool",
  "linkType": "external",
  "rel": "nofollow sponsored",
  "isNofollow": true,
  "isUgc": false,
  "isSponsored": true,
  "opensNewTab": true,
  "crawledAt": "2026-07-12T09:41:03.512Z"
}

Integrations & automation

Run the Website Link Graph Crawler on a schedule to re-map a site every week and diff the results — catch newly added outbound links, links that lost their nofollow, or internal links that disappeared. On the Apify platform you can:

• Schedule recurring runs (daily / weekly) to keep an always-fresh snapshot of a site's link graph.
• Fire webhooks on run completion to push the new edges straight into your own pipeline.
• Export to Google Sheets, Amazon S3, or any datastore from the run's dataset.
• Connect to Zapier, Make, n8n, or Pipedream via the Apify integrations to trigger downstream automations (alerts, dashboards, reports) whenever the link graph changes.

Export formats

Every run stores results in a dataset you can download or stream as:

• CSV
• JSON
• JSONL (newline-delimited JSON)
• Excel (.xlsx)
• XML

FAQ

How do I extract all links from a website?

Enter one or more URLs in Start URLs and run the actor. It follows internal links to discover pages and records every <a href> it finds as a link edge (sourceUrl → targetUrl) with anchor text, link type, and rel flags. Leave linkScope on All links to capture both internal and outbound links.

How do I export links to CSV or JSON?

After a run, open the Dataset tab and choose Export → CSV, JSON, JSONL, Excel, or XML. Via API, call the run-sync-get-dataset-items endpoint to get JSON directly, or request the dataset items with ?format=csv for a CSV download.

Can I get only outbound / external links?

Yes. Set linkScope = External / outbound only (external) and the dataset will contain only links that point to another domain. Group by targetDomain to see every external site you link to, ranked by frequency, and filter on isNofollow to separate followed vs nofollow outbound links.

Does it extract anchor text and nofollow links?

Yes. Every edge includes anchorText (the visible link text) and parsed rel flags: isNofollow, isSponsored, and isUgc, plus the raw rel string and an opensNewTab flag for target="_blank" links.

How is this different from a broken link checker?

A broken link checker requests each link and reports its HTTP status (200 / 404 / redirect). This actor maps the link graph instead — it records every link edge with source, target, anchor text, link type, and rel flags, but it does not check HTTP status codes. That makes it faster and higher-volume for structural, internal-linking, and outbound-link analysis. If you specifically need to find dead links, use the Broken Link Checker.

Does it render JavaScript?

No. It reads the server-rendered HTML over plain HTTP — no headless browser — which is what makes it fast and cheap. Links that exist only after client-side JavaScript execution are not captured.

How many links can I extract per run?

There's no fixed cap on edges — each crawled page typically yields many links, so a run can produce thousands of edges. Control volume with maxPagesToCrawl: keep the default (30) for a quick sample, raise it for deeper coverage, or set it to 0 to crawl the whole site.

How do I audit internal linking or find orphan pages?

Set linkScope = Internal only and crawl the whole site (maxPagesToCrawl: 0, crawlScope: same-domain). Then group the edges by sourceUrl and targetUrl: pages that never appear as a targetUrl are orphan pages (nothing links to them), and the anchorText column shows how each page is linked internally so you can optimize link equity.

Is it free / does it need an API key?

The actor needs no target-site API key or login — it works on public HTML. You run it on the Apify platform under Apify's usage-based pricing (there's a free tier to try it), and you only need your own Apify API token for programmatic access.

Is crawling a website's links legal?

The actor only reads publicly available HTML and follows links the same way a browser or search engine would. As with any crawling, respect each site's Terms of Service and robots.txt, keep maxConcurrency reasonable so you don't overload the target, and only crawl sites you're authorized to analyze.

Related actors

Actor	What it does
Broken Link Checker	Crawls a site and reports broken links with their HTTP status codes.
Website SEO Audit Crawler	On-page SEO audit across a whole site (titles, meta, headings, and more).
Sitemap to URL Crawler	Expands a sitemap into a clean list of URLs to feed other crawlers.

---

📄 Documentation only — the Actor runs on the Apify platform. ▶️ Run it: https://apify.com/logiover/website-link-graph-crawler

MIT © 2026 logiover