Skip to content

README.md

Latest commit

 

History

History
346 lines (242 loc) · 21.4 KB

README.md

File metadata and controls

346 lines (242 loc) · 21.4 KB
Raw

Facebook Post Scraper

Facebook Post Scraper

Facebook Post Scraper for extracting post captions, authors, images, permalinks and post IDs from Facebook.com. This repo has a free Facebook post web scraping script you can run right now, and a Facebook post data API that turns a post permalink into structured JSON. Other public Facebook surfaces have their own reference in the Facebook Scraper repo.

Last updated: 2026-07-20. Working against Facebook.com as of July 2026, and re-verified whenever Meta changes their markup.

This repo reads posts published by public brand, business and organisation Pages, and collects no personal data. Every capture, screenshot and example here targets a post published by an institution on its own public Page (NASA, NASA's Johnson Space Center, the Washington Nationals). The endpoint returns the publishing Page's own creative: caption, media, author and permalink. It has no surface that returns commenters, reactors, taggees or any individual's name or photo, and none appears in any sample or image committed here.

Every JSON block on this page was captured from the live API on 2026-07-20. Long CDN image URLs are truncated where marked and each block says exactly what was cut; every field shown is verbatim. Full uncut samples are committed in facebook_post_scraper_api_data/, so you can diff this page against them. Every code example calls the actual API and is runnable from facebook_post_scraper_api_codes/.

pip install requests
export CHOCODATA_API_KEY="your_key"     # free: 1,000 requests, one-time, no card
python facebook_post_scraper_api_codes/post.py "https://www.facebook.com/NASA/posts/1426466738848698/"

Those three lines return this, live from Facebook.com:

{
  "post_id": "1426466738848698",
  "page": "NASA",
  "author": "NASA - National Aeronautics and Space Administration",
  "caption": "NASA is planning on rolling out the Artemis II rocket and spacecraft from the Vehicle Assembly Building to Launch Pad 39B at NASA's Kennedy Space Center no earlier than Jan. 17. The four-mile journey...",
  "reactions_count": null,
  "data_source": "opengraph"
}

...that is 6 of the 16 fields on a post; the full 16 are below. Point it at a list of permalinks and you get a row each:

Retrieved Facebook post data

That is the whole point of this repo. The rest of this page is how it works, starting with the free Facebook post scraper: one HTTP request, no key, real post data.

Contents

Free Facebook Post Scraper

Facebook server-renders a public post's caption, author and media into the OpenGraph <meta> tags of the permalink, before any JavaScript runs. So you can extract post data without a headless browser, JavaScript rendering, or a login. No key, no cost:

python free_scraper/facebook_post_free_scraper.py
python free_scraper/facebook_post_free_scraper.py "https://www.facebook.com/NASAJSC/posts/1011509817757722/"

Source: free_scraper/facebook_post_free_scraper.py. It reads og:title, og:description, og:image and og:url, parses the page and post id out of the permalink, and emits post_id, page, url, author, caption, image. It parses first and only reports "no data" when the OpenGraph block is genuinely absent, so a post that renders is never miscounted as a block.

After running the command, your terminal should look something like this:

The free scraper on two permalinks, one of which has no post in it

Both of those are real runs. The first is 325 KB of HTML in and one caption out. The second is a different permalink that came back with no post in it, and the next section is what is actually happening there.

Avoid getting blocked when scraping Facebook posts

Two things make this surface awkward, and neither of them is a 403.

The User-Agent that looks most legitimate is the one that gets refused. A full desktop Chrome string sent from a plain HTTP client came back 400 with a 1,542-byte page titled Error, on 6 of 6 requests across a post permalink and a page URL. curl/8.4.0, python-requests/2.34.2 and this repo's own self-describing agent each got the real document on 6 of 6 in the same run:

Facebook served four different User-Agents

Reproduce it in about a minute:

python free_scraper/ua_test.py 3

That is why the free scraper here sends its own name, and why swapping in a browser string you copied from your own devtools will make things worse rather than better.

The second problem is that a refusal looks like a success, which is the second run in the screenshot above. Facebook answers 200 either way and varies what it puts in the body. The NASA permalink returned 325,335 bytes with the full og block; another public news Page's permalink returned a 303,972 to 305,308 byte document titled Facebook with zero og: tags, on 3 of 3 requests. Same status code, same client, same minute. Nothing about the response line tells you which one you got, which is why the script parses first and reports on the payload rather than on the status.

Here is what genuinely bites you, none of which a proxy fixes:

What bites you Why What it costs you
A copied browser User-Agent is refused A desktop Chrome string got 400 and 1,542 bytes on 6 of 6 requests, while curl, python-requests and a self-describing agent got the real document on 6 of 6. The one change most people make first is the one that breaks it, and a 400 sends you hunting for a bug in your own query string.
HTTP 200 is not proof of data The empty variant is a ~304 KB page titled Facebook with no og: block. It is not a 404, a 403 or a redirect. If your pipeline trusts the status code, you will store nulls and never find out.
Engagement counts are not in the document Reactions, comments and shares load over an authenticated call after the page renders, so they are in none of the logged-out documents we parsed. If a logged-out scraper hands you reaction counts, they did not come out of this page. Check where they did come from before you trust them.
caption is truncated by Facebook, not by you The og description stops at roughly 200 characters and Facebook appends its own .... Three of the four posts committed here came back at 202, 201 and 193 characters. Long-form posts arrive clipped. You are storing the opening, not the copy.
Signed media URLs expire image carries Facebook's own oh signature and oe expiry parameters. The link in a row you stored last month is already dead. Fetch the bytes if you need to keep the creative.
The markup moves The og shape changes a few times a year. Your regex silently returns None. Ongoing maintenance, plus alerting smart enough to tell "empty" from "broken".

Using the Chocodata Facebook Post Scraper API

The managed option. The Chocodata Facebook Post Scraper API turns a post permalink into parsed JSON instead of ~325 KB of HTML, with a ~99% success rate, a pinned egress location so two runs are comparable, and the User-Agent and document-shape handling above absorbed for you. Across 20 calls to the four posts committed in this repo on 2026-07-20 it returned a parsed post with caption, author, url and image on 20 of 20. Four different permalink shapes for the same post, interleaved six times each, all returned the same post: the slug-bearing URL, the bare numeric URL with and without a trailing slash, and page + post_id. Free for the first 1,000 requests.

The API earns its money when the parser stops being something you want to own: when you need the response already parsed and shaped, the null-handling below already mapped, and someone else on the hook the next time Meta moves the markup.

Facebook Post Scraper API reference

Below is the Facebook Post Scraper API reference to get you started: one endpoint, its parameters, a runnable call, and the response it returns.

Quickstart

curl "https://api.chocodata.com/api/v1/facebook/post?api_key=YOUR_KEY&url=https://www.facebook.com/NASA/posts/1426466738848698/"
import requests

r = requests.get(
    "https://api.chocodata.com/api/v1/facebook/post",
    params={"api_key": "YOUR_KEY", "page": "NASA", "post_id": "1426466738848698"},
    timeout=90,
)
r.raise_for_status()          # a 502 means no public post document came back
post = r.json()
print(post["author"], "|", len(post["caption"]), "chars")
# NASA - National Aeronautics and Space Administration | 202 chars

caption, author, url and image come back on every call. page_id and the three engagement counts do not, which the field notes cover.

After running the command, your terminal should look something like this:

Running the Facebook Post Scraper API post endpoint

Get a key at chocodata.com (1,000 requests, one-time, no card).

Authentication

Pass api_key as a query parameter on every request. That is the whole auth model.

Global parameters

Param Type Required Default Description
api_key string yes - Your Chocodata API key.
country string (ISO-2) no us Egress location. Must be exactly 2 characters: usa returns a 400. Facebook localises what it renders to the location it sees, so pin this for reproducible results.

Each request costs 5 credits (= 1 request). Responses are billed only on success (2xx).

Errors

Real captured error bodies, not paraphrases. Nothing below is billed: you are only charged on a 2xx.

Status error code Meaning Billed What to do
400 invalid_params A required param is missing or the wrong type. Body lists the exact issue under issues. no Fix the query string.
401 INVALID_API_KEY Key missing, unrecognised, or revoked. no Check api_key. Get one at chocodata.com.
402 INSUFFICIENT_CREDITS Balance exhausted. no Top up ($0.90 / 1,000 requests, never expires) or upgrade.
429 RATE_LIMITED Over your plan's concurrency. no Back off and retry; see Rate limits.
502 extraction_failed No public post document came back for this permalink. no Retry once with a few seconds' gap.

Two response shapes exist: auth and billing errors nest under error.code (uppercase), while scrape-layer errors are flat with a lowercase error string. All four below are committed in errors.json.

There is no 404. A post id that does not resolve returns 502 extraction_failed, which we saw on both of the two invented ids we tried against a real Page. A 502 says no public post document came back, and nothing more: do not read a deleted post into it.

The scripts in this repo map every status in the table above onto an actionable message, so a typo'd key does not hand you a stack trace:

Facebook Post Scraper API error handling

A bad key, verbatim:

curl "https://api.chocodata.com/api/v1/facebook/post?api_key=totally_invalid_key_123&url=https://www.facebook.com/NASA/posts/1426466738848698/"
{"error": {"code": "INVALID_API_KEY", "message": "Api key not recognised."}}

A call with no identifying param, verbatim. Note it names the exact requirement rather than returning a generic 400:

{
  "error": "invalid_params",
  "issues": [
    {
      "code": "custom",
      "message": "url, or page + post_id, is required",
      "path": []
    }
  ]
}

And a country that is not an ISO-2 code, verbatim, which is the one most people hit:

{
  "error": "invalid_params",
  "issues": [
    {
      "code": "too_big",
      "maximum": 2,
      "type": "string",
      "inclusive": true,
      "exact": true,
      "message": "String must contain exactly 2 character(s)",
      "path": ["country"]
    }
  ]
}

Rate limits and concurrency

Two limits apply. A rate limit of 120 requests per 60 seconds per key (a sustained 2 requests/second), and a concurrency cap on how many requests you may have in flight at once.

Plan Concurrent requests
Free 10
Vibe 30
Pro 50
Custom 100 to 500+

Exceed it and you get 429, not a queue. The endpoint is a synchronous GET: there is no webhook, callback, or async job to poll. A request typically takes ~1.5s (see Measured latency), and the examples use timeout=90 for headroom.

Sizing: at Pro (50 concurrent) and a ~1.5s median call, one worker pool sustains roughly 50 / 1.5 = 33 requests/second. Fan out with a thread pool:

from concurrent.futures import ThreadPoolExecutor
import requests

def one(url):
    r = requests.get("https://api.chocodata.com/api/v1/facebook/post",
                     params={"api_key": KEY, "url": url}, timeout=90)
    return url, (r.json() if r.ok else None)

with ThreadPoolExecutor(max_workers=10) as pool:   # <= your plan's concurrency
    for url, post in pool.map(one, permalinks):
        print(post["author"], post["post_id"] if post else "no public document")

1. Post: captions, authors, images and permalinks

One public post published by a brand, business or organisation Page, by permalink or by page + post_id. Post-level metadata only: no commenters, no reactors, no personal data.

Param Type Required Default Description
url string (URL) one of url / page+post_id - Full post permalink. The slug-bearing form and the bare /posts/<id> form both work, with or without a trailing slash.
page string with post_id - Page vanity or numeric id (e.g. NASA).
post_id string with page - The post / story id (e.g. 1426466738848698).
country string (ISO-2) no us Egress location. Exactly 2 characters. 
curl "https://api.chocodata.com/api/v1/facebook/post?api_key=YOUR_KEY&url=https://www.facebook.com/NASA/posts/1426466738848698/"

Real response. Complete object, all 16 fields verbatim; the two CDN image URLs are truncated where marked and caption carries Facebook's own trailing ... (full sample):

{
  "id": "1426466738848698",
  "page": "NASA",
  "page_id": null,
  "post_id": "1426466738848698",
  "url": "https://www.facebook.com/NASA/posts/nasa-is-planning-on-rolling-out-the-artemis-ii-rocket-and-spacecraft-from-the-ve/1426466738848698/",
  "author": "NASA - National Aeronautics and Space Administration",
  "author_url": "https://www.facebook.com/NASA/",
  "caption": "NASA is planning on rolling out the Artemis II rocket and spacecraft from the Vehicle Assembly Building to Launch Pad 39B at NASA's Kennedy Space Center no earlier than Jan. 17. The four-mile journey...",
  "title": "NASA is planning on rolling out the Artemis II rocket and spacecraft from the Vehicle Assembly Building to Launch Pad 39B at NASA's Kennedy Space Center no earlier than Jan. 17. The four-mile journey...",
  "image": "https://scontent-bos5-1.xx.fbcdn.net/v/t39.30808-6/613581536_1426466702182035_6281715881025376...",
  "thumbnail": "https://scontent-bos5-1.xx.fbcdn.net/v/t39.30808-6/613581536_1426466702182035_6281715881025376...",
  "reactions_count": null,
  "comments_count": null,
  "shares_count": null,
  "source": "facebook",
  "data_source": "opengraph"
}

caption is the field most people come for: the post copy as the Page published it, straight from the OpenGraph block. data_source is the one to gate on, and it is doing more work than it looks like.

Field coverage across 20 calls to the four committed posts

Read the field behaviour, it is load-bearing:

  • data_source tells you what kind of document was parsed, and the endpoint does not validate the host. Hand it any URL and it answers 200: https://example.com comes back with author: "Example Domain", data_source: "title-fallback" and page, post_id and caption all null, because the parser fell back to the <title> when it found no og block. Both samples are committed side by side (post.json, post-non-facebook-url.json). Gate on data_source == "opengraph" and a non-null post_id before you store a row. is_facebook_post() in post.py is that check; copy it.

    The same endpoint answers 200 for a URL that is not a post

  • reactions_count, comments_count and shares_count are null on every post. They were null on all 20 calls across four posts. Facebook serves post engagement over an authenticated call, so it is not in the logged-out document, and we return null rather than invent a number. What this endpoint gives you is the creative: copy, media, author, permalink.

  • page_id is null here, on all 20 calls, even though page resolves. The vanity is what you get on this surface. If you need the stable numeric Page id, that lives on the Page surface, not the post one.

  • caption is capped by Facebook at roughly 200 characters with a trailing ... that Facebook adds itself. Three of the four committed posts came back at 202, 201 and 193 characters, and each was identical on all five calls to it. A short post arrives complete: the Nationals post is 55 characters with no ellipsis. The cap is the source's, not ours.

  • title is a copy of caption, not a separate headline. When the og block is missing entirely it becomes a truncated snippet from the <title> instead, which is what data_source: "title-fallback" is telling you.

  • author is the Page's display name, and it can be long. NASA's is NASA - National Aeronautics and Space Administration, not NASA. author_url is built from the vanity in the permalink, so key your own storage on post_id.

  • image and thumbnail are the same URL. It carries Facebook's own oh signature and oe expiry parameters, so it stops working: fetch the bytes if you need to keep the creative.

Runnable: facebook_post_scraper_api_codes/post.py

Archive post copy and creative from a list of permalinks

Keeping a brand's published copy after the signed media URLs expire is the main reason people scrape Facebook posts, so that use case is in the repo end to end rather than as a snippet. post_archive.py takes any number of permalinks, stores every observation as a local dataset in SQLite, and tells you which captions changed since the last run:

python facebook_post_scraper_api_codes/post_archive.py \
    "https://www.facebook.com/NASA/posts/1426466738848698/" \
    "https://www.facebook.com/NASAJSC/posts/1011509817757722/"

Archiving Facebook post copy and creative

That is a real first run: nothing to compare against yet, so it seeds the database and says so. Export it with one command:

sqlite3 -header -csv facebook_posts.db "select * from observations" > posts.csv

Two details in that script are worth copying. It refuses to store anything whose data_source is not opengraph, so a mistyped URL that answers 200 never enters the dataset as a post. And it keeps every observation rather than overwriting, because a caption edited after publication is exactly the thing you wanted the archive for, and you cannot detect it if you only keep the latest value.

What it stores is the caption text and the media URL, not the media itself. The URL is signed and expiring, so if the creative matters, fetch and store the bytes on the same pass.

Measured latency

Real end-to-end wall-clock, measured from a laptop against the live API on 2026-07-20. This includes the upstream fetch, the anti-bot handling, and the parse:

Endpoint Median Range 2xx n
/facebook/post 1.5s 1.1 to 2.2s 20/20 20

That is one laptop on one afternoon against four permalinks, and n=20 is too small a sample to say anything about the ~99% platform benchmark either way. Reproduce it with the scripts in this repo.

License

MIT. See LICENSE.