x-timeline-scraper

Scrape the recent timeline of any public X (Twitter) account into a JSONL file for offline analysis. Uses Twikit — no developer API key required; you authenticate as a normal logged-in user once, and the session is reused via cookies.json.

The repo ships five small scripts forming a complete pipeline:

Scrapers

Script	What it captures
scrape_timeline.py	Top-level tweets only (timeline view)
scrape_timeline_with_replies.py	Tweets + replies via the Replies endpoint (catches thread continuations)
scrape_timeline_complete.py	Both endpoints, deduped, sorted newest-first — recommended for full corpus

Post-processors

Script	Purpose
organize_threads.py	Stitches an account's self-replies back into reconstructed threads, with [1/N] numbering.
extract_tweet_text.py	Strips a JSONL down to clean plain text (one tweet per paragraph) for LLM / corpus / embedding use.

Setup

python -m venv .venv
# Windows PowerShell:
. .venv/Scripts/Activate.ps1
# macOS/Linux:
source .venv/bin/activate

pip install -r requirements.txt

Credentials

Copy .env.example to .env and fill in your X login (or export the vars into your shell). These are your login credentials — they're used by Twikit to perform a real login on first run. The session cookie is then written to cookies.json and reused on later runs.

X_USERNAME=your_handle
X_EMAIL=login_email@example.com
X_PASSWORD=your_password

PowerShell:

$env:X_USERNAME = "your_handle"
$env:X_EMAIL    = "login_email@example.com"
$env:X_PASSWORD = "your_password"

.env, cookies.json, and the data/ directory are all gitignored. Never commit them.

Usage

1. Scrape

# Last 365 days of top-level tweets:
python scrape_timeline.py SomeAccount

# Custom window + custom output path:
python scrape_timeline.py SomeAccount --days 90 --out data/sample.jsonl

# Tweets + replies (preserves threads):
python scrape_timeline_with_replies.py SomeAccount --days 365

# Full corpus (recommended):
python scrape_timeline_complete.py SomeAccount --days 365

Default output paths:

• data/<handle>_tweets_<days>d.jsonl
• data/<handle>_tweets_replies_<days>d.jsonl
• data/<handle>_complete_<days>d.jsonl

2. Reconstruct threads (optional)

If you scraped with replies, group self-replies into proper threads:

python organize_threads.py data/someaccount_complete_365d.jsonl

Produces:

• data/someaccount_complete_365d_organized.jsonl — structured records
• data/someaccount_complete_365d_threads_text.txt — readable text with [1/N] numbering

You can override paths with --out-jsonl and --out-text.

3. Extract clean plain text (optional)

For corpus / LLM / embedding use:

python extract_tweet_text.py data/someaccount_tweets_365d.jsonl

Produces data/someaccount_tweets_365d_clean.txt — one tweet per paragraph, retweets and replies filtered out, URLs and RT @ prefixes stripped, short fragments dropped (default --min-len 10).

Note: extract_tweet_text.py is designed for the tweets-only output of scrape_timeline.py. If you feed it _with_replies or _complete output, every reply will be filtered out — use organize_threads.py first if you want thread text.

Output format

Scraper JSONL — one JSON object per line

{
  "id": "1234567890",
  "created_at": "Wed Mar 12 14:22:00 +0000 2026",
  "text": "...",
  "lang": "en",
  "is_retweet": false,
  "in_reply_to_status_id": null,
  "type": "tweet"
}

type ("tweet" or "reply") is only present in scrape_timeline_complete.py output. is_reply is only present in scrape_timeline_with_replies.py output.

organize_threads.py JSONL

{"type": "standalone", "tweet": {...}}
{"type": "thread", "thread_id": "...", "tweet_count": 14, "tweets": [...]}

Threads are sorted oldest-first within (posting order); the file as a whole is sorted newest-first by thread root.

Notes & caveats

• Respect platform Terms of Service and rate limits. The scrapers sleep 1.5s between pages by default. If you hit limits, increase the sleep or reduce the --days window.
• Login challenges. If X serves you a captcha or 2FA prompt, log into the account once in a browser, then re-run. Twikit will reuse the session.
• Dependency security. twikit is a third-party library that automates the X web client. Use a dedicated/burner account if possible — accounts doing heavy scraping have been suspended in the past.
• Only public data. These scripts only access tweets that are publicly visible to the logged-in user.
• Thread reconstruction is author-only. organize_threads.py only detects threads where the author replies to themselves. Conversations with other users are not stitched.

License

MIT — see LICENSE.