gmail-orders-scan

Scan Gmail for order, invoice, and receipt emails and export them to a semicolon-delimited CSV — useful for tax preparation and expense tracking.

Supports two fetch modes (IMAP and Gmail MCP) and two LLM backends (local Ollama and remote Grok/x.ai).

---

Features

• IMAP mode — connects via Gmail App Password, no Google Cloud setup needed
• MCP mode — connects via Gmail OAuth through the @gongrzhe/server-gmail-autoauth-mcp Node.js server
• LLM extraction — uses a local Ollama model (default gemma3:4b) or Grok (x.ai) to extract structured data from each email
• PDF attachment extraction — downloads PDF invoices attached to emails and extracts text with pdfminer.six
• Parallel parsing — configurable concurrency for LLM calls
• Deduplication — deduplicates by Order ID and vendor before writing CSV

---

Quick Start

# 1. Create and activate virtualenv
python -m venv .venv
source .venv/bin/activate      # Windows: .venv\Scripts\activate

# 2. Install Python dependencies
pip install -r requirements.txt

# 3. Scan via IMAP (easiest — needs a Google App Password)
python gmail_orders_scan.py \
  --mode imap \
  --email you@gmail.com \
  --password "xxxx xxxx xxxx xxxx" \
  --year 2025 \
  --output tax-2025.csv

# 4. Scan via Gmail MCP (requires OAuth setup below)
python gmail_orders_scan.py \
  --mode mcp \
  --year 2025 \
  --output tax-2025.csv

---

Gmail App Password (IMAP mode)

1. Enable 2-Step Verification on your Google Account
2. Go to Google Account → Security → 2-Step Verification → App passwords
3. Create an app password for "Mail"
4. Use the 16-character password as --password

---

Gmail OAuth Setup (MCP mode)

MCP mode uses server-gmail-autoauth-mcp under the hood.

Prerequisites: Node.js >= 18

# Install the MCP Gmail server globally
npm install -g @gongrzhe/server-gmail-autoauth-mcp

Google Cloud credentials:

1. Go to Google Cloud Console → create or select a project
2. Enable the Gmail API under APIs & Services → Library
3. Create an OAuth 2.0 credential: Credentials → Create Credentials → OAuth client ID → Desktop application
4. Download the JSON file and place it:

mkdir -p ~/.gmail-mcp
cp ~/Downloads/client_secret_*.json ~/.gmail-mcp/gcp-oauth.keys.json

5. Run the first-time OAuth flow (opens browser):

npx @gongrzhe/server-gmail-autoauth-mcp

Credentials are cached in ~/.gmail-mcp/ for subsequent runs.

---

LLM Backends

Ollama (default, local)

Requires Ollama running locally.

ollama pull gemma3:4b    # or llama3.1:8b for better quality

python gmail_orders_scan.py --mode mcp --year 2025 --llm-model gemma3:4b --output tax-2025.csv

Grok / x.ai (cloud)

Provide an x.ai API key. Uses the OpenAI-compatible endpoint — no extra package needed.

python gmail_orders_scan.py \
  --mode mcp --year 2025 \
  --grok-api-key "xai-..." \
  --grok-model grok-3-mini \
  --output tax-2025.csv

Or set the key via environment variable:

export XAI_API_KEY="xai-..."
python gmail_orders_scan.py --mode mcp --year 2025 --output tax-2025.csv

Disable LLM (pattern-matching only)

python gmail_orders_scan.py --mode mcp --year 2025 --no-llm --output tax-2025.csv

---

All CLI Options — gmail_orders_scan.py

Option	Default	Description
--mode	imap	imap or mcp
--email		Gmail address (IMAP mode)
--password		Google App Password (IMAP mode)
--year	2025	Year to scan
--output	orders-gmail-YEAR.csv	Output CSV file
--llm-model	gemma3:4b	Ollama model
--grok-api-key		x.ai API key (overrides Ollama)
--grok-model	grok-3-mini	Grok model name
--no-llm		Disable LLM, use pattern-matching only
--concurrency	4	Parallel LLM calls
--npx	npx	Path to npx binary (MCP mode)

---

Other Scripts

amazon_orders.py — Amazon order history via browser

Scrapes your Amazon order history page using Playwright.

pip install playwright && playwright install chromium
python amazon_orders.py --year 2025 --email you@amazon.com
python amazon_orders.py --year 2025 --email you@amazon.com --headless --output amazon-2025.csv

amazon_orders_mbox.py — Amazon emails from mbox

Parses Amazon order/shipment emails from a local .mbox file (e.g. exported via Google Takeout).

python amazon_orders_mbox.py --mbox Amazon2025.mbox/mbox --year 2025 --output amazon-2025.csv

mbox_orders_scan_v2.py — Universal mbox scanner

Scans any mbox archive for order/invoice emails. Lower quality than direct Gmail access but works fully offline.

python mbox_orders_scan_v2.py --mbox Gmail2025.mbox/mbox --year 2025 --output orders-2025.csv

cleanup_orders.py — Post-processing

Normalizes vendor names, drops non-purchase emails (marketing, surveys), and deduplicates records.

python cleanup_orders.py orders-2025.csv orders-2025-clean.csv

simple_imap_server.py — Test IMAP server

Serves an mbox file as a local IMAP server for development and testing.

python simple_imap_server.py Gmail2025.mbox/mbox

---

Output Format

Semicolon-delimited CSV:

Order ID;Order Date;Items;Total (EUR);Vendor;Status

Column	Example
Order ID	302-3472087-4867566
Order Date	2025-03-15
Items	Logitech MX Keys
Total (EUR)	99,00 EUR
Vendor	Amazon
Status	Delivered

---

Dependencies

beautifulsoup4>=4.12.0
lxml>=4.9.0
pdfminer.six>=20231228
mcp>=1.0.0

Optional for Playwright-based Amazon scraping:

playwright

---

License

MIT