@@ -445,13 +445,173 @@ claude agent lens-validator "Verify doc accuracy"
445445
446446---
447447
448+ ## Data Architecture (2025-10-22)
449+
450+ ### Separation: AI Memory vs Human Vault
451+
452+ ** Problem Solved** : Eliminated duplicate work between Supabase (AI memory) and Obsidian (human knowledge).
453+
454+ ** Architecture** : Three-layer automated rollup system
455+
456+ #### Layer 1: AI Memory (Machine-Readable)
457+
458+ ** Storage** : Supabase pgvector (direct access, bypassing MCP)
459+
460+ ** Why Direct Access** : MCP can be unreliable; production requires direct Supabase client
461+
462+ ** Purpose** :
463+ - Agent memory recall
464+ - Semantic search (milliseconds)
465+ - Complete interaction history
466+ - Vector embeddings for context
467+
468+ ** Auto-Storage** : Every agent call writes to Supabase automatically
469+
470+ ** Human Access** : Never (machine-only)
471+
472+ ** Code Location** : [ backend/services/memory/memory-supabase.cjs] ( backend/services/memory/memory-supabase.cjs )
473+
474+ ** Migration Plan (VPS Deployment)** :
475+ ``` javascript
476+ // CURRENT: Via MCP (unreliable)
477+ await mcpClient .callTool (' supabase' , ' insertData' , { ... });
478+
479+ // FUTURE (VPS): Direct Supabase client
480+ import { createClient } from ' @supabase/supabase-js' ;
481+ const supabase = createClient (SUPABASE_URL , SUPABASE_KEY );
482+ await supabase .from (' agent_memories' ).insert ({ ... });
483+ ```
484+
485+ ** Cost** : $0/month (free tier), scales to $25/month at 500k interactions
486+
487+ ---
488+
489+ #### Layer 2: JSONL Archive (Analytics)
490+
491+ ** Storage** : ` /home/michael/Documents/soulfield-archive/2025-10/session-summaries/ `
492+
493+ ** Format** : Newline-delimited JSON (append-only)
494+
495+ ** Purpose** :
496+ - Audit trail (forever)
497+ - Analytics source data
498+ - Rollup generation input
499+ - Debugging history
500+
501+ ** Auto-Storage** : Every agent call appends to daily JSONL file
502+
503+ ** Human Access** : Rarely (scripts read for rollups)
504+
505+ ** Example Entry** :
506+ ``` json
507+ {"timestamp" :" 2025-10-22T14:32:15Z" ,"agent" :" marketing" ,"prompt" :" Create campaign..." ,"success" :true ,"lenses" :[" truth" ," causality" ],"response_length" :2450 ,"files_created" :[" campaigns/email-q4.md" ]}
508+ ```
509+
510+ ** Code Location** : [ backend/services/auto-obsidian-sync.cjs] ( backend/services/auto-obsidian-sync.cjs )
511+
512+ ** Cost** : $0/month (~ 18MB/year, local storage)
513+
514+ ---
515+
516+ #### Layer 3: Obsidian Vault (Human-Readable)
517+
518+ ** Storage** : ` workspace/docs/Obsidian/ `
519+
520+ ** Purpose** : Your source of truth, graph view, planning
521+
522+ ** Update Frequency** :
523+ - Daily notes: Real-time (lightweight summaries only)
524+ - Weekly rollups: Auto-generated Sunday or manual
525+ - Monthly rollups: Auto-generated end-of-month or manual
526+ - Reference files: Monthly review
527+
528+ ** Human Effort** : 15 min/month (reviewing auto-generated rollups)
529+
530+ ** Directory Structure** :
531+ ```
532+ workspace/docs/Obsidian/
533+ ├── daily/ # 3KB summaries (7-day retention → archive)
534+ ├── weekly/ # Auto-generated rollups (permanent)
535+ ├── monthly/ # Auto-generated rollups (permanent)
536+ └── docs/reference/ # AI-extracted knowledge (monthly updates)
537+ ```
538+
539+ ** Data Flow** :
540+ ```
541+ Agent executes
542+ ↓
543+ Supabase + JSONL (auto, 50ms)
544+ ↓
545+ Daily note summary (auto, ~3KB)
546+ ↓
547+ [Sunday 11:59 PM or manual]
548+ ↓
549+ Weekly rollup auto-generated from JSONL
550+ ↓
551+ [End of month or manual]
552+ ↓
553+ Monthly rollup auto-generated from weeklies
554+ ↓
555+ Reference files updated (AI-extracted)
556+ ```
557+
558+ ** Code Locations** :
559+ - Auto-sync: [ backend/services/auto-obsidian-sync.cjs] ( backend/services/auto-obsidian-sync.cjs )
560+ - Rollup service: [ backend/services/rollup-service.cjs] ( backend/services/rollup-service.cjs )
561+ - Knowledge extraction: [ tools/extract-knowledge.cjs] ( tools/extract-knowledge.cjs )
562+
563+ ** Cost** : ~ $2/month (Claude API for weekly/monthly summaries)
564+
565+ ** Automation Commands** :
566+ ``` bash
567+ # Weekly rollup (auto via cron or manual)
568+ node backend/services/rollup-service.cjs generateWeeklySummary 2025-W43
569+
570+ # Monthly rollup (auto via cron or manual)
571+ node backend/services/rollup-service.cjs generateMonthlyRollup 2025-10
572+
573+ # Cron setup (VPS deployment):
574+ # Weekly: 0 23 * * 0 (Sunday 11:59 PM)
575+ # Monthly: 0 23 L * * (Last day of month 11:59 PM)
576+ ```
577+
578+ ** Full Architecture** : [[ DATA-ARCHITECTURE-SEPARATION.md]]
579+
580+ ---
581+
582+ ### VPS Deployment Notes (Future)
583+
584+ ** When** : $500/month revenue milestone
585+
586+ ** Server Strategy** :
587+ - ** Option 1** : VPS ($5-20/month) - DigitalOcean, Linode, Vultr
588+ - ** Option 2** : Z840 Workstation (home server, $0/month after hardware)
589+
590+ ** Direct Supabase Benefits on VPS** :
591+ - ✅ No MCP reliability issues
592+ - ✅ Connection pooling for performance
593+ - ✅ Transaction support
594+ - ✅ Batch operations (faster)
595+ - ✅ Real-time subscriptions (future feature)
596+
597+ ** Migration Path** :
598+ 1 . Revenue hits $500/month → Purchase VPS or Z840 upgrade
599+ 2 . Install direct Supabase client (` @supabase/supabase-js ` )
600+ 3 . Replace MCP calls with direct client calls
601+ 4 . Keep MCP as fallback for local development
602+ 5 . Set up cron jobs for automated rollups
603+
604+ ** Z840 Deployment Checklist** : [[ docs/Z840-DEPLOYMENT-CHECKLIST.md]]
605+
606+ ---
607+
448608## MCP Server Integration
449609
450610Both Claude Code Web and CLI subagents can access MCP servers:
451611
452612### Available MCP Servers
453613
454- 1 . ** Supabase** - Database operations, vector memory
614+ 1 . ** Supabase** - Database operations, vector memory (⚠️ unreliable, moving to direct access on VPS)
4556152 . ** Google Workspace** - Calendar, Docs, Sheets, Gmail, Drive
4566163 . ** Playwright** - Browser automation, testing
4576174 . ** Perplexity** - Research, reasoning, search
@@ -462,6 +622,9 @@ Both Claude Code Web and CLI subagents can access MCP servers:
462622
463623- ** Claude Code Web** : Can connect to remote MCP servers
464624- ** CLI Subagents** : Can use MCP if specified in subagent definition
625+ - ** Production (VPS)** : Direct Supabase client, MCP for other services
626+
627+ ** Note** : Supabase MCP is suitable for development but unreliable for production. VPS deployment will use direct ` @supabase/supabase-js ` client.
465628
466629---
467630
0 commit comments