Skip to content

Commit 1ce21bc

Browse files
author
Michael
committed
docs: Add data architecture separation + VPS deployment notes
Architecture Changes (2025-10-22): - 3-layer data flow: Supabase (AI) → JSONL (analytics) → Obsidian (human) - Eliminated duplicate work between AI memory and human vault - Auto-rollup system: daily summaries → weekly → monthly - Human effort reduced to 15 min/month (from 2+ hours/week) VPS Deployment Planning: - Direct Supabase client planned (bypassing unreliable MCP) - Migration path documented for $500/month revenue milestone - Cron automation for weekly/monthly rollups - Z840 workstation option documented Updated Files: - DATA-ARCHITECTURE-SEPARATION.md (new, 582 lines) - Soulfield-Development-Workflow.md (added Layer 1/2/3 architecture) - AGENT-VAULT-WORKFLOW-EXPLANATION.md (created earlier) - KILO-CODE-WEB-DEVELOPMENT.md (landing page docs) Philosophy: Make money FIRST, upgrade infrastructure with earnings
1 parent 8d0e480 commit 1ce21bc

3 files changed

Lines changed: 1292 additions & 1 deletion

File tree

workspace/docs/Obsidian/docs/reference/workflows/Soulfield-Development-Workflow.md

Lines changed: 164 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -445,13 +445,173 @@ claude agent lens-validator "Verify doc accuracy"
445445

446446
---
447447

448+
## Data Architecture (2025-10-22)
449+
450+
### Separation: AI Memory vs Human Vault
451+
452+
**Problem Solved**: Eliminated duplicate work between Supabase (AI memory) and Obsidian (human knowledge).
453+
454+
**Architecture**: Three-layer automated rollup system
455+
456+
#### Layer 1: AI Memory (Machine-Readable)
457+
458+
**Storage**: Supabase pgvector (direct access, bypassing MCP)
459+
460+
**Why Direct Access**: MCP can be unreliable; production requires direct Supabase client
461+
462+
**Purpose**:
463+
- Agent memory recall
464+
- Semantic search (milliseconds)
465+
- Complete interaction history
466+
- Vector embeddings for context
467+
468+
**Auto-Storage**: Every agent call writes to Supabase automatically
469+
470+
**Human Access**: Never (machine-only)
471+
472+
**Code Location**: [backend/services/memory/memory-supabase.cjs](backend/services/memory/memory-supabase.cjs)
473+
474+
**Migration Plan (VPS Deployment)**:
475+
```javascript
476+
// CURRENT: Via MCP (unreliable)
477+
await mcpClient.callTool('supabase', 'insertData', { ... });
478+
479+
// FUTURE (VPS): Direct Supabase client
480+
import { createClient } from '@supabase/supabase-js';
481+
const supabase = createClient(SUPABASE_URL, SUPABASE_KEY);
482+
await supabase.from('agent_memories').insert({ ... });
483+
```
484+
485+
**Cost**: $0/month (free tier), scales to $25/month at 500k interactions
486+
487+
---
488+
489+
#### Layer 2: JSONL Archive (Analytics)
490+
491+
**Storage**: `/home/michael/Documents/soulfield-archive/2025-10/session-summaries/`
492+
493+
**Format**: Newline-delimited JSON (append-only)
494+
495+
**Purpose**:
496+
- Audit trail (forever)
497+
- Analytics source data
498+
- Rollup generation input
499+
- Debugging history
500+
501+
**Auto-Storage**: Every agent call appends to daily JSONL file
502+
503+
**Human Access**: Rarely (scripts read for rollups)
504+
505+
**Example Entry**:
506+
```json
507+
{"timestamp":"2025-10-22T14:32:15Z","agent":"marketing","prompt":"Create campaign...","success":true,"lenses":["truth","causality"],"response_length":2450,"files_created":["campaigns/email-q4.md"]}
508+
```
509+
510+
**Code Location**: [backend/services/auto-obsidian-sync.cjs](backend/services/auto-obsidian-sync.cjs)
511+
512+
**Cost**: $0/month (~18MB/year, local storage)
513+
514+
---
515+
516+
#### Layer 3: Obsidian Vault (Human-Readable)
517+
518+
**Storage**: `workspace/docs/Obsidian/`
519+
520+
**Purpose**: Your source of truth, graph view, planning
521+
522+
**Update Frequency**:
523+
- Daily notes: Real-time (lightweight summaries only)
524+
- Weekly rollups: Auto-generated Sunday or manual
525+
- Monthly rollups: Auto-generated end-of-month or manual
526+
- Reference files: Monthly review
527+
528+
**Human Effort**: 15 min/month (reviewing auto-generated rollups)
529+
530+
**Directory Structure**:
531+
```
532+
workspace/docs/Obsidian/
533+
├── daily/ # 3KB summaries (7-day retention → archive)
534+
├── weekly/ # Auto-generated rollups (permanent)
535+
├── monthly/ # Auto-generated rollups (permanent)
536+
└── docs/reference/ # AI-extracted knowledge (monthly updates)
537+
```
538+
539+
**Data Flow**:
540+
```
541+
Agent executes
542+
543+
Supabase + JSONL (auto, 50ms)
544+
545+
Daily note summary (auto, ~3KB)
546+
547+
[Sunday 11:59 PM or manual]
548+
549+
Weekly rollup auto-generated from JSONL
550+
551+
[End of month or manual]
552+
553+
Monthly rollup auto-generated from weeklies
554+
555+
Reference files updated (AI-extracted)
556+
```
557+
558+
**Code Locations**:
559+
- Auto-sync: [backend/services/auto-obsidian-sync.cjs](backend/services/auto-obsidian-sync.cjs)
560+
- Rollup service: [backend/services/rollup-service.cjs](backend/services/rollup-service.cjs)
561+
- Knowledge extraction: [tools/extract-knowledge.cjs](tools/extract-knowledge.cjs)
562+
563+
**Cost**: ~$2/month (Claude API for weekly/monthly summaries)
564+
565+
**Automation Commands**:
566+
```bash
567+
# Weekly rollup (auto via cron or manual)
568+
node backend/services/rollup-service.cjs generateWeeklySummary 2025-W43
569+
570+
# Monthly rollup (auto via cron or manual)
571+
node backend/services/rollup-service.cjs generateMonthlyRollup 2025-10
572+
573+
# Cron setup (VPS deployment):
574+
# Weekly: 0 23 * * 0 (Sunday 11:59 PM)
575+
# Monthly: 0 23 L * * (Last day of month 11:59 PM)
576+
```
577+
578+
**Full Architecture**: [[DATA-ARCHITECTURE-SEPARATION.md]]
579+
580+
---
581+
582+
### VPS Deployment Notes (Future)
583+
584+
**When**: $500/month revenue milestone
585+
586+
**Server Strategy**:
587+
- **Option 1**: VPS ($5-20/month) - DigitalOcean, Linode, Vultr
588+
- **Option 2**: Z840 Workstation (home server, $0/month after hardware)
589+
590+
**Direct Supabase Benefits on VPS**:
591+
- ✅ No MCP reliability issues
592+
- ✅ Connection pooling for performance
593+
- ✅ Transaction support
594+
- ✅ Batch operations (faster)
595+
- ✅ Real-time subscriptions (future feature)
596+
597+
**Migration Path**:
598+
1. Revenue hits $500/month → Purchase VPS or Z840 upgrade
599+
2. Install direct Supabase client (`@supabase/supabase-js`)
600+
3. Replace MCP calls with direct client calls
601+
4. Keep MCP as fallback for local development
602+
5. Set up cron jobs for automated rollups
603+
604+
**Z840 Deployment Checklist**: [[docs/Z840-DEPLOYMENT-CHECKLIST.md]]
605+
606+
---
607+
448608
## MCP Server Integration
449609

450610
Both Claude Code Web and CLI subagents can access MCP servers:
451611

452612
### Available MCP Servers
453613

454-
1. **Supabase** - Database operations, vector memory
614+
1. **Supabase** - Database operations, vector memory (⚠️ unreliable, moving to direct access on VPS)
455615
2. **Google Workspace** - Calendar, Docs, Sheets, Gmail, Drive
456616
3. **Playwright** - Browser automation, testing
457617
4. **Perplexity** - Research, reasoning, search
@@ -462,6 +622,9 @@ Both Claude Code Web and CLI subagents can access MCP servers:
462622

463623
- **Claude Code Web**: Can connect to remote MCP servers
464624
- **CLI Subagents**: Can use MCP if specified in subagent definition
625+
- **Production (VPS)**: Direct Supabase client, MCP for other services
626+
627+
**Note**: Supabase MCP is suitable for development but unreliable for production. VPS deployment will use direct `@supabase/supabase-js` client.
465628

466629
---
467630

0 commit comments

Comments
 (0)