[Solution]: Developing an Deep Research Workspace

[kstij1]

1. Solution Overview: Deep Research Workspace

The Deep Research Workspace makes rigorous analysis practical. Powered by AI, it turns open-ended questions into curated source lists, structured notes, and citation-backed reports that fit Weam’s UX. Teams can run multi-step research, refine sections iteratively, and export polished findings—all inside the Weam ecosystem.

Rather than a single-shot summary, this solution enables iterative work. Users curate sources, regenerate sections as understanding evolves, and maintain traceable links back to originals. The end result is a readable, citation-backed report (Markdown/PDF) that teams can share, review, and refine—ideal for market analysis, technical evaluations, or policy briefs.

Plan, gather, and synthesize multi-source research.

Key features:

• Multi-step web research (search, crawl, dedupe, rank)
• Source board with notes and tagging
• Outline builder and report generator
• Export report (Markdown/PDF)
• Seamless Weam auth/session integration

---

2. Tech Stack & Framework Standards

Frontend

• Next.js + Weam theme components

Backend

• Node.js / Express.js for search/crawl/summarize
• Optional Python microservices via /pyapi/

Database

• MongoDB with naming:

  solution_{solutionName}_{tableName}
  

  Example: solution_research_projects

---

3. Database Connection

Local:

MONGODB_URI=mongodb://localhost:27017/weam

Cloud:

DB_HOST=db.weam.ai
DB_USERNAME=yourUser
DB_PASSWORD=yourPass

---

4. Theming & UI Consistency

Use Weam’s design system for layouts, colors, and typography.

---

5. Authentication & Access Control

Require Weam auth; if unauthenticated, return 404. Use ironSession in Next.js.

---

6. Session Management

Persist user and companyId in project documents:

{
  "query": "Impact of LLMs on support ops",
  "user": { "id": "123", "email": "john@domain.com" },
  "companyId": "weam_company_456"
}

---

7. Sidebar Integration

src/seeders/superSolution.json:

{
  "name": "Research",
  "path": "/research",
  "icon": "ResearchIcon",
  "description": "Long-form research with sources and reports"
}

---

8. Deployment & Routing

Base path:

NEXT_PUBLIC_API_BASE_PATH="/research"

Examples:

Page: https://os.weam.ai/research
API:  https://os.weam.ai/research/api/run

---

9. NGINX Configuration

# Deep Research Workspace
location /research/ {
    proxy_pass http://localhost:3013/research/;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

---

10. Final Checklist

✅ Next.js frontend
✅ Node/Express APIs
✅ Mongo naming convention
✅ user + companyId stored
✅ Theming matches Weam
✅ Auth (404 if missing)
✅ ironSession used
✅ Sidebar entry added
✅ Base path env set
✅ NGINX updated

---

11. Example Workflow

1. User opens Research in Weam; enters a research question.
2. System gathers ≥10 sources (metadata + summaries).
3. User curates sources, adds notes, builds outline.
4. Backend generates a report with citations.
5. Export Markdown/PDF; data stored in solution_research_projects.

[aryasoni98]

I’m happy to work on this! Could you please assign the issue to me?

[kstij1]

hi @aryasoni98 i, I would request you to go through the updated description of this and let me know your approach. thanks.

[aryasoni98]

Implementation Approach for Deep Research Workspace

Phase 1: Foundation & Setup (Priority: High)

1.1 Database Schema Design

Create MongoDB collections following the naming convention:

// Collections to create:
solution_research_projects          // Main project metadata
solution_research_sources           // Curated sources
solution_research_notes             // User notes on sources
solution_research_outlines          // Report outlines
solution_research_sections          // Report sections with citations
solution_research_exports           // Export history

Schema Structure:

// solution_research_projects
{
  _id: ObjectId,
  title: String,
  query: String,                    // Original research question
  status: Enum('gathering', 'curating', 'outlining', 'generating', 'completed'),
  user: {
    id: String,
    email: String,
    fname: String,
    lname: String
  },
  companyId: String,
  workspace: { id, name },           // Optional workspace association
  metadata: {
    sourceCount: Number,
    wordCount: Number,
    createdAt: Date,
    updatedAt: Date
  },
  settings: {
    minSources: Number (default: 10),
    maxSources: Number (default: 50),
    citationStyle: String (default: 'numbered')
  }
}

// solution_research_sources
{
  _id: ObjectId,
  projectId: ObjectId (ref),
  url: String,
  title: String,
  summary: String,                   // AI-generated summary
  content: String,                   // Crawled content
  metadata: {
    author: String,
    publishDate: Date,
    domain: String,
    credibilityScore: Number,
    relevanceScore: Number
  },
  tags: [String],
  isCurated: Boolean,
  userNotes: String,
  companyId: String,
  createdAt: Date
}

1.2 Backend Structure (Node.js)

Create following structure in nodejs/src/:

nodejs/src/
├── models/
│   ├── researchProject.js
│   ├── researchSource.js
│   ├── researchNote.js
│   ├── researchOutline.js
│   └── researchSection.js
├── routes/
│   └── web/
│       └── research.js              // All research routes
├── controller/
│   └── web/
│       └── researchController.js
├── services/
│   ├── researchService.js           // Core business logic
│   ├── webSearchService.js          // Search & crawl
│   ├── contentAnalysisService.js    // Summarization, ranking
│   └── reportGeneratorService.js    // Report generation
└── tools/
    └── researchTools.js             // Web scraping, deduplication

---

Phase 2: Backend Implementation (Priority: High)

2.1 Core Services to Build

A. Web Search Service (webSearchService.js)

// Key functions:
- searchSources(query, count)        // Use multiple search APIs
- crawlUrl(url)                       // Extract content
- deduplicateSources(sources)         // Remove duplicates
- validateSource(url)                 // Check accessibility

Recommended APIs:

• SerpAPI / Google Custom Search
• Bing Search API
• Brave Search API (privacy-focused)
• Fallback to Puppeteer/Playwright for scraping

B. Content Analysis Service (contentAnalysisService.js)

// Key functions:
- summarizeContent(content)           // AI summarization
- extractKeyPoints(content)           // Key insights
- calculateRelevance(source, query)   // Scoring
- detectLanguage(content)             // Language detection
- extractMetadata(html)               // Author, date, etc.

C. Report Generator Service (reportGeneratorService.js)

// Key functions:
- generateOutline(sources, query)     // Create structure
- generateSection(outline, sources)   // Write sections
- insertCitations(text, sources)      // Add citations
- exportMarkdown(report)              // Markdown export
- exportPDF(report)                   // PDF generation (use puppeteer)

2.2 API Endpoints

// Projects
POST   /api/research/projects/create
GET    /api/research/projects/list
GET    /api/research/projects/:id
PUT    /api/research/projects/:id
DELETE /api/research/projects/:id

// Sources
POST   /api/research/projects/:id/sources/search     // Trigger search
GET    /api/research/projects/:id/sources
POST   /api/research/projects/:id/sources/add        // Manual add
PUT    /api/research/projects/:id/sources/:sourceId  // Update/tag
DELETE /api/research/projects/:id/sources/:sourceId
POST   /api/research/projects/:id/sources/:sourceId/notes

// Outline & Generation
POST   /api/research/projects/:id/outline/generate
PUT    /api/research/projects/:id/outline
POST   /api/research/projects/:id/report/generate    // Generate report
GET    /api/research/projects/:id/report
POST   /api/research/projects/:id/report/regenerate  // Regenerate section

// Export
GET    /api/research/projects/:id/export/markdown
GET    /api/research/projects/:id/export/pdf

---

Phase 3: Frontend Implementation (Priority: High)

3.1 Page Structure

Create in nextjs/src/app/(page)/research/:

research/
├── page.tsx                         // Main dashboard
├── new/
│   └── page.tsx                     // Create new project
├── [id]/
│   ├── page.tsx                     // Project detail
│   ├── sources/
│   │   └── page.tsx                 // Source board
│   ├── outline/
│   │   └── page.tsx                 // Outline builder
│   └── report/
│       └── page.tsx                 // Report view/export
└── layout.tsx                       // Research layout

3.2 Component Structure

Create in nextjs/src/components/Research/:

Research/
├── ProjectCard.tsx                  // Project list item
├── ProjectList.tsx                  // Dashboard projects
├── CreateProjectModal.tsx           // New project form
├── SourceBoard/
│   ├── SourceCard.tsx               // Individual source
│   ├── SourceList.tsx               // Grid of sources
│   ├── SourceFilter.tsx             // Filter/search
│   └── SourceNoteEditor.tsx         // Notes panel
├── Outline/
│   ├── OutlineBuilder.tsx           // Drag-drop outline
│   ├── OutlineSection.tsx           // Section item
│   └── OutlinePreview.tsx           // Preview
├── Report/
│   ├── ReportViewer.tsx             // Rendered report
│   ├── ReportEditor.tsx             // Edit sections
│   ├── CitationList.tsx             // Citations panel
│   └── ExportOptions.tsx            // Export buttons
└── SearchProgress.tsx               // Loading states

3.3 UI/UX Flow

Step 1: Dashboard

• Show list of projects (cards)
• Filter: All / In Progress / Completed
• Create New button → Modal

Step 2: Create Project

┌─────────────────────────────────────┐
│ Start New Research                  │
├─────────────────────────────────────┤
│ Research Question:                  │
│ [________________________]          │
│                                     │
│ Number of Sources: [10-50]         │
│                                     │
│ [Cancel]  [Start Research]         │
└─────────────────────────────────────┘

Step 3: Source Gathering (Automatic)

• Show progress bar
• Display sources as they're found
• Real-time updates via SSE or WebSocket

Step 4: Source Board

┌──────────────────────────────────────────┐
│  Sources (15)   [Filter] [Sort] [Search] │
├──────────────────────────────────────────┤
│ ┌──────────────┐  ┌──────────────┐      │
│ │ ✓ Source 1   │  │ ✓ Source 2   │      │
│ │ domain.com   │  │ site.org     │      │
│ │ Summary...   │  │ Summary...   │      │
│ │ [View][Tag]  │  │ [View][Tag]  │      │
│ └──────────────┘  └──────────────┘      │
│                                          │
│ [+ Add Manual Source]                    │
│ [Generate Outline →]                     │
└──────────────────────────────────────────┘

Step 5: Outline Builder

• Drag-drop sections
• Edit section titles
• Preview structure
• Regenerate outline option

Step 6: Report Generation

• Generate all sections
• Edit individual sections
• Regenerate specific sections
• Citations automatically added

Step 7: Export

• View rendered report
• Export as Markdown
• Export as PDF

---

Phase 4: Integration & Configuration (Priority: Medium)

4.1 Seeder Configuration

Update nodejs/src/seeders/superSolution.json:

{
  "solutionApps": [
    // ... existing apps ...
    {
      "name": "Deep Research",
      "description": "Plan, gather, and synthesize multi-source research with AI-powered analysis and citation-backed reports",
      "icon": "research-icon",
      "category": "Research & Analysis",
      "isActive": true,
      "pathToOpen": "/research",
      "sequence": 5
    }
  ]
}

4.2 Routes Configuration

Update nextjs/src/utils/routes.ts:

research: '/research',
researchNew: '/research/new',
researchProject: '/research/:id'

4.3 Icon Integration

Create nextjs/src/icons/ResearchIcon.tsx:

export const ResearchIcon = (props) => (
  <svg {...props} viewBox="0 0 24 24">
    {/* Research icon SVG */}
  </svg>
);

Update nextjs/src/utils/iconMapping.ts:

'research-icon': ResearchIcon

4.4 Environment Variables

Add to .env files:

# Research Solution
NEXT_PUBLIC_RESEARCH_BASE_PATH=/research
RESEARCH_PORT=3013

# External APIs
SERP_API_KEY=your_key
OPENAI_API_KEY=your_key  # For summarization

4.5 NGINX Configuration

Add to nginx config:

location /research/ {
    proxy_pass http://localhost:3013/research/;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_read_timeout 300s;  # Long timeout for report generation
}

---

Phase 5: Advanced Features (Priority: Low)

5.1 AI Enhancements

• Smart source ranking algorithm
• Automatic outline generation with LLM
• Section regeneration with context
• Citation style options (APA, MLA, Chicago)

5.2 Collaboration

• Share projects with team members
• Commenting on sources
• Version history of reports

5.3 Export Options

• Multiple PDF templates
• DOCX export
• HTML export
• LaTeX for academic papers

---

Phase 6: Testing & Deployment (Priority: High)

6.1 Testing Strategy

// Unit tests
- Source deduplication
- Relevance scoring
- Citation insertion

// Integration tests
- End-to-end project creation
- Report generation flow
- Export functionality

// Performance tests
- Handle 50+ sources
- Large report generation
- Concurrent users

6.2 Monitoring

• Track search API usage
• Monitor report generation time
• Log failed crawls
• Usage analytics per company

---

[aryasoni98]

@kstij1 Could you please assign the issue to me?