[WIP] Fix core background jobs and media processing issues

[Copilot]

Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan and make progress.

Original prompt

Phase 1: Fix Core Background Jobs & Media Processing

Critical Issues to Fix

1. Background Scheduler Reliability

File: site/background-scheduler.js

Current issues:

• Jobs can fail silently
• No proper error recovery
• Missing job execution tracking

Fix:

• Add try-catch around all job executions
• Implement exponential backoff for retries (3 attempts: 30s, 2m, 5m)
• Add job execution history to database
• Prevent duplicate job runs with locking mechanism
• Add graceful shutdown handling
• Log all job starts, completions, and errors

2. Fix Gallery Downloads (403 Errors)

File: site/lib/media-downloader.ts

Current issue:

Failed to process gallery: Error [AxiosError]: Request failed with status code 403

Fix:

• Update Reddit API calls to use proper authentication
• Add Reddit session cookies to gallery requests
• Implement fallback methods:
  1. Try Reddit API with session cookies
  2. Try old.reddit.com JSON endpoint
  3. Try scraping as last resort
• Don't crash on failure - mark post as READY with external URL
• Add proper error logging (not verbose spam)

3. Fix Video Downloads (yt-dlp)

File: site/lib/media-downloader.ts

Current issues:

• Response from the Engine was empty errors
• Reddit videos failing to download
• Cookie file issues

Fix:

• Update yt-dlp command with better Reddit support:

  yt-dlp -f "bestvideo[height<=720]+bestaudio/best[height<=720]/best" \
    --merge-output-format mp4 \
    --cookies <cookie-file> \
    --add-header "Referer:https://www.reddit.com/" \
    --add-header "User-Agent:Mozilla/5.0..." \
    --extractor-args "reddit:player_only=False" \
    -o <output-file> \
    <reddit-post-url>

• Ensure cookie file exists before use
• Implement fallback download methods:
  1. Try yt-dlp with post URL
  2. Try direct DASH manifest download
  3. Try video-only download (no audio)
  4. Mark as failed if all methods fail
• Add timeout (max 5 minutes per video)
• Clean up temporary cookie files

4. Fix Prisma Connection Errors

File: site/lib/db/prisma.ts

Current issue:

Response from the Engine was empty

Fix:

• Implement proper connection pooling
• Add connection timeout settings
• Retry failed queries (3 attempts)
• Close connections properly in error handlers
• Add global error handler for Prisma errors

5. Improve Job Control API

File: site/app/api/admin/jobs/route.ts

Add:

• Endpoint to start/stop individual jobs
• Endpoint to manually trigger job execution
• Endpoint to get job execution history
• Endpoint to retry failed jobs
• Return proper status codes and error messages

6. Fix Auto Process Posts Job

File: site/background-scheduler.js or relevant cron file

Current issue: Posts stay in NEW status forever

Fix:

• Run every 2 minutes (not 30 seconds)
• Process 10 NEW posts at a time
• For each post:
  • Download media (with retries)
  • Generate thumbnails
  • Update status to READY if successful
  • Update status to FAILED after 3 failed attempts
• Skip posts already being processed (use locking)
• Don't block on individual failures
• Log success/failure metrics

7. Fix Auto Publish Posts Job

File: site/background-scheduler.js or relevant cron file

Fix:

• Run every 1 minute
• Publish 5 READY posts at a time
• Update publishedAt timestamp
• Update status to PUBLISHED
• Clear any stale locks

8. Enhance Error Logging

All files:

• Reduce log verbosity for expected errors (403, timeouts)
• Log ERROR only for unexpected failures
• Log INFO for normal operations
• Add structured logging with context:

  logger.error({
    job: 'auto-process',
    postId: post.id,
    error: error.message,
    attempt: attemptNumber
  }, 'Failed to process post')

9. Add Job Monitoring

File: site/app/admin/jobs/page.tsx

Enhance admin UI:

• Show real-time job status (use polling every 5 seconds)
• Display last run time for each job
• Display next run time for each job
• Show success/failure metrics
• Display recent errors
• Add manual trigger buttons
• Add "View Logs" button for each job

10. Database Optimizations

File: site/prisma/schema.prisma

Add indexes:

model Post {
  // ... existing fields
  
  @@index([status, createdAt])
  @@index([subreddit, publishedAt])
}

model Asset {
  // ... existing fields
  
  @@index([postId, type])
}

Then run:

npx prisma db push

---

Testing Requirements

After implementation, verify:

1. Background scheduler starts automatically with PM2
2. Jobs can be started/stopped from admin panel
3. Gallery posts download successfully (or fail gracefully)
4. Videos download successfully (or fail gracefully)
5. NEW posts are processed to READY within 5 minutes
6. READY posts are published to homepage within 2 minutes
7. No Prisma connection errors in logs
8. Error logs are clean and actionable
9. Admin panel shows correct job status
10. Manual trigger buttons work

---

###...

This pull request was created as a result of the following prompt from Copilot chat.

Phase 1: Fix Core Background Jobs & Media Processing

Critical Issues to Fix

1. Background Scheduler Reliability

File: site/background-scheduler.js

Current issues:

• Jobs can fail silently
• No proper error recovery
• Missing job execution tracking

Fix:

• Add try-catch around all job executions
• Implement exponential backoff for retries (3 attempts: 30s, 2m, 5m)
• Add job execution history to database
• Prevent duplicate job runs with locking mechanism
• Add graceful shutdown handling
• Log all job starts, completions, and errors

2. Fix Gallery Downloads (403 Errors)

File: site/lib/media-downloader.ts

Current issue:

Failed to process gallery: Error [AxiosError]: Request failed with status code 403

Fix:

• Update Reddit API calls to use proper authentication
• Add Reddit session cookies to gallery requests
• Implement fallback methods:
  1. Try Reddit API with session cookies
  2. Try old.reddit.com JSON endpoint
  3. Try scraping as last resort
• Don't crash on failure - mark post as READY with external URL
• Add proper error logging (not verbose spam)

3. Fix Video Downloads (yt-dlp)

File: site/lib/media-downloader.ts

Current issues:

• Response from the Engine was empty errors
• Reddit videos failing to download
• Cookie file issues

Fix:

• Update yt-dlp command with better Reddit support:

  yt-dlp -f "bestvideo[height<=720]+bestaudio/best[height<=720]/best" \
    --merge-output-format mp4 \
    --cookies <cookie-file> \
    --add-header "Referer:https://www.reddit.com/" \
    --add-header "User-Agent:Mozilla/5.0..." \
    --extractor-args "reddit:player_only=False" \
    -o <output-file> \
    <reddit-post-url>

• Ensure cookie file exists before use
• Implement fallback download methods:
  1. Try yt-dlp with post URL
  2. Try direct DASH manifest download
  3. Try video-only download (no audio)
  4. Mark as failed if all methods fail
• Add timeout (max 5 minutes per video)
• Clean up temporary cookie files

4. Fix Prisma Connection Errors

File: site/lib/db/prisma.ts

Current issue:

Response from the Engine was empty

Fix:

• Implement proper connection pooling
• Add connection timeout settings
• Retry failed queries (3 attempts)
• Close connections properly in error handlers
• Add global error handler for Prisma errors

5. Improve Job Control API

File: site/app/api/admin/jobs/route.ts

Add:

• Endpoint to start/stop individual jobs
• Endpoint to manually trigger job execution
• Endpoint to get job execution history
• Endpoint to retry failed jobs
• Return proper status codes and error messages

6. Fix Auto Process Posts Job

File: site/background-scheduler.js or relevant cron file

Current issue: Posts stay in NEW status forever

Fix:

• Run every 2 minutes (not 30 seconds)
• Process 10 NEW posts at a time
• For each post:
  • Download media (with retries)
  • Generate thumbnails
  • Update status to READY if successful
  • Update status to FAILED after 3 failed attempts
• Skip posts already being processed (use locking)
• Don't block on individual failures
• Log success/failure metrics

7. Fix Auto Publish Posts Job

File: site/background-scheduler.js or relevant cron file

Fix:

• Run every 1 minute
• Publish 5 READY posts at a time
• Update publishedAt timestamp
• Update status to PUBLISHED
• Clear any stale locks

8. Enhance Error Logging

All files:

• Reduce log verbosity for expected errors (403, timeouts)
• Log ERROR only for unexpected failures
• Log INFO for normal operations
• Add structured logging with context:

  logger.error({
    job: 'auto-process',
    postId: post.id,
    error: error.message,
    attempt: attemptNumber
  }, 'Failed to process post')

9. Add Job Monitoring

File: site/app/admin/jobs/page.tsx

Enhance admin UI:

• Show real-time job status (use polling every 5 seconds)
• Display last run time for each job
• Display next run time for each job
• Show success/failure metrics
• Display recent errors
• Add manual trigger buttons
• Add "View Logs" button for each job

10. Database Optimizations

File: site/prisma/schema.prisma

Add indexes:

model Post {
  // ... existing fields
  
  @@index([status, createdAt])
  @@index([subreddit, publishedAt])
}

model Asset {
  // ... existing fields
  
  @@index([postId, type])
}

Then run:

npx prisma db push

---

Testing Requirements

After implementation, verify:

1. Background scheduler starts automatically with PM2
2. Jobs can be started/stopped from admin panel
3. Gallery posts download successfully (or fail gracefully)
4. Videos download successfully (or fail gracefully)
5. NEW posts are processed to READY within 5 minutes
6. READY posts are published to homepage within 2 minutes
7. No Prisma connection errors in logs
8. Error logs are clean and actionable
9. Admin panel shows correct job status
10. Manual trigger buttons work

---

Expected Results

After Phase 1:

• ✅ All background jobs working reliably
• ✅ Gallery downloads working or failing gracefully
• ✅ Video downloads working with fallbacks
• ✅ No Prisma errors
• ✅ Clean error logs
• ✅ Admin panel shows accurate status
• ✅ Posts flow from NEW → READY → PUBLISHED automatically
• ✅ System can run 24/7 without manual intervention

---

Implementation Notes

• Don't break existing functionality
• Keep all changes backward compatible
• Follow existing code patterns
• Add comments for complex logic
• Test each change individually
• Commit frequently with clear messages

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.