Conversation
|
@T4RuN05 is attempting to deploy a commit to the shreevathsa05's projects team on Vercel, but is not a member of this team. To resolve this issue, you can:
To read more about collaboration on Vercel, click here. |
There was a problem hiding this comment.
Pull request overview
Adds an “AI Avatar” survey-taking mode to the citizen frontend and introduces backend support for uploading voice answers and transcribing them via the AI service, plus a new avatar-trigger API scaffold.
Changes:
- Frontend: adds Avatar mode UI, 3D avatar rendering, scripted avatar flow, and voice recording for text answers.
- Backend (main2): accepts multipart survey submissions with audio files and asynchronously runs STT to replace placeholders with transcripts.
- Backend (ai): adds an STT endpoint for raw browser recordings and a dynamic avatar script endpoint; adds tooling for static avatar prompt audio.
Reviewed changes
Copilot reviewed 16 out of 26 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| frontend/citizen/src/pages/SurveyPage.jsx | Adds Avatar mode entry point and switches submission to multipart when audio blobs are present. |
| frontend/citizen/src/components/avatar/InteractionModal.jsx | New modal UI for answering avatar questions (MCQ/checkbox/voice). |
| frontend/citizen/src/components/avatar/AvatarSurveyMode.jsx | New scripted avatar survey runtime (audio playback + sequencing). |
| frontend/citizen/src/components/avatar/AvatarCanvas.jsx | New 3D avatar canvas with basic jaw/head animation driven by audio intensity. |
| frontend/citizen/package.json | Adds react-three dependencies (three, @react-three/*). |
| frontend/citizen/package-lock.json | Locks new 3D dependency tree. |
| backend/main2/src/utils/audioProcessor.js | New STT forwarder that updates stored SurveyResponses with transcripts. |
| backend/main2/src/routes/responseRoute.js | Adds multer to accept multipart uploads on response submission. |
| backend/main2/src/routes/avatarRoute.js | New authenticated API route scaffold for “trigger avatar survey”. |
| backend/main2/src/models/surveySchema.js | Adds avatar to allowedChannels. |
| backend/main2/src/controllers/responseController.js | Parses FormData JSON fields and invokes STT processing in the background worker. |
| backend/main2/src/controllers/avatarController.js | Adds placeholder “dispatch” logic for future avatar/FOD integration. |
| backend/main2/src/app.js | Registers the new /api/avatar route. |
| backend/ai/src/utils/tts.js | Switches Sarvam TTS speaker voice. |
| backend/ai/src/utils/generate_avatar_statics.js | Adds a script to generate and upload static avatar prompt audio. |
| backend/ai/src/router/speech_conversion.js | Adds raw-file STT endpoint and avatar script generation route; supports static audio bucket routing. |
| backend/ai/package.json | Adds multer dependency for STT file uploads. |
Files not reviewed (1)
- frontend/citizen/package-lock.json: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @@ -0,0 +1,303 @@ | |||
| import React, { useState, useRef, useEffect } from 'react'; | |||
| import { Mic, Square, Check, Loader2 } from 'lucide-react'; | |||
Comment on lines
+29
to
+42
| useEffect(() => { | ||
| setSelectedOptions([]); | ||
| setSubmittedOption(null); | ||
| setIsRecording(false); | ||
| setRecordingTime(0); | ||
| setMicIntensity(0); | ||
| audioChunksRef.current = []; | ||
| if (timerRef.current) clearInterval(timerRef.current); | ||
| if (vadReqRef.current) cancelAnimationFrame(vadReqRef.current); | ||
| if (audioCtxRef.current) { | ||
| audioCtxRef.current.close(); | ||
| audioCtxRef.current = null; | ||
| } | ||
| }, [question?.qid]); |
Comment on lines
+176
to
+180
| if (node.step === 'greeting' || node.step === 'outro' || node.step === 'instruction') { | ||
| await playAudio(node.audioId, node.fallbackText); | ||
| } else if (node.step === 'question') { | ||
| await playAudio(node.audioId, node.fallbackText); | ||
| } |
Comment on lines
+59
to
+70
| // Cleanup audio on unmount | ||
| useEffect(() => { | ||
| return () => { | ||
| if (reqAnimRef.current) cancelAnimationFrame(reqAnimRef.current); | ||
| if (audioRef.current) { | ||
| audioRef.current.pause(); | ||
| audioRef.current = null; | ||
| } | ||
| window.speechSynthesis.cancel(); | ||
| setAudioIntensity(0); | ||
| }; | ||
| }, []); |
Comment on lines
+28
to
+34
| // Parse JSON if received via FormData | ||
| if (typeof response === "string") { | ||
| try { response = JSON.parse(response); } catch(e) {} | ||
| } | ||
| if (typeof paraInfo === "string") { | ||
| try { paraInfo = JSON.parse(paraInfo); } catch(e) {} | ||
| } |
Comment on lines
+63
to
+66
| // Cleanup temp file | ||
| if (fs.existsSync(file.path)) { | ||
| fs.unlinkSync(file.path); | ||
| } |
Comment on lines
+63
to
+65
| if (req.file && fs.existsSync(req.file.path)) { | ||
| fs.unlinkSync(req.file.path); | ||
| } |
| import fs from 'fs'; | ||
| import { SurveyResponse } from '../models/responsesSchema.js'; | ||
|
|
||
| const AI_SERVICE_URL = process.env.AI_SERVER_URL || 'http://localhost:3001'; |
| import fs from "fs"; | ||
| import { sarvam_voice } from "../models/llms.js"; | ||
|
|
||
| const upload = multer({ dest: "uploads/" }); |
Comment on lines
+5
to
+7
| import multer from "multer"; | ||
|
|
||
| const upload = multer({ dest: "uploads/" }); // Temporary local storage before processing |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.