Pretalk

A multi-modal AI voice assistant designed to enhance the current voice model with speaker diarization and background noise detection.

#clone it
git clone https://github.com/zkalykov/pre_neuratalk.git
cd pre_neuratalk

#Set up env
python -m venv venv
source venv/bin/activate

#install dependencies
pip install -r requirements.txt

.env setup

Create a .env file in the root directory and add your API keys:

OPENAI_API_KEY=your_openai_api_key_here
NGROK_URL=ngrok_url

Dependencies

We need Goolge Colab or any other place to host our speaker id and background noise detection models In our case it is Google Colab. Please make sure to select T4 GPU for better performance. we_speech.ipynb ⟵ Look up for this file or visit online:

https://colab.research.google.com/drive/18TmhMJ60yDwhYVK7duDQ0iE5g9oq9liF?usp=sharing

Dependencies

• requirements.txt ⟵ Look for dependencies

Audio Analysis

• transformers>=4.20.0 - Transformer models for audio processing
• torch>=1.12.0 - PyTorch deep learning framework
• torchaudio>=0.12.0 - Audio processing with PyTorch
• librosa>=0.9.0 - Audio analysis library
• pyannote.audio>=3.1.0 - Speaker diarization and audio segmentation
• openai-whisper>=20230314 - OpenAI's Whisper speech recognition
• scikit-learn>=1.0.0 - Machine learning utilities
• silero-vad>=5.1.0 - Voice activity detection

RAG and Embeddings

• faiss-cpu>=1.7.4 - Efficient similarity search and clustering
• sentence-transformers>=2.2.2 - Sentence embeddings

External Models and Libraries

• OpenAI Whisper: State-of-the-art speech recognition model
• Pyannote.audio: Speaker diarization and audio segmentation toolkit
• Silero VAD: Voice activity detection models
• Transformers: Hugging Face's transformer models for various NLP tasks

Project Structure

pre_neuratalk/
├── agentic_rag.py          # RAG implementation
├── data_processor.py       # Data processing utilities
├── get_backend_analyze.py  # Backend analysis tools
├── get_last_ten_chat.py   # Chat history management
├── pretalk.py             # Main application entry point
├── prompt.yaml            # Configuration for prompts
├── save_audio_file.py     # Audio file handling
├── speech_to_text.py      # Speech recognition module
├── text_to_speech.py      # Text-to-speech synthesis
├── requirements.txt       # Python dependencies
├── metadata_*.json        # Metadata files
└── temp/                  # Temporary files directory

Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add some amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• OpenAI for Whisper and GPT models
• Hugging Face for transformers and model hosting
• Pyannote team for speaker diarization tools
• Google Colab for cloud computing infrastructure