Add realtime audio support by JackismyShephard · Pull Request #233 · JackismyShephard/ultimate-rvc

JackismyShephard · 2025-12-14T19:56:18Z

No description provided.

+
+        """
+        self.cpt = (
+            torch.load(weight_root, map_location="cpu", weights_only=True)


The optimal fix is to prevent the use of torch.load (or any other Pickle-enabled loader) on user-supplied file paths. Instead, restrict weight_root (the model path) to a well-defined set of trusted filenames or to files located within an approved directory. Ideally, maintain a whitelist or catalog of available models on the server side, only allowing clients to request models that have been pre-verified and deployed.

To implement this, before calling torch.load, check that weight_root matches one of the allowed filenames (e.g., from a predefined set or directory), and refuse/raise/log if it is not. You may fetch the model list from a specific directory, and only allow loading files that exist in that set. Additionally, to avoid directory traversal, ensure only filenames (not arbitrary paths) are accepted from the client, and join to the models’ root directory.

Required changes:

In RealtimeVoiceConverter.load_model, before calling torch.load, check if weight_root is within a safe folder and is not trying to escape it (no ../ or absolute paths).

Tighten up loading so that only models already present in, say, models/ can be loaded, and that the path is securely joined.

Optionally, in the websocket endpoint, ensure that only legitimate model names are passed through from the client (ideally, also enforce on the API boundary).

Add helper logic to safely join and validate model paths.

+        """
+        self.cpt = (
+            torch.load(weight_root, map_location="cpu", weights_only=True)
+            if pathlib.Path(weight_root).is_file()


To fix this vulnerability, we need to prevent uncontrolled user input from being used to construct paths that can access arbitrary files. The standard solution is to restrict the search for model files to a known directory (e.g., a models/ folder). Input should be normalized with os.path.normpath and checked to ensure it does not escape this directory (i.e., no directory traversal is possible) before file operations (including checking existence and loading). We should reject any path that would fall outside the allowed root after normalization.

Edit the load_model method in RealtimeVoiceConverter so that:

Only files within a designated model directory can be loaded.

Before loading, we normalize weight_root, join it to the models directory and check that the resolved path is still inside the models directory.

If the check fails, raise an exception or otherwise fail safely.

Required changes:

Add an import for os if not already present.

Define the safe model root explicitly (e.g., a constant at the module level, or within the class).

Normalize and validate the path before loading or checking existence.

Update all code in load_model that operates on weight_root to use the validated, normalized path only if it is within the allowed root.

+
+
+def load_faiss_index(file_index):
+    if file_index != "" and pathlib.Path(file_index).exists():


To fix the problem, we need to ensure that the file path provided by the user does not allow access to files outside an intended safe directory (root). The safest and most general solution is to:

Define a root directory for index files, e.g., a directory like /server/static/faiss_indexes or similar.

Resolve the user-supplied path using os.path.normpath (or pathlib.Path.resolve()), join it to the root directory, and verify that the resulting absolute path is a child of the root directory.

Only then, open/read the file if allowed; otherwise, return an error or treat as "not found".

If no explicit "root directory" is already defined, we will need to define one. For illustration, let's use the directory containing the script as the root for FAISS index files.

In pipeline.py, edit load_faiss_index:

Set a ROOT_INDEX_DIR (e.g., in the current working directory as a subdir "faiss_indexes" or configurable).

Build the full path as the normalized path joined with the root directory.

Check that the resulting path starts with the root directory (using pathlib's .resolve()).

Only allow the read if the check passes.

Update the function to handle the case where the index file is invalid or outside the root.

Add an import for os.

No additional third-party dependencies are required.

Copilot

Pull request overview

This PR adds comprehensive realtime audio support to the RVC (Retrieval-Based Voice Conversion) system, enabling live voice conversion through WebSocket connections and direct audio device streaming. The implementation includes Voice Activity Detection (VAD), noise reduction, audio effects processing, and support for various audio devices and APIs (WASAPI, ASIO).

Key Changes:

Realtime voice conversion pipeline with streaming audio processing and circular buffer management
WebSocket server for remote audio processing with FastAPI
Audio device management with support for multiple audio APIs and monitoring
Voice Activity Detection using WebRTC VAD for efficient processing

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 21 comments.

Show a summary per file

File	Description
`src/ultimate_rvc/rvc/realtime/utils/vad.py`	Voice Activity Detection implementation using webrtcvad with frame-based speech detection
`src/ultimate_rvc/rvc/realtime/utils/torch.py`	Circular buffer write utility for efficient audio streaming
`src/ultimate_rvc/rvc/realtime/pipeline.py`	Core realtime voice conversion pipeline with F0 extraction and feature processing
`src/ultimate_rvc/rvc/realtime/core.py`	Main realtime processing classes (Realtime, VoiceChanger) with audio effects and noise reduction
`src/ultimate_rvc/rvc/realtime/client.py`	FastAPI WebSocket server for remote audio processing
`src/ultimate_rvc/rvc/realtime/callbacks.py`	Audio callback handling with threading support for stream processing
`src/ultimate_rvc/rvc/realtime/audio.py`	Audio device enumeration and stream management with sounddevice
`pyproject.toml`	Added dependencies: sounddevice and webrtcvad

Comments suppressed due to low confidence (2)

src/ultimate_rvc/rvc/realtime/client.py:97

Except block directly handles BaseException.

        except:

src/ultimate_rvc/rvc/realtime/client.py:97

'except' clause does nothing but pass and there is no explanatory comment.

        except:

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-14T20:02:18Z

+        proposed_pitch_threshold: float = 155.0,
+    ):
+        if self.vc_model is None:
+            raise RuntimeError("Voice Changer is not selected.")


The error message 'Voice Changer is not selected.' is unclear. Since this is a RuntimeError raised when vc_model is None, a more descriptive message would be 'Voice changer model not initialized' or 'Voice changer initialization failed'.

Suggested change

raise RuntimeError("Voice Changer is not selected.")

raise RuntimeError("Voice changer model not initialized.")

Copilot · 2025-12-14T20:02:19Z

+        return torch.clip(output, -1.0, 1.0, out=output)
+
+
+class Realtime_Pipeline:


The class name 'Realtime_Pipeline' doesn't follow Python naming conventions. Class names should use PascalCase without underscores (e.g., 'RealtimePipeline').

Suggested change

class Realtime_Pipeline:

class RealtimePipeline:

Copilot · 2025-12-14T20:02:19Z

+now_dir = pathlib.Path.cwd()
+sys.path.append(str(now_dir))
+
+import pathlib


The variable 'pathlib' is imported twice - once at line 2 and again at line 17. Remove the duplicate import at line 17.

Suggested change

import pathlib

Copilot · 2025-12-14T20:02:19Z

+        except:
+            pass


The bare except clause on line 97 catches all exceptions without logging or handling them properly. This can hide errors and make debugging difficult. Consider catching specific exception types or at least logging the error.

Suggested change

except:

pass

except Exception as e:

logger.exception("Exception occurred while closing WebSocket: %s", e)

Copilot · 2025-12-14T20:02:20Z

+            assert audio.dim() == 1, audio.dim()
+            feats = audio.view(1, -1).to(self.device)
+
+            formant_length = int(np.ceil(return_length * 1.0))


The magic number 1.0 is used in the formant_length calculation but appears redundant (multiplying by 1.0 has no effect). If this is a placeholder for a formant shift parameter, it should be clarified or removed.

Suggested change

formant_length = int(np.ceil(return_length * 1.0))

formant_length = int(np.ceil(return_length))

Copilot · 2025-12-14T20:02:23Z

+
+import numpy as np
+
+import torch


Module 'torch' is imported with both 'import' and 'import from'.
Module 'ultimate_rvc.rvc.realtime.utils.torch' is imported with both 'import' and 'import from'.

Copilot · 2025-12-14T20:02:23Z

+import numpy as np
+
+import faiss
+import torch


Module 'torch' is imported with both 'import' and 'import from'.
Module 'ultimate_rvc.rvc.realtime.utils.torch' is imported with both 'import' and 'import from'.

Copilot · 2025-12-14T20:02:23Z

@@ -0,0 +1,8 @@
+import torch


The module 'torch' imports itself.
The module 'ultimate_rvc.rvc.realtime.utils.torch' imports itself.

Copilot · 2025-12-14T20:02:23Z

+    except Exception as e:
+        logger.exception("An error occurred while querying the audio device: %s", e)
+        audio_device_list = []
+    except OSError as e:


This except block handling OSError is unreachable; as this except block for the more general Exception always subsumes it.

Copilot · 2025-12-14T20:02:23Z

+    except Exception as e:
+        logger.error("An error occurred while querying the host api: %s", e)
+        hostapis = []
+    except OSError as e:


This except block handling OSError is unreachable; as this except block for the more general Exception always subsumes it.

add realtime audio support

723175c

Copilot AI review requested due to automatic review settings December 14, 2025 19:56

JackismyShephard self-assigned this Dec 14, 2025

JackismyShephard added the in-progress label Dec 14, 2025

Copilot started reviewing on behalf of JackismyShephard December 14, 2025 19:56 View session

github-advanced-security AI found potential problems Dec 14, 2025

View reviewed changes

Copilot AI reviewed Dec 14, 2025

View reviewed changes

@@ -15,6 +15,7 @@
             sys.path.append(str(now_dir))
             import pathlib
+            import os
             from ultimate_rvc.rvc.configs.config import Config
             from ultimate_rvc.rvc.infer.pipeline import AudioProcessor, Autotune
@@ -36,6 +37,7 @@
                     Initializes the RealtimeVoiceConverter with default configuration, and sets up models and parameters.
                     """
                     self.config = Config()  # Load configuration
+                    self.MODELS_DIR = pathlib.Path(os.environ.get("SAFE_MODELS_DIR", "models")).resolve()
                     self.tgt_sr = None  # Target sampling rate for the output audio
                     self.net_g = None  # Generator network for voice conversion
                     self.cpt = None  # Checkpoint for loading model weights
@@ -53,12 +55,39 @@
                         weight_root (str): Path to the model weights.
                     """
-                    self.cpt = (
-                        torch.load(weight_root, map_location="cpu", weights_only=True)
-                        if pathlib.Path(weight_root).is_file()
-                        else None
-                    )
+                    # Only allow loading models from a predefined directory with valid filename
+                    safe_path = self.get_safe_model_path(weight_root)
+                    if safe_path and safe_path.is_file():
+                        self.cpt = torch.load(str(safe_path), map_location="cpu", weights_only=True)
+                    else:
+                        logger.error(f"Attempt to load model from unsafe path: {weight_root!r}")
+                        self.cpt = None
+                @staticmethod
+                def is_valid_filename(filename):
+                    # Only allow alphanumeric, dashes, underscores, and certain extensions
+                    import re
+                    return re.match(r"^[\w\-\.]+$", filename) is not None
+                def get_safe_model_path(self, model_candidate_path):
+                    import os
+                    base_dir = self.MODELS_DIR
+                    if os.path.isabs(model_candidate_path):
+                        # Only allow relative paths
+                        logger.warning("Absolute path given for model: %r", model_candidate_path)
+                        return None
+                    filename = os.path.basename(model_candidate_path)
+                    if not self.is_valid_filename(filename):
+                        logger.warning("Invalid filename for model: %r", filename)
+                        return None
+                    candidate = base_dir / filename
+                    candidate = candidate.resolve()
+                    # Prevent directory traversal
+                    if not str(candidate).startswith(str(base_dir)):
+                        logger.warning("Directory traversal attempt for model: %r", model_candidate_path)
+                        return None
+                    return candidate
                 def setup_network(self):
                     """
                     Sets up the network configuration based on the loaded checkpoint.

@@ -1,8 +1,10 @@
             import logging
             import pathlib
             import sys
+            import os
-            import numpy as np
+            # Define a directory where model files are stored. You should update this path as appropriate.
+            MODEL_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "../../../models"))
             import faiss
             import torch
@@ -50,15 +51,18 @@
                     Loads the model weights from the specified path.
                     Args:
-                        weight_root (str): Path to the model weights.
+                        weight_root (str): Path to the model weights, relative to the allowed models directory.
                     """
+                    # Normalize the user-supplied path and join to known safe root
+                    model_path = os.path.abspath(os.path.normpath(os.path.join(MODEL_ROOT, weight_root)))
+                    if not model_path.startswith(MODEL_ROOT + os.sep):
+                        raise ValueError("Model path escapes the allowed model directory")
                     self.cpt = (
-                        torch.load(weight_root, map_location="cpu", weights_only=True)
-                        if pathlib.Path(weight_root).is_file()
+                        torch.load(model_path, map_location="cpu", weights_only=True)
+                        if os.path.isfile(model_path)
                         else None
                     )
                 def setup_network(self):
                     """
                     Sets up the network configuration based on the loaded checkpoint.

@@ -1,6 +1,7 @@
             import logging
             import pathlib
             import sys
+            import os
             import numpy as np
@@ -346,17 +347,35 @@
                     return feats
+            # Define a root directory for safe index file storage (edit as appropriate)
+            ROOT_INDEX_DIR = pathlib.Path.cwd() / "faiss_indexes"
             def load_faiss_index(file_index):
-                if file_index != "" and pathlib.Path(file_index).exists():
+                index = big_npy = None
+                if file_index:
+                    # Normalize and join the provided file_index with the root
                     try:
-                        index = faiss.read_index(file_index)
-                        big_npy = index.reconstruct_n(0, index.ntotal)
+                        # Remove leading/trailing whitespace and unwanted characters (preserve existing cleaning)
+                        cleaned_file_index = (
+                            file_index.strip()
+                            .strip('"')
+                            .strip("\n")
+                            .strip('"')
+                            .strip()
+                            .replace("trained", "added")
+                        )
+                        # Construct full normalized path
+                        potential_path = ROOT_INDEX_DIR / cleaned_file_index
+                        resolved_path = potential_path.resolve()
+                        root_path = ROOT_INDEX_DIR.resolve()
+                        if not str(resolved_path).startswith(str(root_path)):
+                            logger.warning("Attempted FAISS index access outside of ROOT_INDEX_DIR: %s", resolved_path)
+                        elif resolved_path.exists() and resolved_path.is_file():
+                            index = faiss.read_index(str(resolved_path))
+                            big_npy = index.reconstruct_n(0, index.ntotal)
                     except Exception as error:
                         logger.warning("An error occurred reading the FAISS index: %s", error)
                         index = big_npy = None
-                else:
-                    index = big_npy = None
                 return index, big_npy



		def load_faiss_index(file_index):
		if file_index != "" and pathlib.Path(file_index).exists():

	raise RuntimeError("Voice Changer is not selected.")
	raise RuntimeError("Voice changer model not initialized.")

		return torch.clip(output, -1.0, 1.0, out=output)


		class Realtime_Pipeline:

	formant_length = int(np.ceil(return_length * 1.0))
	formant_length = int(np.ceil(return_length))

Uh oh!

Conversation

JackismyShephard commented Dec 14, 2025

Uh oh!

Check failure

Uh oh!

Copilot Autofix

Check failure

Uh oh!

Copilot Autofix

Check failure

Uh oh!

Copilot Autofix

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Dec 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 14, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants