[Feature Request] Using custom TTS voices?

[toric-gh]

I recently went through the long process of creating a custom model with my own voice through Piper only to find out that simply dumping it in the Piper folder that NaturalSpeech uses doesn't actually do anything, and there doesn't seem to be any other way to work around this issue. I assume most people who use this plugin may not actually want to go through all the trouble, but it would be great if we could upload a custom voice all the same. Thanks for your consideration either way.

[phyce]

Will be possible to do soon. Care to share your process of creating a model? I've been trying to find time to learn but haven't had the time yet

[toric-gh]

Sure! But just as a fair warning, it's kind of a messy affair until someone somewhere streamlines the process (I'd do it myself but I'm not savvy with Python scripting).

I used Piper Recording Studio (PRS) and recorded about 100 lines before exporting it to a dataset. I then tried to set up Piper locally to train a model with my dataset, but there was a lot of frustration with getting everything set up (I think it's due to PyTorch libraries being constantly updated), so I ended up using a training notebook that someone else came up with for the newer version of Piper TTS. They have a walkthrough for it here, but the way they formatted their metadata.csv is different than how PRS exports it -- I'd suggest going in and adjusting the CSV to be similar to how the tutorial video shows it just to be safe.

The notebook currently has an issue with exporting the ONNX model, but there appears to be a fix for that here. I was able to export the model and its associated JSON because of it, but I had Gemini implement the fix for me while it helped me figure out what the problem was because, again, I know very little about Python.

Hopefully that helps you figure out a workflow if you want to streamline it yourself, but let me know if there's anything that you get stuck on.

(Also, I just tested the exported model that I had, and PiperUI crashed when I tried to generate speech. But I was trying to figure out why the hell my checkpoint wouldn't export as an ONNX model, so I might have to recreate the model now that I have a better idea of what I should be doing.)

[toric-gh]

Okay, update on recreating the model. Here's what I did to get a successful test running:

1. Recorded my voice with Piper Recording Studio (already done)
2. Exported my recordings into a dataset (already done)
3. Opened the previously linked training notebook and edited the metadata.csv format to match the one shown in the previously linked walkthrough.
4. Opened Google Drive and created folders to upload my dataset: /tts_data/myvoice/wavs/
5. Uploaded metadata.csv to myvoice and the voice recordings to wavs
6. Connected to a GPU in the training notebook and ran each cell up to Step 2 ("Clone repo fresh")
7. Implemented the previously linked fix that sets "dynamo" to false so that the checkpoint can be exported to an ONNX model.
8. Ran each cell for Steps 3-8 while referencing the walkthrough then began training in Step 9.
9. Let the training run for about 2 hours while idly monitoring the notebook to make sure the runtime didn't die halfway through.
10. Copied path of checkpoint, found in: /content/piper1-gpl/lightning_logs/version_0/checkpoints/
11. Pasted path of checkpoint in the first cell of Step 10 then ran both cells to export the model and its JSON file for downloading.
12. Ran model in PiperUI to make sure speech generation worked. If you want to test it that way, you'll need to create a folder labeled "custom" in the release build's folder then copy in your model and JSON files in the custom folder before running the executable.