Face2Voice

A web application that synthesizes and outputs voice when you enter face image and text.

We propose a model for synthesizing human voice audio from human face input images and develop a web application that can serve it.
Our model takes human face images and text as inputs. Note that our goal is not to synthesize accurate human voices, but to implement characteristic voice functions that correlate with input images.

Input Text	Input Image	Synthesized Voice	Original Voice
Put the face image you want to convert to voice and listen to the voice in the text you want.		SynthesizedVoice1.mp4	OriginalVoice1.mp4
Put the face image you want to convert to voice and listen to the voice in the text you want.		SynthesizedVoice3.mp4	OriginalVoice3.mp4
Put the face image you want to convert to voice and listen to the voice in the text you want.		SynthesizedVoice2.mp4	OriginalVoice2.mp4

Video demonstration

face2voice.mp4

Face2Voice project is maintained By Hyunghee Park , Gaeun Kim , Minha Bae

Face2Voice Model

Face2Voice is a face speech conversion program, mainly based on SV2TTS, Lip2Speech

Face2Voice

The overall framework is the same as SV2TTS. Unlike SV2TTS, we use the Speaker Encoder of Lip2Speech because we need to extract the Speaker Vector using the pictures we received as input.

---

SV2TTS

Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis, NIPS 2018.
Learn Speaker verification embeding to create a natural voice with only short voice data

Input : Short speech segments and text
Output : Voice reading (input) text in a voice similar to the input voice

Lip2Speech

Show Me Your Face, And I'll Tell You How You Speak, CVPR 2022.
The identity of the speaker's voice is captured through facial features such as age, gender, and ethnicity, and the voice is generated by conditioning it with the movement of the lips.

Input : Mute video
Output : Voice that matches the video

DataSet

SV2TTS Model Training with Librispeech

Librispeech is a corpus of approximately 1000 hours of 16kHz read English speech, prepared by Vassil Panayotov with the assistance of Daniel Povey. The data is derived from read audiobooks from the LibriVox project, and has been carefully segmented and aligned.

Lip2Speech Model Training with AVSpeech Dataset

AVSpeech Dataset is a large collection of video clips of single speakers talking with no audio background interference. The dataset is based on public instructional YouTube videos (talks, lectures, HOW-TOs), from which we automatically extracted short, 3-10 second clips, where the only visible face in the video and audible sound in the soundtrack belong to a single speaking person. Below is a "small" sample of 10,000 clips from the dataset. If you click on a video you will see just a segment from the video on YouTube that is included in the dataset.

Setup

1. Install Requirements

1. I recommend setting up a virtual environment using venv, but this is optional.
2. pip install flask. This is necessary.
3. pip install -r requirements.txt

2. Download Pretrained Models

• Encoder - Copyright (c) 2021 Christen M
• Synthesizer - Copyright (c) 2019 Corentin Jemine
• Vocoder - Copyright (c) 2019 Corentin Jemine

Please ensure the files are extracted to these locations within your local copy of the repository:

savedmodels/lip2speech_final.pth
savedmodels/synthesizer.pt
savedmodels/vocoder.pt

3. Run the application

Use the flask command python app.py or flask run