Inquiry Regarding Text Input Feature in SynTalker

[Cal-DCosta]

Hi Robin,
I wanted to express my appreciation for your work on SynTalker; it’s a app/project I’ve been searching for quite a while.
I have a question regarding the text input functionality mentioned in your paper. While I explored the Hugging Face demo, I was unable to locate an input box for text alongside the audio file input. Could you clarify how the text input is integrated into the application? Is it available in the demo, or is it implemented in a different part of the codebase?
Thank you for your time, and I look forward to your response.

[RobinWitch]

Thank you for paying attention to our work.
Now it only supports pure speech audio input. Enable text input simultaneously with speech is in schedule.
You can see our release plan. Our syntalker can recieve both speech and text input simultaneously. I will release it soon.
Because I am a bit busy lately and have some other things to do.
I think I will release sampling code in two weeks, online web demo and training code in three months.
Hope you can understand!
If it's urgent and you need to reproduce it yourself, feel free to contact with me.

[Cal-DCosta]

Thank you looking forward to the full demo, all the best!

[SreehariRajan]

Is the training script with text input available?

[RobinWitch]

Available.
Due to being occupied with other matters and the many dependencies involved in text inference, the progress of open source has been somewhat delayed. I plan to organize and release the inference code within a month, while the training code will be released within two months.

Is the training script with text input available?

[SreehariRajan]

So currently in the repo only speech to motion training script is present ?

[RobinWitch]

Yes

[SreehariRajan]

Is the motion auto-encoder trained with BEATX and HumanML3D datasets for the given pre-trained weights ?

[RobinWitch]

Is the motion auto-encoder trained with BEATX and HumanML3D datasets for the given pre-trained weights ?

Yes, we have provided it in this update. Please check Enabling Text Control.

[SreehariRajan]

Is the code for visualizing with text prompt available ?

[RobinWitch]

Is the code for visualizing with text prompt available ?

Yes! Now it's available, please checkout our latest update.

[SreehariRajan]

fatal: No url found for submodule path 'libs/human_body_prior/body_visualizer' in .gitmodules
Failed to recurse into submodule path 'libs/human_body_prior'

Can you add the URL ?

[RobinWitch]

fatal: No url found for submodule path 'libs/human_body_prior/body_visualizer' in .gitmodules Failed to recurse into submodule path 'libs/human_body_prior'

Can you add the URL ?

Sorry, I forgot it, now I have added it !

[SreehariRajan]

Running the command python preprocess/download_hf.py downloads the dataset "beatx_1-30_amass_h3d." Does this include the complete AMASS dataset for training?

[RobinWitch]

Running the command python preprocess/download_hf.py downloads the dataset "beatx_1-30_amass_h3d." Does this include the complete AMASS dataset for training?

No, it only includes preprocessed BEATX datasets (which also named BEAT2, include speaker 1-30), using AMASS data for training will additionally need another files, I will upload it to huggingface tomorrow. After downloading it, you need to set use_amass in configs/diffusion_h3d.yaml to True for using AMASS data in training.

I'm so busy recently. I think I will release preprocess raw data code in two weeks. Thank you for your attention and patience.

[SreehariRajan]

Ok, thank you,
The dataset you will upload in huggingface will be the dataset with corrected fps ?