UTMOS上比GT还高？

[lastapple]

震惊，难道GT还不如生成的？

[zhenye234]

Thank you for your interest and observation!
It's becoming more common for synthesized results to outperform ground truth in TTS. For instance, models like NS2, StyleTTS2, and VALL-E 2 have all shown instances where the generated outputs surpass the original recordings. Instead, our approach focus on the shortcomings of existing codecs, significantly enhancing the TTS performance.

Moreover, we've open-sourced the codec's checkpoint, making it easy for you to replicate our experiments using VALL-E (https://github.com/lifeiteng/vall-e). I've also just upload my VALL-E results for you to listen to (https://drive.google.com/file/d/1irlGr-5fpnPwIzHMkMTGbU5T3OpiPsIS/view?usp=sharing). I look forward to your thoughts and further discussion.

[patriotyk]

It sounds really fantastic. As I understand it can be also used with StyleTTS2? Do you have an example how it could be applied?

[zhenye234]

Thank you for your question! StyleTTS2 is trained end-to-end, so it might be challenging to apply our approach directly. For non-autoregressive (NAR) TTS models like NS2, our method might be more applicable, but I'm not sure if it will work. It would be interesting to explore whether unifying semantic and acoustic representations could further improve NAR audio generation models.

[patriotyk]

So as I understand the biggest problem in StyleTTS2 is vocoder? But maybe it could be replaced with codes based one?

[zhenye234]

You're right