Multithreading, server process size

[stefan-reich]

Hi, I have asked this before here, but was redirected to coqui-ai (and rightly so). I'm now using coqui-ai TTS directly from GitHub. So here are the questions.

Q1: Is the inference generally multithreaded?

Q2: Is the server.py process supposed to be this big? I am seeing 1.4 GB resident process size in Linux with model tts_models/en/ljspeech/vits.

If inference is not multithread and the process has to be this big, this is quite the resource hungry application, bordering on impractical...

I understand the system is in development, so both of these issues may possibly be improved in the future? Or are just some voices this big?

Many greetings,
Stefan

[erogol]

Provided inference endpoints are just for testing/trying out models and not optimized for deployment, production, or anything serious.

But of course, we are open for ✨PRs✨ and discussions for such improvements

[stefan-reich]

I see. Are there no efforts underway somewhere to make the inference system end-user-ready? Mozilla's slogan "text-to-speech for all" is very intriguing.

[BillyBobQuebec]

I see. Are there no efforts underway somewhere to make the inference system end-user-ready? Mozilla's slogan "text-to-speech for all" is very intriguing.

@stefan-reich well it is "End-user ready", for the most part, it's just not optimized as best as it can possibly be for inference speed. Having inference made more efficient is definitely something taken into consideration though, the Tacotron2-DDC model for example has the 2nd decoder pruned for inference since it's only needed for training, Hifi-GAN has something similar(you can inference with the Generator alone)

You can see other production/deployment-related goals on the roadmap, such as Onnx and Torchscript exports, which are formats more optimized and aimed towards inferencing in production applications.