$50k bounty for working implementation

[ryanod2014]

We are looking for someone to create a working implementation of SoundStorm that we can use internally on our own GPU's.

If you are interested please contact [email protected] or respond in this discussion and our team will reach out

For payment the $50k we are happy to put in escrow and will be released upon successful completion.

We are looking for someone who can move fast and has a rock solid track record.

[somehowchris]

We have a promising experiment running, but there are a lot of code changes needed to get the same system setups as described in the paper and even our current GPU setup is nothing compared to Googles Research TPU clusters

[lucasnewman]

I don't want to dissuade you from pursuing something cool, but SoundStorm is fairly tricky to train and has generally been surpassed by other methods these days imho.

The work I did with Phil on voicebox might be easier to train as an end-to-end TTS solution (I even put up a small pretrained model for it on HF) and it uses the same semantic generation model, Spear-TTS, as this repository, so a lot of the work wouldn't be duplicative.

Good luck with your project! 🚀

[somehowchris]

thanks, and yes there has been a lot of research but what we see as unique is the use of soundstream; as described in the paper spear-tts like conditioning is only one option and part to get things going to have a setup to which one could generate samples like they show on their github page.
This also takes a unique turn as one can make LLMs aware of soundstream tokens. Let's see where this goes :)

[lucasnewman]

Yep, makes sense!

You may be aware of all of this, but if you're looking for a shortcut to training SoundStream (the neural audio codec), encodec from Meta and descript are both high quality options that have pretrained models available — you could even prototype your multimodal LLM with those while you train your own codec.

There's also a really nice vocoder for encodec here that would let you handle both the output side of your LLM as well, similar to the approach used by AudioPaLM.