AI Tools: ElevenLabs V3 Voices, SFX & Eleven Music by Jeff Foster

I was first in beta a few years ago, so I used the ElevenLabs TTS (from text to speech) tool and have seen the quality and ability gradually improve.

The latest updates in the generation of voice, sound FX and music really outweigh all previous features. Now they currently have tools for a variety of audio needs. Voice/Vo, audiobooks, conversation/API, music, sound FX, local audio dubbing.

This overview won't go too deep, but if you're interested in one tool, I'll provide plenty of links to dig deeper on your own.

voice

Not only does ElevenLabs engineers offer more control to users about how they play with synthesized voices and scripts, but the Voice Design V3 release has more expressive voices and commands. There is also a voice changer tool that extends the functionality of audiobooks, provides APIs for live conversation voices, and changes recorded voices.

Speech from text

You can insert the bracket [commands] For emotions, intensity, accents, natural pauses, sighs and laughter, whispers, sounds. (Even AI-generated farts)

I quickly tested this with a short script and a voice that I thought would match the avatar I was thinking.

I then uploaded it to Hagen's stock avatar, and this is a raw result coming out of my single pass audio. No further edits were used:

https://www.youtube.com/watch?v=9bflyhqykj4

This will reveal that it adds more personality and realism to my VO leads for video and animation projects. (Well, it's probably not just a belch and fart) 😉

Voice Changer

The new voice changer tool maintains all the variation and nuances of the original audio – whether it's a live audio recording (like your own) or a synthetic one like I've created here:

The original Eleven Love produced female voices:

Audio changer playback to male audio from Northern England created from the exported audio file above:

You can see some useful applications of this tool. In particular, you need to read the characters in a concrete way, and you can get all the tones and inflections by recording your own voice and changing it to the character's voice.

Audio dubbing

ElevenLabs Dubbing Studio allows you to exchange videos from your native language for different spoken language. Upload a video or select a YouTube video to process and select the language.

You can also launch the dubbing editor to improve your project. You can play, expand, and readjust segments. Now I haven't tried anything commercially recorded with music or other effects, so I don't know how to maintain audio integrity compared to Heygen (which changes mouth movement to match the dubbing), but I'll do a quick comparison test.

Here is an example of dubbed audio. You can hear slight changes in the voice change, but the sigh and laughter have been removed.

sound fx

I've been using ElevenLabs Sound FX for a while, and it was a real game changer with basic sound. Some don't allow you to generate exactly what you need, but to get quick stock content you can mix/layer/change it in the audio editor.

This is an edit of my meeting session at Design + AI Summit last year, where I recorded the entire 40-minute presentation as a clone in front of a “Ted Talk” AI audience. I share some of the ambient sounds I used for my main presentation in this video and some of the tricks I used for SFX.

https://www.youtube.com/watch?v=kcxaz_9-nwu

This lets you see how far you can go by combining your VO and SFX needs using one tool.

I also introduced ElevenLabs SFX in my last article on PVC. AI Tools: Video and animations appear in Midjourney:

Not much is known about another fun tool they have (In fact, it's buried under “Audio Tools” in the ElevenLabs sidebar) This is an audio/SFX soundboard app with a looper called “SB1”. You can play online now without an account here: https://elevenlabs.io/sound-effects/soundboard

11 music

This is big on many levels! It's not of the same quality as professionally recorded and produced music Sudtrack, but I think it's as good as most stock audio tracks from providers like Pond5 and SoundStripe. And it is editable in Eleven Love, which is highly customizable and editable. For social media and short industrial projects, this is a game changer for content producers.

Eleven Music takes the description and creates it (with or without lyrics) to create a soundtrack in the engine.

I experimented a bit with the instrumentals only to do some tests and demos for my clients, but so far I have passed the AI ​​sniff test in real productions.

You can play any part of the music by selecting a segment, playing or adjusting the timing and length of each segment, and dragging the end handle to move it.

Make your edits and re-register the tracks and get the correct length and movement throughout your video project. Export MP3, WAV, M4A, and FLAC formats for easy editing in NLE.

Here is an example of a completed edit for this particular track:

Music still has undescribed digital noise, excessive rewinding, and metallic unnatural sounds. I couldn't “listen” this, but if it's buried under the Vo, you really can't hear it. Anyway, I tend to remaster all the audio in client video productions, so I tend to warm up just like the AI ​​voice I use to sound warmer and realistic.

Just for Shits & Gigles, we've generated some stupid pop songs based on the prompt “I want a pop song about how AI is taking away our work.” – Here is what the lyrics were generated (one of two different variations).

Here are the lyrics:

Digital Night

And here is the generated mp3 track:

Finally, I tried some short videos from the video using the music (alpha) tool, but the results didn't really bother me. It will take longer to play this feature and report to a later follow-up article.

There is now a lot to explore in Eleven Love. By the time I go through everything, I'm sure it will be updated and there will be modifications and improvements made to the various tools.

Note: Elevenlabs seem serious about IP, and therefore include a musical term sheet for legal and unapproved prompts for reference: https://elevenlabs.io/music-terms