ElevenLabs Voice Clone - Lessons Learned

I’ve been experimenting with Elevenlabs to create a professional voice clone of my voice to investigate the possibility of producing an audiobook that way. You need the creator level to create a professional voice clone. They give you the first month half off ($11, then $22/month after that). So for a budget of $55 I gave myself 3 months to investigate.

Here’s what I learned.

1) The software is very easy to learn and use.

2) The A.I voice tech support is slow and annoying, but thankfully I didn’t need it much. I found emailing tech support quicker and easier.

3) I used Thomas’s tool to create a pronunciation guide and this worked well. But my nonfiction book doesn’t have many words to stumble over — just my name.

Voice Cloning:

They recommend uploading 3 hours of recorded audio, no music, no other voices. I have 500 podcasts so this was no problem to find.

My Voice 1.0 - I used podcasts from a variety of seasons to create the clone. This was a mistake. The audio levels of the older seasons was not as good as the newer seasons. Elevenlabs tried to faithfully recreate the older audio and would randomly switch between the different sounds, making it sound like you’d walked into a new room or gotten in a car.

Deleted My Voice 1.0.

My Voice 2.0 - I used podcasts from the same season that were produced on the same equipment using the same levels. MUCH Better! This voice sounded like me and the levels were consistent.

BUT it had an annoying habit of ending sentences with the voice going up (like a question, only the sentence was not a question). These wrong inflections can be fixed in production but it would take a lot of time and credits to hunt them all down. Might as well read the book myself if I’m going to do that.

After consulting with tech support, I reviewed my audio input and discovered I have a habit of ending sentences with my voice going up when I’m reading a list and the list is not over. But ElevenLabls was putting the inflected voice in random inappropriate places.

Deleted My Voice 2.0

My Voice 3.0 - I recorded 4 new podcasts being very careful to watch my inflections. I created a new voice clone with these new podcasts.

Success — wait, no. I successfully solved the inflection problem, but the voice doesn’t sound like me. It sounds much more like a child reading out loud in front of the class. I think I watched my inflections too critically & carefully.

After consulting with tech support, I’ve decided to try again with 2 podcasts of careful inflection and 2 ‘normal’ podcasts. Stay tuned. We’ll see how 4.0 turns out.

Lessons Learned:

The audio input determines what you get out. ElevenLabs is very good at reproducing EXACTLY what you give it - warts and all.

If you can get the audio input right, you can get a very realistic sounding “you.” For nonfiction, it is probably be good enough. I’m not sure if it’s good enough for fiction, especially because fiction competes with full-cast audio.

Note: To actually produce an audiobook, you need to bump up to the Professional Plan ($99/month) because it’s the only plan that will export audio at the required specs for audiobooks. But you can stay on the lower plan until you create or find the voice clone you want. Bump to the higher plan, create your book and then bump back down.

One more note: The ElevenLabs voices are all real people who license their voice for use. If you choose one of them to read your audiobook, an actual real person gets paid. Like regular audiobooks, the best voices cost more than the others. Look at what the person charges before falling in love with their voice for your book.