One Audiobook Narrator to Rule Them All

I know it's a bad idea to switch narrators in the middle of a series, but would it be advisable to use different narrators for my different books? I don't have enough time to become proficient at narration before my upcoming launch, but I may do my future books. As a reader, would this irritate you?