Playing around with AI Review and Editting

I’m using a piece of software called Novel Crafter. One of the selling points of this software is the ability to ‘Chat’ with your book. The feature set goes a bit deeper allowing you to build custom prompts, selecting context (chapters, acts, codex, etc) , and it has a reasonable amount of boilerplate prompts for specific asks such as characterization, developmental editing, pacing, and other concerns.

I have found their prompts to be very helpful for the more mechanical aspects of editing. You don’t need a very powerful model to catch mistakes in dialog tags, slipping on voice, POV, Tense, and all of the little mistakes you make when drafting.

Having reached roughly a quarter of my planned novel length I’ve started playing with the more advanced tools as they relate to characterization, differentiating my characters, pacing across chapters, consistency of narration, descriptions, developmental editing, etc etc etc. The chats are stored so I can revisit them in the future for reference when that’s useful.

One of the more interesting discoveries I have made to date is that the frontier models push back on your story in surprisingly underhanded ways. Sonnet 5.5 has been the worst offender so far, but it’s also been the most accurate for the mechanical asks which is why I ended up using it the most over the last few days. To give an example. I have a group of 100 people that have been selected by a group of aliens for a short list of traits. Sonnet 5.5 was extremely insistent that the story couldn’t work if all 100 people were aligned in some of their opinions / beliefs, but it was very concerned that anyone would have any dissent about other opinions / beliefs. It never presented the opinions or beliefs as being problematic themselves. It only insisted that dissent was absolutely necessary for some of them and impossible to even consider for others.

Having noted this I tried some of the Models that are listed as being NSFW friendly. I haven’t bothered to explore NSFW content creation as I’m not interested, but I was curious if they would show the same bias of opinion. The larger models in this category shared the same behavior where they insisted on the opposing view points, but smaller models did not. Though the smaller models weren’t generally helpful at all in regards to the more complex questions. That was not particularly surprising to me as the objective in creating NSFW models was about getting distilled models to do things the frontier models wouldn’t, which doesn’t require contending with biases in the model.

Another rather shocking discovery was the extent to which some models took extreme issue with Character’s that lacked a formal college education. Any Character with a blue collar career path that demonstrated an understanding of basic physics chemistry, or any form of scientific principle was identified as too much of a stretch for the reader to accept. In some of those cases there were very clear explanations that the thing they were engaging with had been a life long hobby for that person, but their lack of a piece of paper was too much for the AI Model to stomach.

My take away from all of this is that I’m going to have to be much more careful in what I accept as feedback from the AI. The biases show up in rather surprising ways that aren’t immediately oblivious if you’re not familiar with their behavior. Most of my experience being code based these aren’t normally issues that I need to contend with when reviewing AI outputs. There’s just not much bias regarding languages or frameworks unless you bake it into the documentation and that’s usually desirable for continuity.

Those complaints aside I did find useful feedback in relation to pacing, whether my characters were distinct, the fact that I have omitted sharing inner thoughts for some characters, genuine drift in facts about the character (pantsing here). All of these were largely accurate and should be fixed prior to seeing a real human editor. I do believe the cost benefit to be broadly positive in terms of what AI can help you find and solve without needing a lot of direct support.

Things like my characters best moments, best lines, whether the dialog works, strongest emotional moments and things like that I’m far less confident about in terms of feedback. It does do a good job of identifying where I’ve wasted the readers time instead of ending a scene properly. That one in particular seems to work really well from what I can tell.

I’m thinking the Sanderson advice about taking maybe 10-20% of feedback is the right spot for AI, discounting the mechanic editing stuff, I probably need all of that and more 🤣