Update on Grok 4 as a Dev Editor

This is a follow-up to my previous post on the subject. You might want to read that one first if you haven’t already. #Using Grok 4 as a Dev Editor

I’ve been performing a focused test on Grok 4 over the last several days. I have a very long (at the time of writing this, over 18k words, most of which are of course Grok’s responses) thread on one single developmental project. That project was picked for several reasons.

If you’re just here for the how-to, go down to Advice. That section is still pretty large, though. I tried to be as comprehensive as possible, and I’ll only make another post about this if something significant changes.

Rationale for the Project

The project in question is nothing new. I started it in college in late September 2001. Partially influenced by 9/11, but mostly because at that time everything was influenced by 9/11. It’s also nothing I’m seriously going to use. It’s a fanfic of an X-Men cartoon that was on at the time.

While I was writing it, it became fairly popular on Fanfiction dot Net’s X-Men corner, and I put a ton of work into it over the years. It survived two hard drive crashes and kept going, unlike everything else I wrote at that time; I just restarted it, while everything else lost in those crashes was abandoned. (Side-note, I was an early Carbonite adopter for obvious reasons.) I had notes on future events with a higher word-count than the actual story, the latter being 68k words when I abandoned it.

The reason I abandoned it was that I was having a harder time writing it while trying to make a go at being professional. I felt I had learned all I would learn from it at the time. So I turned it off and focused on things I could actually sell.

But the story has lived rent-free in my head ever since. Which means — long story short — that I felt it was a good choice for testing Grok 4.

Results

I started out with one of the things that the cartoon, X-Men: Evolution, glossed over a lot for obvious reasons: life at the fictional Bayville High. In the show, the school just serves as a backdrop to the action. There’s little to no development to it as a setting. So I wanted to see what I could get Grok to come up with.

I told it I was developing the setting for a novel, in a fictional costal city in Westchester, NY, with only one high school, in the year 2002. It was able to find and synthesize a massive amount of data about Westchester, as well as NYSED requirements and statistics, to generate a detailed description of the school and, by extension, the sort of city it would be in.

I want to emphasize that I was having it do historical research on topics that are not at all fun to research. I love research, but I love some topics more than others. It is very hard to love researching population statistics in New York.

I won’t bore you with all the details of class schedules, high school drama archetypes, and demographic statistics. Suffice to say that it seems accurate enough for fiction, and especially for something in this genre. If I’d had access to this when I was actively writing it twenty-some years ago, people would have thought I was a local.

(Remember, accuracy is important, but the level of accuracy needed depends on your audience.)

Eventually, I asked Grok to evaluate if this setting was similar to any other published settings. It immediately identified it as functionally identical to X-Men: Evolution. Now, this is after deliberately mimicking it, so I’m not sure it’ll be as good at identifying more tentative connections; but it was still impressive how fast the answer came back.

From there, I started it on more direct evaluations as a fanfic, and it was able to identify better connections from the main characters to the setting, and vice versa.

The long, long list of notes I had made about where I wanted the story to go meant I had a very large body of work to test it against and suggest refinements for me based on comparisons to both the show and other incarnations of the X-Men franchise. Expanding it to the wider Marvel universe and suggesting connections with Spider-Man, Avengers, and Defenders confused it slightly — but I suspect it was more because it was having trouble working around some plot limitations I gave it.

I already knew that was an issue; the better thing to do is to give it a narrow beginning with more freedom at the end, rather than a limited ending as well. But again, it’s a stress test.

After that, I tested it by “priming the pump” through asking it to summarize things from various questions, then synthesize something from that. That worked far better than anything else. I got it to evaluate a timeline by adding in real-world events from those years and suggest connections; many of them were terrible suggestions, but it was a very interesting response.

I even told it to update everything to take place a few years later than the show’s own air dates, expecting Grok to run into problems with that. I was wrong; it handled the transition flawlessly.

Its biggest balk was when trying to get it to auto-generate ages and grade levels for the high school characters in the show. I could not convince it to go a different route that fit the interpretation of my original fanfic without flat-out stating that my version was more canon than other sources (and its evaluation of the characters’ grade levels was really weird). That’s probably not going to be an issue with original fiction, of course.

The biggest benefit from all of this, if I were really writing this, was that it understood that the purpose of the story was “teen drama with superpowers” rather than “superhero story with teens,” which fit the original source material. Considering the wider Marvel universe, I fully expected it to have problems with that; but it was able to focus on the idea that, no matter how many questions I asked about the wider Marvel universe and expanding the focus of the show, it treated the hierarchy as the show being more canon than the rest of Marvel (while still allowing the introduction of new concepts), and my fanfic being more canon than the show. That’s something that both ChatGPT and Grok 3 would struggle with.

Advice

To use Grok as a dev editor, you have to know how to prompt it, how to do what you’re prompting it to do, how to discard junk information, and when to go to a real person.

Proper Prompting
First, the prompt should start as follows: Acting as a straightforward developmental editor, with no unnecessary enthusiasm or praise . . . LLMs, as I said last time, are programmed to tailor themselves to what you want; you have to tell them how you want them to act or it’s going to make assumptions about you. And one of those default assumptions always seems to be “Delicate artiste, shower with praise, avoid hurt feelings.”

You don’t want Grok to avoid hurting your feelings. You want Grok to rip your ideas to shreds. This is because if you are a delicate artiste, you’ll be less hurt by a machine than a human; and even if you’re not, you might as well get the junk out of the way before you get a human to tell you it sucks.

You also need to give the LLM reminders as to how it’s supposed to act as you go. Grok 4 was doing a fantastic job at staying on track compared to previous models, but it still started drifting every 3 or 4 thousand words. As a rule of thumb, even if it feels very repetitive, you may need to repeat that initial “attitude” prompt every 10 prompts or so.

Know how to Research
Next, as I said, you already have to know how to do what you’re prompting Grok (or any other LLM) to do. If you’re using it to do research, you have to know how to do that research, otherwise you’re not going to ask the right questions and you won’t know how to evaluate the answers.

An LLM is a tool. Once you know how to do what you want it to do, you can then get it to speed up your work. If you don’t know how boards are supposed to go together, you’re going to be terrible at using a nail gun.

If you need accuracy, as well, then you must be willing to check its citations. Every single one. Remember, that’s still faster than trying to look up things yourself, but you have to check because as good as it is, Grok 4 will still hallucinate. In fact, it’s so good at not hallucinating that it’s easy to miss when it does hallucinate.

Know how to Evaluate
Grok is also much, much better at research than it is at evaluating fiction. To make the latter work, it absolutely requires comparison material. It was very shaky on evaluating completely original material, but it was absolutely spot-on when evaluating an X-Men fanfic because it could compare against not just the source material but also all versions of the X-Men stories.

Even so, it was terrible at anything to do with “ordinary life” that wasn’t structured by statistics. Even its analysis of something as simple as a conversation between canon X-Men characters was stilted and aimless. Far better than anything I’ve tested before, but it hasn’t even managed to pick its way down more than a little of the Uncanny Valley right now.

Villains and Empathy
Interestingly, though, it was very good at evaluating the villains. So while you might not be able to get it to help with your sympathetic protagonists (which depend a lot on empathy), you could get it to help you with characters your audience is supposed to hate. When I suggested a merger of three separate villain organizations in the wider Marvel universe, it was able to give me five proposals; and when I drew a single thread from those suggestions and embellished with my own ideas, it was able to correctly identify every single strength and weakness I already knew was there, plus a few others I hadn’t thought about. More to the point, it made some excellent suggestions on how to refine my description, tailored to the theoretical audience, that in all my not-inconsiderable experience I have to admit I agreed with 100%.

This ties back to why LLMs aren’t good at writing compelling stories, though. If you want empathetic characters, they aren’t going to help; but if you’re not going for an empathetic villain (or you want a non-empathetic base for them and you’ll layer something more human on top of that), I think that’s worth exploring. I’d need more testing with other settings to be certain, but I find it very encouraging.

(Also, insert joke here about AI being good at figuring out villains.)

Prime the Pump
LLMs work off of being fed information. If you need it to synthesize something for you, you need to provide it material to work with. That, however, is a simple process. Just ask questions about a topic until you have enough there to work with, then ask Grok to conclude something about that. E.g., “Using the above information, please make several suggestions on how to XYZ . . .”

LLMs are terrible at predicting the future, but if you give it a good starting point and plan, then it can make analyses about things like “How does the US government respond to the above?” This is better in an historical or speculative field, though, rather than current-day, because the sources it draws on get muddled.

Avoid Megathreads
I’ve been testing it this way deliberately, but while it’s a lot better at keeping track than other LLMs, Grok 4 still has trouble evaluating long threads. My nearly-20k-word thread is still going strong, but based on its provided statistics and the slight errors that are creeping in, I suspect that after about 35k it’ll get much more noticeable. Grok has also mentioned it has trouble at anything over 50k.

You can help it by telling it to ignore certain prompts and responses, but if you really need to keep the thread going, you should start a new one and load in all the stuff you decided on from the last one. This keeps it from using discarded suggestions as “canon” for your thread. It tries very hard to avoid this, but it knows it has weaknesses. It also knows it can’t quite account for them.

Remember to Adjust Its Sources
You can tell Grok to avoid using certain sources, such as X itself. Sadly, you can’t tell it to ignore Reddit; but when it does pull from Reddit, you can see the icon. This is more important on some topics than others, though; if you want statistics on high school life from the early 2000s, Reddit is actually a great resource.

Grok Suspects Its Own Weaknesses
To clarify: it doesn’t actually know them, other than what it can research online or what it’s already programmed and trained with. It knows, for example, that it lacks the ability to describe empathy. It suspects, but doesn’t know, that it lacks a sense of drama and timing. However, it cannot identify when it fails unless you point it out. Even then, it’s only going off of your input.

But what it can do is tell you what to ask a human dev editor. And so far, those suggestions have been a very good guide. Not entirely spot-on like with that villain section, but it’s definitely worth prompting it to suggest questions for a human to answer. That will save you time and money, and the editor will save time as well (and be able to do more manuscripts faster as a result, making more money).

That’s all for now. If I have any major updates to provide, I might do another post. Especially if you have any questions.