If you're interested in such things, novelist Curtis Sittenfeld wrote a few days ago in the New York Times about a writing competition with ChatGPT. Based on reader input, she chose a few themes (lust, regret, kissing) and details (middle-aged protagonist, flip-flops) and then set out to write a short story. A Times editor worked with ChatGPT to write a competing story.
As it happens, I was pretty sure I could tell which was which, and I was right. Oddly, there were two things that stood out to me. The first is obvious: the Sittenfeld story included odd, unexpected quirks that made it lively, while the ChatGPT story was sort of dull and predictable. But the second reason is that the Sittenfeld story felt awkward in places and made me want to break out my blue pencil. The prose in the ChatGPT story was merely workmanlike but hardly ever seemed objectionable. Sittenfeld basically agrees:
Overall I found it to be proficient on a sentence level but clichéd, and also shallow in sentiment. To me, there’s just something missing — like the literary equivalent of fat-free cookies or a Ken doll’s genitals.
Of course, one might say it's remarkable that ChatGPT could even come close to the level of an accomplished real writer. It's also fair to point out that ChatGPT is very sensitive to its prompts. A different editor that offered different prompts and iterated a few times might have produced something better.
And, as Sittenfeld acknowledges, there's another big difference: she took a couple of weeks to write her story and then had to spend more time paring it down to 1,000 words (the agreed length). ChatGPT wrote its story in 17 seconds and hit the length requirement right on target.
For now, this experiment shows that AI still isn't close to the best humans in creative fields. But it also shows that AI isn't all that far behind. Bewarned.
Curtis Sittenfeld is a woman.
Came here to say that. But maybe "Curtis Suttenfeld" is a man...
The “Ms, Sittenfeld” part in the byline should’ve been the tell even if Kevin didn’t know who she was.
True. Jeff Donnell was and Michael Learned are female actors, Courtney was once a man’s name. Gotta be cautious about this stuff
Can AI write a better novel than NI (natural intelligence)? On the one hand, if NI is inspired to write a novel for the sake of writing a novel, and if NI's novel is better than the AI novel, then AI isn't all that far behind. On the other hand, NI is the wrong job.
I suspect that LLMs will top out at writing novels better than most humans can write novels (perhaps even slightly better than the novels that actually currently get published), but it'll plateau beyond that.
I'm willing to bet the first NYtimes bestselling AI "authored" novel will happen within the next 18 months, but a lot of what will drive that is the novelty of the endeavor (plus a lot of chin scratching zeitgeisty pieces like Sittenfeld's). Nonetheless I would also fully expect the big publishing houses to increasingly rely on LLM-assisted processes to vet submissions, edit materials, and even help produce the sort of. garbage paperbacks that populate airport bookstores. But if machines do get around to writing truly great novels, it won't be LLM-based tech that does it. LLMs are too fundamentally designed to provide the average of what they've been trained on, and the average novel is fluff.
"For now, this experiment shows that AI still isn't close to the best humans in creative fields. But it also shows that AI isn't all that far behind. Bewarned."
Does it? The LLM is trained on a vast amount of written material. Most of that material is, from a literary standpoint, garbage.* So, yeah, the fancy autocomplete is very likely to hit the effective mean of all that material, smoothed out by the army of checkers the AI companies employ: workmanlike competency. There's no reason to think it's going to get significantly better than that, as eating the vanishing portion of written material that has yet to be fed to LLMs is unlikely to raise that mean.** Fundamentally, LLMs are very likely a dead end if what we are pusuing is general AI.
That isn't to say that they aren't extrmely useful, powerful, and likely enormously transformative; just that they aren't likely to produce great, or even quality, art. That is cold comfort to the vast majority of us schmucks--an LLM can't compete with Shakespeare, but of course neither can we. Nonetheless, I fully expect that LLM-assisted writing and editing is going to displace a lot of the work done by the people who churn out beach reads, crime serials, cozy murder mysteries, movie adaptations, and all the rest of the pulp.
_____
*I'm only sorta being a snob here--even if the LLM was trained solely on novels (and novels probably only make up a small minority of the written material on which it is trained), most novels are trash.
**And there probably aren't enough literary gems to create an effective limited training library for a special literary LLM even if that's what you set out to do.
Look! It got all the easy parts in a jiffy. How long could it possibly take to solve the Hard parts?
Right around the corner, I tell you. Once you get the punctuation correct, it is just a short step to Shakespeare.
Well, AI has been trained on basically every bit of writing that has been written (in English), and this is where we're at. This is peak A.I. There's not going to be an order of magnitude more material for training.
A.I. can be very useful--it can predict protein structures from the sequence. That's amazing. Still don't know why they fold the way they do--but with AI, we might be able to figure that out. That worked for AI because the structure of every protein ever determined is in a single database. Not sure how well AI works with membrane proteins--they're hard to crystalize so not as many structures in the database. Without a ton of specific data, AI struggles.
????????????GIGO
"AI has been trained on basically every bit of writing that has been written (in English)" - that is available online. Important distinction there.
So when it comes to writing novels, you completely flip your tune and suddenly understand (or appear to understand) the limitations of LLMs and why they're not coming for all of our jobs?
How freakin' odd.
What is AI doing in baseball analytics? Major League Baseball has a huge database of baseball statistics AI could train on.
You don't have to let AI read everything. I wonder what kind of novel you'd get if you trained your AI machine only on the accepted great writers? Tolstoy, Fitzgerald, Cather, Joyce, Garcia-Marquez, like that. And you can tell it to write a legit sequel to, let's say, Catch-22. Or crank out the next twenty-five books in Robert B. Parker's "Spenser" series. How long would that take? An hour?
Trouble is all those authors don't write like one another, they write like themselves. Throw them all into a blender and you're going to get out a homogenized slurry which writes like no one meaningful... which is exactly what Kevin saw.
I like to think the AI machine, after reading these superb writers, would attempt to produce works in the same class, and would improve after I told it "You can do better!"
It may be artificial, but if it has any intelligence won't it try to compete? I'm looking for a chatbot with which I can discuss the great works of literature.
"Or crank out the next twenty-five books in Robert B. Parker's "Spenser" series."
You mean it isn't already?
Reading an AI-written novel is mere wanking. You may as well have the AI read it for you too.
Of course, one might say it's remarkable that ChatGPT could even come close to the level of an accomplished real writer.
Based on what you just described before this, I wouldn't say that ChatGPT could even come close to the level of an accomplished writer.
I asked ChatGPT “ Write a 1000 word short story involving a nostalgic middle aged Japanese man with a bit of ennui. Include magical realism, an ex girlfriend, classical music and a cat.“
I did not get Haruki Murakami.
I did get grammatically correct sentences that checked off my requests more or less in the order I wrote them, but the content was banal, and there was no life or magic whatsoever.
This doesn't really say much about Large Language Models (LLMs) ability to write a good novel. It just shows that general purpose trained LLMs which have their temperature (how likely they are to be creative) set low aren't creating unpredictable stories. I think what we are going to find that there will need to be a few different classes of LLMs with their own specializations if we expect them to achieve the same level of quality that human specialists do. There is a reasonable argument to be made that it is in fact the limits and clusters of knowledge that authors have which create some of the most interesting stories.
yes, exactly. The best novels often relate to the author's personal experiences and training on millions of random texts can not substitute for that.
Was the fancy auto-complete story actually any good or was it just the passing semblance of a story in the form of gray mush? That's the thing that none of the AI-pushers like Mr. Drum ever seem to be able to answer, or even ask in the first place. If a "story" is just a collection of words that mostly make sense then sure. If it's something that contains actual meaning, a computer will never be able to perform that task.
Short rebuttal: Ask it to write a whodunnit. Somewhat longer rebuttal: Look up the metrics 'busrtiness', and 'surprise' to see just how far off base this real soon now claim really is.
I asked the public version of ChatGPT to write a one-page funny story,
This is one use of AI that hadn't occurred to me:
https://www.nbcnews.com/tech/tech-news/police-officers-are-starting-use-ai-chatbots-write-crime-reports-will-rcna168203
I don't know which is potentially worse -the usual police versions, or ones created by AI with a cop's input.