Here are a pair of charts that show the skyrocketing cost of training new AI models. The top chart shows compute cost, with Google's Gemini the current record holder at about 100 billion petaflops. That's 100 yottaflops.
The bottom chart shows training costs in dollars. Gemini clocks in at nearly $200 million. GPT-4 cost nearly $100 million.
Thank goodness we humans are spending our resources on really important things like this. How much do these things cost to access? $20-30 per month. And up to...
"As of recent updates, the "ChatGPT Pro" premium tier, which offers expanded access to OpenAI's advanced models, costs $200 per month. This includes unlimited access to models like OpenAI o1, o1-mini, GPT-4o, and Advanced Voice mode."
I don't think I would ever pay even $20 a month to be entertained by it.
Report in the Washington Post about the Israelis supercharging their target selection with AI. It can pick targets much faster than intelligence analysts! Woohoo! It’s making the world a better place already! Or something…
I pay $20 a month just so I have all the access I need, when I need it without being throttled during peak hours. It was a hard sell since I hate subscriptions, but the first time I hit the wall when I needed to get something done was enough.
The estimate of next-gen LLM AI training was in the billions. Think about how much power that is sucking up.
Yes, the y-scale is a log scale.
The problem now is that the training data set is basically everything already.
I keep thinking that, along with Trump back in office and the sheer spurt in price relative to earnings in the last couple of years, to fear for the markets. While cash is pouring into development, is it making enough money? There's an obvious parallel in the tech boom from the late 1990s to the market decline in 2001, when another greedy, empty-headed GOP president was also after tax cuts to the rich.
That's not to say that AI isn't the future. So was tech in 2001. Just that markets were out of hand. And it isn't to say I have a clue what to do with money now. Kevin's so hot on AI that he can't trouble to reconcile this with own predictions of a recession that, at four years old, did not have a basis. But this way he can claim a triumph whatever happens.
These are expensive to train, and in a lot of application, you refine the model with individual user data as well.
The big question will be how much the amortized training plus operating costs are, and whether GOOG, FB, MSFT and AMZ can find people to pay those costs.
Did I read recently that our next generation fighter plane is going to be >$100K? Which is going to be more deadly, AGI or fighter?
or a fighter controlled by an AGI?
I'm not sure what's astonishing about this. The title implies it's the computing power, but it's really just a measure of brute force computing and as a moderately tech literate person, I don't find simply putting a bunch of computing power together to be astonishing? It's just like the 2020s version of what us nerds thought about doing in the 2000s with Pentium III's and Playstation CPUs and clustering. Isn't blockchain basically just clustering version 2, but for a specific purpose? Aren't LLMs ("AI") basically just that as well? Not really astonishing.
I don't really find the costs astonishing either, because this stuff is expensive and there are diminishing returns, just like anything in tech and computing. The last petaflop costs a lot more than the ones in the middle.
What's concerning, rather than astonishing, is how power hungry these operations are (including all the data centers being built in places like Virginia). It's a lot of investment for not a lot of payoff, except insofar as it removes pesky workers (with their human demands and some level of resistance to exploitation) from the economy the tech overlords have wet dreams about - and are feverishly trying to create.
Is this really astonishing? It seems suspiciously low to me. Perhaps that's the raw compute cost via some metric, but Google made $88B last quarter. $200M is nothing to them. I'd be shocked if the training costs for a model aren't much higher than that.
A system with one of these GPU cards might have 8 of them at $10K each. So, $100M is 1250 systems training a model. I don't know what the development phase for these things look like, but I have to assume there's a lot of iterative testing and tweaking the model. And there are lots of ML engineers working on a model at one time, although perhaps subsets of the whole model can be generated and tested separately.
Still, for something this critical to a big tech company, those numbers seem low.
There’s no way that those cost numbers can be correct. Nvidia is making umpteen billion dollars a year selling AI training boards, plus all of the money spent by companies who design their own chips, plus the infrastructure to make those chips work (the physical data centers, the electricity, etc.). Yes, there are nontrivial costs associated with running the trained models, but the training itself dwarfs that. It has to be in the billions for the newest versions of the things.
This also does not account for the new hotness: performance also scales with inference-time compute. So while training costs are crazy, the inference costs of o1 and o3 are nuts (hence the $200 monthly).
Training Costs ???
I thought AI was suppose to train itself.
The old saying popped into my old weak brain: Garbage In Garbage Out.
Astonishing? Kevin has predicted this for years now. It should not be "astonishing" to him, should it?
Ed Zitron makes a very compelling case that the level of investment to keep the things running cannot be sustained and there is no conceivable glide path to making money.