Kevin Drum

This is a typical weekend crowd along the Seine checking out the famous green bookstalls. These days about half the stalls sell mostly the same mass-produced bits of tourist stuff, but the other half still have interesting specialties (music, art, Romanian poetry, whatever) and are fun to browse around in.

In case you're interested, Bob Carpenter has a short technical explanation of how modern Large Language Models (like GPT-4) work:

In a nutshell, language modeling is the simple task of predicting the next subword (“called a token”) based on the previous sequence of subwords. The state-of-the-art had stalled for years on n-gram models that use the previous n subwords (usually with n < 5). In 2017, a team of Google researchers released a paper titled "Attention is all you need," which introduced the current state-of-the-art neural network architecture for language modeling. The breakthrough was in extending the context length into the thousands (GPT 3.5 uses 4K, GPT 4 has 8K and 32K models) with an attention model that figured out which parts of the context to concentrate on. The fundamental bottleneck is that computation is quadratic in context length (though it's all on GPU, so that's a massive numbers of flops for relatively low power).

There's more at the link. It's an interesting short read.

Here is James Fallows:

I think this interp is correct.

—Thomas gets all the attention.

—But Alito is the one who has defined, and utterly discredited, the current court.

Both are resentment-driven. But Alito is resentful about ... what, exactly? https://t.co/9RJIs1DEpv

— James Fallows (@JamesFallows) May 1, 2023

Alito has always been by far the worst Supreme Court justice. Whatever their faults, the others have some kind of of ideology and some sort of animating force that drives their decisionmaking. Alito doesn't. He simply votes in support of the Republican Party view and barely even tries to disguise the fact. He is the very definition of a hack.

This is a pair of sculptures by Fred Eversley in the Orange County Museum of Art. Eversley, originally an engineer, did most of his work in Venice (California), though he now lives in New York. He is, according to the David Kordansky Gallery, "a key figure in the development of contemporary art from Los Angeles during the postwar period."

His pioneering use of plastic, polyester resin, and industrial dyes and pigments reflects the technological advances that define the postwar period even as his work reveals the timeless inner workings of the human eye and mind. Eversley’s abstract, three-dimensional meditations on color—including the luminous lens-like objects for which he is best known—entice the viewer to approach, prompting questions about how the biological and optical mechanics of sight determine how we see and understand each other, and communicating a kinetic, palpable sense of the mysterious presence of energy throughout the universe.

Roger that. These pieces will be on display in New York starting a couple of weeks from now.

My doctors have been insistent that if I spike a fever above 100.4 I should get myself down the ER. Last night that happened—though just barely. The ER thermometer clocked me at 100.6, and that set off a truly impressive blitzkrieg of testing. I didn't really understand it. They took blood tests by the dozens; two chest x-rays in case I had pneumonia; a COVID test; three different bacteriological tests to narrow down the source of any possible infection (in my blood, in PICC line #1, or in PICC line #2) even though there was no reason to think I had an infection in the first place.

I took all this with my usual sunny disposition¹ and eventually they let me go to sleep. It's now 6 am, my fever is gone, the tests have all come back negative, and I still have three hours before I report to the Day Hospital. It's Day +6! Only eight days to go.

Anyway, here's a picture of me in the ER last night. Happy May Day, everyone.

¹Ha ha ha. I bitched and moaned the entire time. I had been expecting a quick evaluation, maybe an antibiotic, and then back to the hotel. I was not expecting the Spanish Inquisition.

UPDATE: I have tested negative for every possible thing and my temperature is down to 98.4. I'm back in the hotel and things have returned to normal.

I had a bit of insomnia last night and produced this fine reenactment of old-school blogging:

This chart shows the change in federal spending during every postwar administration.¹ For example, Eisenhower's first budget clocked in at 18.14% of GDP while his final budget set spending at 17.38% of GDP. So over the course of his administration spending declined by 0.76 percentage points.

Add this up across the years and Republicans have increased spending levels by 5 percentage points. Democrats have reduced spending levels by 3 percentage points.

Since the mid-50s, federal spending has roughly increased from 16% of GDP to 20% of GDP. Over the next decade or two this will almost certainly have to increase to about 25% of GDP.

¹I didn't include Truman's enormous spending cuts because they were merely the result of the end of World War II. Likewise, I didn't include Trump's enormous increases because they were a bipartisan response to the COVID emergency.

In the New York Times Magazine this weekend David Wallace-Wells has an interview with Dr. Anthony Fauci. I knew David a little bit when I was at the Washington Monthly and I've followed him with admiration ever since, so I was surprised at how combative the interview was. In one sense, David was just asking tough questions and letting Fauci clear the air, but a few too many of the questions were premised on MAGA nonsense that really didn't deserve to see the light of day. Fauci got noticeably annoyed at several points, and I don't blame him.

That said, there are two big things I'd take away from the interview. The first is this:

Wallace-Wells: Did we do enough to communicate the age skew of the disease?...I still think, honestly to this day, that almost no one appreciates just how wide that age skew really is, given that the risk to someone in their 80s or 90s is perhaps hundreds of times as high as it is to someone in their 20s or 30s.

Fauci: You are hitting on some terrific points. Did we say that the elderly were much more vulnerable? Yes. Did we say it over and over and over again? Yes, yes, yes. But somehow or other, the general public didn’t get that feeling that the vulnerable are really, really heavily weighted toward the elderly. Like 85 percent of the hospitalizations are there. But if you ask the person in the street, they may say, “Oh, yeah, elderly are more vulnerable, but everybody’s really vulnerable” — which is true, but to a much lesser extent.

This is an example of something that doesn't get appreciated enough. There was an unending cacophony of voices during the first year of the pandemic. There was the CDC, but there was also WHO. There was the Donald Trump show. There were TV doctors. There were local mayors, governors, and health departments. There were charlatans. There was Twitter. Everybody had an opinion.

But we forget all that with the passage of time. Everything gets mushed together and then blamed on "the CDC." But if you go back and look you'll often find that the CDC didn't make the recommendations that we now think are so wrong. Rather, it was the hive voice.

This isn't to excuse every mistake. Some of the CDC's recommendations were wrong. This leads to my second takeaway. It comes from Don McNeil, former COVID reporter for the New York Times. His review of the Fauci interview is scathing—much too scathing, I'd say. But he makes this key point:

The truth is that many of the early guesses made by science proved wrong. When the data changed, good scientists changed their advice.

Read the Fauci interview for examples of this. The COVID virus surprised scientists at nearly every turn. It spread asymptomatically. It was airborne. It mutated wildly. It was more transmissible than anyone expected.

All of these things required scientists to change their advice. That's not a symptom of incompetence, it's a symptom of how the real world works. Fauci acknowledges that some things could have happened more quickly, but overall I think most people don't realize just how fast science worked during the pandemic. We complain about the fact that it took a few months to learn about asymptomatic transmission, for example, but this is something that normally might have taken years. As near as I can tell, scientists blew the doors off of previous speed records. Most of them should be getting medals, not Twitter mobs at their door.

My doctor was happy to see me today because apparently I'm one of his few CAR-T patients who's currently lucid. The others are staring blankly at the ceiling and trying to remember who the president is.

But I'm not out of the woods yet. If I start posting charts of, say, oatmeal consumption vs. AAA battery usage you'll know that I too have fallen prey to "neurological fuzziness," as they call it.

I almost wish I had. At least it's evidence that the T-cells are (over) working. Instead, I have to wait a month and then get a boring old M-protein test to see if anything happened.

Today I learned that archeologists recently dug up the bodies of some soldiers who died in the Battle of Camden in 1780:

The battleground remains have been determined to belong to 12 Continental soldiers, one British loyalist and one British regular. Thirteen were honored as heroes in ceremonies planned by countless volunteers, both civilian and military. The 14th individual was determined to have had at least some Native American ancestry and so will be buried with help from the Catawba Nation and the Lumbee Tribe.

Wait. Why are we burying a British soldier? It's not like I have any hard feelings at this point, but shouldn't this redcoat be honored as a hero by Britain, not us?

I'm not sure why this amuses me so much, but it does:

This is from a study comparing human doctors to GPT 3.5. The methodology was sort of fascinating: the authors collected 195 questions and responses from real doctors on Reddit and then fed the exact same questions into the chatbot. Then they jumbled up all the responses and had them evaluated by health care professionals.

As the chart shows, the pros concluded that the chatbot's answers were more accurate and more empathetic. So what was up with the doctors? Were they telling people to suck it up and just accept the pain? Or what? Here's an example:

(Sorry this is so small. As always, click to embiggen.)

In this case, I empathize with the human doctor. My response probably would be along the lines of "ffs, it's just a toothpick," so I think the doctor was heroically patient here.

Still, the chatbot answer is demonstrably better. One reason is that it's not time restricted. Most human doctors just don't have the patience or time to write long answers with lots of little verbal curlicues. The chatbot has no such problem. It used three times as many words as the doctor and could have used ten times more with no trouble. It simply doesn't require any effort for the chatbot to be empathetic and provide lots of information that might be of only minor importance.

On the downside, chatbots also have a habit of making stuff up. Then again, my experience is that human doctors are a little too prone to this as well.

In any case, chatbots aren't ready for unsupervised prime time yet, but they probably will be before long. And here we all thought that truck drivers were the first ones who would be out of jobs thanks to AI.