The Washington Post has a story today about NewsGuard, a service that rates the reliability of news outlets. It's the usual thing: conservatives are upset because lots of conservative sites are flagged as unreliable. Imagine that.
As it turns out, NewsGuard doesn't publish its ratings. You have to pay for them. But the Post provides a sample:
National Review scores higher than the New York Times! Sadly, there's no rating for jabberwocking.com. I wonder how I'd do?
But this is not what I'm really interested in. It turns out that NewsGuard also grades the accuracy of AI chatbots, and they do release these ratings publicly. Sort of:
NewsGuard says they don't reveal the individual chatbot names because of the "systemic nature" of the accuracy problem. Boo! At the very least I'd like to know the identity of Chatbot 10, which had a perfect record. For that matter, I'd also like to know the identity of Chatbot 5, which seems worth avoiding.
If anyone at NewsGuard wants to leak the names of the chatbots, my email address is at the upper right.
Who says they're unreliable? They reliably deliver the pseudo-facts their readers demand.
Fox is significantly more reliable than MSNBC? I wonder what they were flagging as unreliable? For the most part MSNBC is sort of bland. They sometimes give their opinions but I think they label them as opinions.
This only makes sense if they are ignoring opinion and entertainment programs, as well as editorials, opinion pieces written by a certain Supreme Court justice, etc.
Any methodology that has Fox significantly ahead of MSNBC and any chatbot having 100% accurate responses is bullshit. So fuck Newsgard.
The highest chatbot is at 73%. 100% accuracy is reserved for certain MSM sources, including WSJ (right-wing slant) and WaPo (left-wing slant).
I heard David Cay Johnston talking about this today; he commented to the effect that Steven Brill (one of the two people behind NewsGuard) is a scrupulously fact-based guy.
Outlets can apparently get away with a lot of bias, if they don't actually publish false information and distinguish news from opinion. One of the criteria, worth 12.5%, is
"when reporting news, do not egregiously cherry pick facts or stories to advance opinions"
I would say that Huffpost (one of the few I actually read) does not meet this criterion. Do The National Review and other rightist outlets not select what they publish to back up their opinions? It seems to be a fact that Fox News cable is highly selective (although again I do not watch it myself).
If an outlet publishes 10% facts and 90% highly biased opinion, say, they could get a perfect score if they label fact and opinion correctly and meet the above criterion. The NY Times and WaPo make a point of publishing both left and right opinion, as do many newspapers. This is apparently not required to get a high score.
Daily Caller more reliable than CNN? Maybe the problem is that CNN invites Republicans onto their shows so they can lie to the viewers? Aside from that possibility, I call complete bullshit. Daily Caller is ridiculous.
I asked ChatGPT (I'm aware of the risks): "what chatbot is the most accurate"
Reply:
The most accurate chatbot depends on various factors ...
GPT-4 by OpenAI (ChatGPT)
Accuracy: Known for its ability to...
Strengths: Strong conversational abilities, can handle ...
[Accuracy/Strengths displayed for each of these]
Google Bard
Claude by Anthropic
Bing Chat (Powered by GPT-4)
Mistral
Meta's LLaMA (Large Language Model Meta AI)
DeepMind's Gemini
IBM Watson
Jasper AI
In summary, GPT-4 (ChatGPT) and Google Bard are generally considered among the most accurate for general-purpose conversations and up-to-date information. However, the best chatbot for accuracy will depend on the specific needs of the user and the domain of knowledge required.
👍👍👍
I want to know which one is Chatbot 5. That's the one to completely avoid. Not even half right -- that's objectively failing.
What we have is an epistemological argument with conservatives. They believe that science is an opinion, and it is no better than any other opinion (remember alternative facts?). You can pick and choose what you want to be true.
Everyone else believes that there are objective facts in the world and you must accommodate your beliefs to them, not the other way round.
But this is the whole meaning of reliability -- adherence to the facts. Conservatives intrinsically score less reliable precisely because of their adherence to alternative facts. There is no meaningful definition of reliability that fits that epistemology. It isn't conservative media per se, but conservative media addicted to and profiting from outright lies.