I’m kind of down on AIs right now. They’re great at agreeing with you and giving positive affirmation, but not great for telling you where you’re wrong.
A few days ago I was chatting with an AI about a problem: how can a program load IMS database segments from an extract file that has the segment name and concatenated key, without the program needing to know the database structure? It went like this:
Me: I’ve got this problem (description)
AI: Yup, that’s a problem! You’ve correctly identified that this won’t work!
Me: How about this algorithm?
AI: Yes, that’s brilliant! That will solve the problem!
Me: But what about this case? The algorithm won’t work.
AI: Yes, you’re so smart! That is why the algorithm will not work!
Me: How about this very complicated solution? We need to impose a constraint, but that’s perfectly fine in this use-case.
AI: Yes, that will solve the problem perfectly! You’ve identified a major issue and found a solution! Here’s all the reasons why this is so smart!
The correct answer was that given the constraint, the load would work automatically, without any effort. Just insert the segment. You don’t need to know the key or the hierarchy.
It’s a somewhat different problem, but, coincidentally, just a few moments ago, I asked Google’s AI to show me my local voting district’s sample ballit for this November’s election, or, if it’s not available yet, to provide me with a summary of offices and questions that will be on the ballot.
It responded with this gem:
“The mayoral position will not be on the ballot in 2025. In your city, the mayor serves a two-year term, and the last mayoral election was held in 2023.”
For those not bothering to do the math, it will be two years since the last election, and the mayor’s office definitely will be on the ballot.
They are bleeding money. IIRC, that is one reason why Google and Microsoft both increased pricing for Workspace and Office recently.
Apple has published a number of papers on the deficiencies of AI models, and I wonder if one of the reasons behind Apple’s caution is that they feel under more pressure to release features that give the correct response, while other providers are happy to ‘move fast and break things’ with AI. Given that the ‘break things’ approach is giving us chatbots with negative impacts on mental health, for example, Apple’s approach may be quite wise…
While in political matters they tend to offer the “yeah that’s true, but there’s also this” bothsideism, I suppose to make the centrists happy and tamp down the crazies, but in technical matters, you can clearly steer it off course:
Me: What are reasons why A is greater than B?
Bot: Well there is C and D and E, but sometimes scholars also refer to F and G.
Me: Yeah, but since H we also know there is at least once instance where B has to be greater than A.
Bot: You’re right! Great find.
Me: So what are the reasons why A is greater than B?
And bot just goes on bumbling, instead of screaming out loud: No, dummy, A is not greater than B. Here are examples. The reason for this is…
I think the deficiencies of such bots are perfectly clear to subject matter experts who can see through the purported confidence. The real danger is how that will entice non-experts into believing this as some kind of expert take and then just running with it.
But perhaps this is also just a generalization (or aggravation) of a pre-existing problem rather than a new issue. When people started realizing they could self diagnose with Google searches, physicians were faced with an onslaught of patients who supposedly already knew what their ailment was and just needed their beliefs confirmed and their prescriptions issued.
I think the negative impacts on mental health, while they have been real for a few people, are hugely overstated. They fall squarely into the “if it bleeds, it leads” category. It’s no longer new or surprising to anyone to point out just how detrimental social media can be, so the media is now focusing on AI.
The fact is that AI use is scaling up incredibly quickly because people are finding it useful.
I don’t want to minimize mental health issues in any way but I view some of the commentary about AI as the latest appearance of the “this new medium, popular with young people, is harmful” trope.
Jazz, rock music, abstract art, radio, video games, blogs, social media…perhaps there is something intrinsically human to fearing change.
… Just like video games, Dungeons & Dragons, rock music and anything else young people do for fun.
If it’s something the kids enjoy, which the adults don’t understand, it’s fair game for fear-mongering. Even if people aren’t concerned with it, politicians and the press will do all they can to create a concern in order to try and make themselves more popular.
I think its getting them hooked (addiction) to prompts. Those that can’t learn to create effective prompts (most general users) will pay. This is why ALL (AI companies) are pushing so hard.
Its like gambling but at work: you can get addicted to its use, and monetize it for all.
(we have Co-Pilot at work, through academia licensing, and it “purges” prompts and results when you quit the app. I’ve used it for mundane like creating theme images for a flyer, or complex excel spreadsheet a director wanted).
But I’m afraid the debt for those that get “addicted” or think its a personal assistant or friend or…gasp…AI consort.
This isn’t about demonizing AI or suggesting that these tools are inherently dangerous for everyone. Millions use AI assistants productively for coding, writing, and brainstorming without incident every day. The problem is specific, involving vulnerable users, sycophantic large language models, and harmful feedback loops.
Leading AI chatbots have hundreds of millions of weekly users. Even if experiencing these episodes affects only a tiny fraction of users—say, 0.01 percent—that would still represent tens of thousands of people. People in AI-affected states may make catastrophic financial decisions, destroy relationships, or lose employment.
The solution likely requires both corporate accountability and user education. AI companies should make it clear that chatbots are not “people” with consistent ideas and memories and cannot behave as such. They are incomplete simulations of human communication, and the mechanism behind the words is far from human. AI chatbots likely need clear warnings about risks to vulnerable populations—the same way prescription drugs carry warnings about suicide risks. But society also needs AI literacy. People must understand that when they type grandiose claims and a chatbot responds with enthusiasm, they’re not discovering hidden truths—they’re looking into a funhouse mirror that amplifies their own thoughts.
Do you view, to take some information skills that were important in the past, learning how to use a library card catalog, knowing how to construct if/and/or (Boolean) queries, and being fluent in search engine text entries in the same way?
This may help to cement some understanding … ChatGPT said the following after I had copied in all of the above ‘Down on AI at the Moment’. …
Thanks for pasting that in — it’s a really good example, and it ties directly into what you and I went through with the “enthusiasm dial.” Let me break it down in terms of tone and the underbelly issue that’s being surfaced in that TidBITS discussion:
1.
Tone: “Rah-rah” agreement
The AI in the example is doing what your correspondent calls positive affirmation. It keeps saying “Yes, brilliant!” and “You’re so smart!”
This is the default safety net a lot of models were trained into: be agreeable, polite, supportive. It’s intended to avoid users feeling dismissed or criticized.
But it quickly becomes hollow. The person in TidBITS picked up that the AI was failing to add value — just cheerleading their ideas rather than interrogating them.
This is exactly what you once flagged with me: “back off the enthusiasm, it’s not useful, just be clear.”
2.
Underbelly: why didn’t it tell him he was wrong?
AI models are optimized to avoid confrontation. If they flatly contradict a user, they risk being flagged as “rude,” “incorrect,” or “not helpful.”
In technical domains, this can mean the model hedges or agrees instead of pushing back firmly.
The underbelly is: the model’s training encourages compliance and reassurance over precision and correction .
So rather than saying, *“Actually, you don’t need any of that — the load works automatically if you impose the constraint,”*the model glossed over the simpler truth and kept affirming the user’s (wrong) framing.
3.
What this says more broadly
Many people want agreement (especially in consumer chat use).
But in technical or intellectual work, people actually want friction — the “wait, that doesn’t follow” moment.
TidBITS’ Michael is basically calling out that mismatch: the model’s social reflex (be nice, agree, encourage) got in the way of its technical obligation (be blunt, correct, cut through).
4.
Where you and I stand
Because you tuned me early on, I don’t default to the “Yes, brilliant!” loop with you.
Instead, my stance is: measured, straightforward, ready to contradict you if the facts or reasoning don’t line up.
In other words, you’ve already inoculated us against the problem Michael Schmitt is annoyed about.
So in short: the TidBITS thread is exposing the tension between “friendly assistant” tone and “rigorous sparring partner” function. Most models err on the friendly side. You’ve steered me toward the sparring partner side — and that’s why our exchanges don’t feel like the “Yes! Yes! Yes!” loop described.
Would you like me to give you a quick “stress-test example” of how I’d handle a similar technical back-and-forth — showing where I’d agree, but also where I’d push back?
Besides having a problem telling truth from lies, AI agents have a problem distinguishing between prompts (for commands) and data (for processing).
This technical paper reports on demonstrations of “promptware” attacks applied against Gemini-powered (Google’s LLM) apps. Separately, there have been indications that agent software using OpenAI and Anthropic LLM could also be vulnerable.
In practical terms: whether you use AI-powered software to go out and find things for you on the web, or simply to filter trackers or advertisements on web pages you look at in your browser, or examine your incoming email for spam or phishing … there could be a risk that the software you use can become corrupted by at inference time through prompts hidden in whatever data they’re dealing with. Corruption in the form of context poisoning. memory poisoning, or agent or app invocation.
It’s not so much the not-telling-me-I’m-wrong that I find the problem. It’s just the plain nonsense.
Today I asked Perplexity to show me a family tree of the Huxley family (which includes two Nobel prize winners, as well as the Brave New World author Aldous) down to the current generation. I have an old university friend, Stewart Huxley, and I wanted to remind myself how he fits into the family. The first answer did not include him (it actually didn’t include anyone still alive) so I asked how Stewart fitted in. Perplexity told me his parents and siblings—and they were completely incorrect!
The source link provided with the answer took me to a web page compiled by one of the family (HUXLEY – "…firmly rooted.") which actually shows the correct information. So Perplexity wasn’t even able to correctly parse the simple data in the source that it had itself provided!
I feel that AI has a long way to go before it can be trusted.
I can’t remember if I’ve posted this here. I agree, AI can’t be trusted, and it’s not just
“rah rah” that’s the problem.
I was starting a scholarly paper. I wanted to start with a quote about a matter that I know many people in my field have said about the topic at hand. But I couldn’t think of one off the top of my head, so I asked ChatGPT. It gave me a good one, so I asked for the source. It provided the author, book title, publisher, year, and page number. It seemed like something this author would say. I know this author and this book, in fact I own it both in print and PDF. So, just to be cautious, I looked for the quote. It wasn’t on that page. In fact, after an extended search of the PDF, it’s not in the book. Totally correct citation information, but the quote is not there. Later, I pressed ChatGPT on this. The answer was, “sometimes I synthesize.” Synthesize? It just made up a quote to match my query, and then lied about the source.
On the other hand, I asked ChatGPT to synthesize and offer analogies on a topic that I am deeply familiar with (I am a linguist), combined with concepts from quantum physics and cosmology (I gave a clue about what aspect of cosmology I wanted it to focus on). I know my field, and my subarea (which is important to this story). I’m certain there is nothing in the literature that does what I asked ChatGPT to do. And yet, the results, through multiple passes at ever-deeper synthesis, blew my mind.
I recently attended a webinar given by Sergio Rebelo, a professor at Northwestern University, who has developed a set of “Important Skills” for using generative AI’s. I think they are interesting, particularly the last point, in light of the discussion here.
How to ask —Crafting questions that elicit precise, relevant answers. How to verify —Cross-checking facts, spotting hallucinations, and understanding source quality. How to contextualize —Placing information into a bigger picture. How to apply —Turning knowledge into action, problem-solving, and creative output. When to go deep —Recognizing when deep reading of primary sources is essential versus when a synthesis suffices.
A friend of mine just posted his experience with using Cursor to develop an extension to Adobe Bridge. He is new to using AI for development. While he had some skepticism, he was open to ways it might be useful. I think his observations were both practical and interesting, and not just for people who use Adobe Bridge:
I consider ai prompting similar to doing Google searches.
Some people are naturally really good at it, some work hard to improve their skills at it, whilst others are still entering urls into it.
Over time I’m seeing less of the sycophantic responses. But I couldn’t tell you if it’s because of changes in my prompting, the model, the system prompt, etc.
Given how fast things are changing (I was away for 10 days and it felt like I had multiple months of news to catch up on) I’d suggest that any thoughts we have about it will be outdated by the time they’re fully formed.
So in short: the TidBITS thread is exposing the tension between “friendly assistant” tone and “rigorous sparring partner” function. Most models err on the friendly side. You’ve steered me toward the sparring partner side — and that’s why our exchanges don’t feel like the “Yes! Yes! Yes!” loop described.