I’ve been curious about something similar with only feeding it my DEVONthink databases.
Years ago I lived in Northern California during the Loma Prieta Earthquake. Power was out for 3 days. I went to a hardware store for batteries and bottled water (along with hundreds of others). They couldn’t take credit cards without electricity, so it was cash-only. But teenage clerk couldn’t do basic math to give back change. Like you’d give her a $20 bill and your total was $18.73 and she was completely stumped. It was so sad.
1 Like
The following is only indirectly related to Adam’s article, but seems to address similar problems.
I found the following article in the Italian newspaper “Corriere della Sera” and had it translated into English by DeepL.
Technology Review (Corriere della Sera)
The initial effects of AI use on university students: they are losing the ability to understand
WALTER QUATTROCIOCCHI
I have just finished one of the exam sessions for the Data Management and Analysis course, one of the courses I teach at university. Every year, this is a time for assessing students, but it also offers a unique insight into how their learning styles are changing. This time, however, I left the lecture theatre with a very bitter feeling that I had never experienced before. What was conspicuously absent was meaning.
Many students passed the exam without difficulty, some with excellent results. But a proportion – far from marginal – seemed to belong to a different world.
I saw students trying to read directly from the ChatGPT chat window during their project presentations, relying on the explanations generated by the model as if they were part of their own reasoning. I saw Pearson correlations used without understanding their meaning, linear regressions interpreted in a way that contradicted what the data actually showed, and even probabilities greater than one (for those not in the field, this is, by definition, impossible). These were not simply errors of preparation. They were errors of a different nature.
Up to now, the students’ errors have reflected an incomplete understanding. A formula applied incorrectly, a concept only partially grasped, a logical step that needed reconstructing. A single question was enough to get that process back on track and, often, the student would arrive at the solution independently. Not this time. The presentations were well-organised, the language fluent, the terminology almost always correct. However, all it took was to deviate from the script – to ask why one technique had been chosen over another, or what the significance of a statistical coefficient was – for the entire structure to crumble. The problem was not the fragility of the reasoning. It was its absence.
What struck me most was the way in which these tools had been used. They were not merely for writing the project. They were for studying. And this is where the issue takes on a completely different character. When a system designed to produce the statistically most plausible answer becomes the primary vehicle for learning, the risk lies not merely in arriving at a few incorrect answers. It lies in internalising a different way of constructing knowledge.
It is no coincidence that the projects thus all ended up looking alike. They followed the same structure, the same vocabulary, the same blatant absence of logic.
Rather than reasoning, they seemed to follow the statistical coherence of language. Plausible combinations of concepts which, at the first attempt at depth, struggled to hold up. In some ways, they were reminiscent of certain contemporary pseudo-academic debates: discourses impeccable in form, rich in the right words, but lacking the conceptual structure necessary to make them truly meaningful and full of gaps.
These tools were sold to us as technologies capable of democratising knowledge, breaking down barriers to learning and making everyone more competent. That promise quickly took hold in the language of tech companies, public debate and even parts of academia. What I had just witnessed, however, suggested a very different question: what happens when a machine designed to optimise statistical plausibility becomes the primary tool through which we learn?
It is precisely from this question that the concept of Epistemia arises. When I proposed it, it represented above all an interpretative framework: the idea that the large-scale adoption of systems based on statistical plausibility was altering the environment in which we construct knowledge, gradually shifting the focus from verification to plausibility.
At the MIT Media Lab, an experimental study has shown that the systematic use of ChatGPT in writing reduces cognitive engagement during the task and leaves measurable traces even in the ability to recall and rework what has just been produced. The researchers refer to ‘cognitive debt’: a cognitive debt that accumulates every time mental work is delegated without constructing an internal representation of the problem.
Another study, conducted by Microsoft Research on professionals who use generative AI systems on a daily basis, describes a complementary phenomenon. Critical thinking is changing its function: it is becoming less and less involved in constructing an answer, and increasingly focused on verifying one that has already been produced. It is a seemingly marginal shift, but it alters the very process of learning: when the generation of the answer is automated, the opportunities in which we build understanding also diminish.
Taken together, these studies tell the story of the same transformation: the environment in which we learn, work and attribute value to knowledge is undergoing a fundamental shift.
LLMs are probably one of the most extraordinary technologies developed in recent years. The misunderstanding does not concern their value, but the way in which we have chosen to describe them. We have built a machine of statistical plausibility and have ended up describing it as a machine of knowledge. We are constantly told that LLMs, almost messianically, democratise knowledge, lower the barriers to learning and make everyone more competent. These expressions have rapidly entered common parlance, featuring in tech companies’ campaigns, at conferences, in books and even in some academic debate. By repeating them so often, we have ended up treating them as descriptions of reality, when they were primarily a promise.
Continuing to portray these tools as something they are not is beginning to take on the dimensions of a historical responsibility. The price of this misunderstanding risks being an entire generation of skills going to waste.
A linguistic model constructs probability distributions based on sequences of words. Every response arises from an estimation of the statistically most plausible continuation given a particular context. It is precisely this ability that makes it extraordinarily effective. Plausibility, however, belongs to a different realm from knowledge. One measures the statistical coherence of a response; the other requires observation, verification, comparison with reality and the construction of explanations.
At first glance, this distinction may seem like a trivial matter. In reality, it concerns the very meaning of knowing. For centuries, words such as ‘understand’, ‘learn’, ‘explain’ and ‘know’ have denoted processes in which an answer was inseparable from the process that made it justifiable. Today, we use the same verbs to describe systems that produce results through a radically different mechanism.
Drastically reducing the cost of producing language is not the same as reducing the cost of knowledge. Quite the opposite, in fact. Billions of plausible texts are generated every day; what is truly scarce is the ability to distinguish between them, interpret them, verify them and attribute the correct meaning to them.
The promise of democratisation thus produces a paradoxical effect. Inequality is growing ever stronger. The new dividing line separates those who retain the tools of judgement from those who are gradually replacing them with a reliance on plausibility.
Meanwhile, the scale of the phenomenon continues to grow. Each new generation of models requires more data, more parameters, more processors, more energy and more investment. There is even talk of the need to build an ‘AI CERN’. It is a proposal that perfectly captures the historical moment: we continue to imagine that the answer lies in scaling up the technology, whilst the crucial question concerns the kind of cognitive environment that this very technology is helping to build.
It is this change that I have proposed to call Epistemia. Even before it is a technological transformation, it is a cultural transformation. A society can produce an almost infinite quantity of flawless texts and, at the same time, progressively lose the ability to distinguish an explanation from a plausible sequence of words. Every time we delegate the construction of an answer, we also delegate part of the process through which that answer acquires meaning.
The history of knowledge coincides with the history of the tools that have expanded our cognitive capacities: writing, the printing press, the telescope, the computer. LLMs belong squarely within this tradition. Every major technology, however, also alters the cognitive environment in which it is adopted. The real question, then, is not about how powerful these machines will become. It concerns the criteria by which we will continue to distinguish what we know from what merely appears plausible. Much more than the success of a technology will depend on that answer. Looking back on that morning, I still have the same impression. What was conspicuously absent was not merely a sense of purpose in a university lecture theatre. It was the very meaning of knowledge itself.
*Walter Quattrociocchi is Professor of Computer Science at La Sapienza University in Rome
5 Likes
Sadly that is nothing new. I gave a cashier $5.25 for a $4.24 charge. She unfortunately predicted I’d just give her a $5 so she plugged that in and hit enter too soon. As one who’s dealt with cash since I was kid, this was an easy one and I wanted the dollar back, not all the change. I explained to her how to figure it out. She had to call the manager who rolled his eyes and said “she gets $1.01 back”
And I said that decades ago when those fancy cash registers came out that calculated the change for you. “Pretty soon we’re going to have people how can’t figure out how to make change”
Didn’t take long.
I used to waitress and did cash out of apron pockets. In the dark.
(yes the equivalent of “walked 5 miles to school in the snow uphill in both directions” 
but it’s true)
5 Likes
My favorite experience was at a crowded fast food restaurant where the system the registers ran on was down. They were doing ok with calculators, but while waiting in line I could see they were laboriously totaling the bill, then manually consulting a paper chart for the sales tax, then adding that to the total. When it was my turn, I told the teenage clerk, “Why don’t you just multiply the bill amount by 1.0775 to get the total with tax?” (The sales tax rate was 7.75%.) The clerk just looked at me wide-eyed, but tried it, checked it against the chart and exclaimed out loud “Oh my god…that works!” He even called the manager over to see, and they both thanked me as if I had just given them the formula to turn base metal into gold.
8 Likes
Appreciate that! Yeah, I experienced that a number of times in the past, even with the electricity working! It was also sad that a number of them had problems processing a check for payment,
But it still goes on today. I experienced it the other day cashing in my (few) chips at a casino. I have $23 in chips, and to make it easier for the cashier, I also gave her $2 (total of $25). Man, she had difficulties processing that!
Luckily, where we get gas for our cars, paying with cash works out well. The cashiers seem to know what they are doing.
But yeah, really sad!
That is what I did at the casino. Gave her $2 cash, so along with the $23 in chips, the amount she needed to give me was $25. But STILL had issues processing that! I even said that to her, that I was just trying to make it easier.
One other thing folks are forgetting about this AI business (same with on line shopping (folks even order food!)) is exactly what my wife says: folks will become more and more isolated, having very little social life, and hardly ever going to s store any more. Again, a robotic society.
Great story! If you really want to blow their minds, show them how to back out the tax. ;)
Small business owners do this all the time. Give a nice round number to the customer but they still have to pay the tax to the state so it’s got to be entered on the books the right way.
Great story! And it says it all. I actually have witnessed where folks cannot figure out what 10% of something is! Besides 1, that is the easiest multiplication problem to do. Yet they still go to their phone, start up the calculator, and perform the simple arithmetic. And what’s more ironic is that here in Washington state, the sales tax is around 10%. Even then folks can’t figure out the final cost (it’s just about the original price!). And god forbid figuring out what 15% of some thing is. Actually a simple problem, as 15% = 10% + 5%, and 5% = 1/2 of 10%.
As Tom Hanks said in “Forest Gump”, “Stupid is as stupid does”.
2 Likes
I don’t agree memorizing that 2+2=4 or 8x4=32 is going to save—or not save, in the case of using a calculator to do arithmetic—civilization. I think having the ability to think critically and the maturity to have empathy for others are much more important.
4 Likes
I’m convinced one of the reasons so many of my fellow Americans are financially illiterate is that they cannot do simple math in their heads. If you believe you need a CFP and a spreadsheet to figure out that 22.9% interest on your 5-figure CC debt is absolutely 100% no questions asked going to kill you, you’re doomed. The best way to spend within your means is to know what your means are. And where all your spending goes. If you need Excel and Quicken and TurboTax and all that other mumbo jumbo to figure these basics out, you’re already in trouble.
People who can do basic arithmetic in their heads are people who can always easily estimate if they are being suckered. It’s not about being able to split the check without a calculator. It’s rather that those people who have the ability to do that without a calculator are the ones who don’t get taken advantage of and move onwards and upwards.
thanks for elaborating on this. I was also negative on AI agents but I hadn’t articulated all the reasons. Very useful to have your post and everyone’s comments on it.
My Ph.D. thesis was Goal Processing in Autonomous Agents, many years ago. I don’t see modern AI agents handling what my work pointed to as problems needing solutions – and I didn’t even look at social requirements.
I’m relieved that I don’t need to learn much more about modern AI agents yet.
Here’s a long, more general (dread-generating) critique of LLM-based AI:
(His Longer PDF here worth reading). I do appreciate many uses of AI, however. I.e., I’m not as skeptical as Kingsbury, though I do appreciate a lot of Kingsbury’s skepticism.
And here is David Sparks’ take, published yesterday, on Apple’s AI strategy:
which is a worthy topic in itself. I asked ChatGPT what it thinks of the latter article and it was generally favorable within limits, claiming (in a nutshell) that server-based AI will still be required for the most knowledge-intense queries.
I agree with you Adam.
When I was working, my job had me dealing with people who had trouble expressing themselves and often trouble understanding what other people were saying to them. I had to deal with each individual as an individual. I do not see AI doing that. One size does not fit all.
I also often have had (and still have) to deal with people on an emotional/spiritual level. Again this is something AI can’t do.
AI also scares me because I see over dependence on it, causing people not to develop their own critical thinking (which I believe is needed even more with AI out there). These are just my thoughts.
The problem is currently, that major companies are worried that there are all these things that you might want, and they are worried that they will be sitting there watching the crowded in a different direction.
Not inconceivable, although it depends on how much you want to hand over to it. I see this even with human coaching, where there’s tension between what the coach has programmed and what the runner feels like doing on any given day, or is invited to do by a visiting friend, or suddenly has a conflict.
On further reflection, I think this may be a somewhat different situation. You’re really talking about coaching, not an assistant. A coach directs your behavior, whereas an assistant takes on tasks you don’t wish to do. Well-trained AIs could do pretty well at coaching in a variety of scenarios.
I agree—I’d be interested in having something to backstop me and help me be better at the things I already do and that I don’t want to or can’t delegate. I could also imagine an AI agent watching the traffic in and out of my computer and querying me about things that looked concerning—the volume and sophistication of security attacks makes me believe that most people stand a chance only with local help by a capable agent.
1 Like
If I ever let an AI agent take my calls, it’s going to have to start any response with, “Hi, I’m Eddy, your friendly purveyor of personalized AI slop,” with apologies to the late Douglas Adams.
1 Like
Adam writes:
Among much else, AI agents are supposedly going to:
Schedule meetings and appointments
Make restaurant reservations
Buy event tickets
Book flights and hotel reservations
Research and buy products
Respond to email, messages, and social media posts
Plan group events with friends
Deal with customer service disputes
In doing so, they will be negotiating with other AI agents set up to do the same thing, and thus cut us out entirely from our own lives.
1 Like
AI reminds me of the “saying”, “It’s a solution searching for a problem”.
As I stated above, we’re becoming a robotic society. It’s been slowly building up to that, and AI will just speed that along.
Your article exactly matches my thinking. I couldn’t trust AI agents to do anything for me.