AI Answer Engines Are Worth Trying

Thanks for this Ron.

My very limited use case was related to a query about a fairly complex medical situation for which I had >30 different data points including clinical lab values.

I appreciated Adam’s response of “Your example, while compelling, is just standard chatbot behavior these days because it doesn’t require access to information retrieved from the Web” yet I can see I did a poor job of presenting my(n=1!) findings that Grok was superior in accuracy and clarity of presentation over other chatbots (ChatGPT, Copilot, Perplexity, Gemini) to which I submitted the exact same query. (some of the data calculations by Perplexity and Gemini were clearly hallucinations)

I ran all of the results by my Internist and Cardiologist and a particular insight offered only by Grok has subsequently helped them fine-tune an important aspect of my treatment. I thought my offering might be of some value to others and it was with this intention (and unexpressed caveat) that it was offered.

But that’s not what you’re addressing.

For me, and I’m not saying anyone else should be like me, all “data” everywhere and all the time is to be called into question including content from the articles you cited. I am forever biased from my psychoanalytic training which has taught me that no one is aware of what they don’t know aka…the problem with the unconscious is…it’s unconscious!

Most of us much of the time are operating from unconscious conditioned complexes which contaminate our beliefs. Our egos don’t like to hear this and will diminish and deny others not in alignment with those beliefs which often results in exclusionary behavior.

I prefer not to exclude those who may carry aspects of my shadow which does not equate to an endorsement. YMMV and I support and respect your’s and everyone’s right to do it their own way from whatever state of consciousness they can muster!

Have a lovely weekend!

3 Likes

Well here is the data. AI answers reduce clicks by about 2/3. There are other studies out there as well, but all seem to show that clicks are reduced by at least 1/3. It is clear that interacting with search will be fundamentally different.

Fascinating data, though it feels highly specific to Google’s current interface for the AI overviews. I suspect other interfaces would likely reduce click-through even more.

I was also struck by how “traditional” the example searches were. That’s how you use Google, but as I found when researching the AI answer engines, it’s more effective to ask for what you want to learn directly, rather than trying to guess keywords.

And, of course, the real question here is what the goal is. The goal of the person doing the search is to get their answer, not to provide click-through eyeballs to advertisers on the sites hosting information related to the search. So, another way to interpret these results is that AI overviews better meet user goals two-thirds of the time.

1 Like

Google has now promoted AI Mode to the main interface.

I see they somewhat color ChatGPT with the “Browse with Bing” annotation…

I like Tufekci’s term “plausibility engine” for LLMs/generative AIs.

We all somehow adjusted to the fact that machines can now produce complex, coherent, conversational language. But that ability makes it extremely hard not to think about L.L.M.s as possessing a form of humanlike intelligence.

They are not, however, a version of human intelligence. Nor are they truth seekers or reasoning machines. What they are is plausibility engines. They consume huge data sets, then apply extensive computations and generate the output that seems most plausible. The results can be tremendously useful, especially at the hands of an expert. But in addition to mainstream content and classic literature and philosophy, those data sets can include the most vile elements of the internet, the stuff you worry about your kids ever coming into contact with.

Zeynep Tufekci - New York Times

While I’m unsurprised that Musk’s opinions may be granted undue importance in Grok responses, this whole thing does feel manufactured. A fake Twitter account, people asking leading questions, etc.

And more to the point of this article, it doesn’t appear that this situation has anything to do with enhancing Web searching with generative AI applied against search results. Whenever a response that doesn’t include data from the Web seems off, I strongly recommend saying, “Confirm with a search.” In my experience, that usually eliminates the confusion or mistakes.

2 Likes

I took up your suggestion. ‘Prompt Engineering for ChatGPT’ was very useful, but even I could see that it’s already out of date. Worthwhile, but out of date.

Interestingly, I just had a rather complete fail with ChatGPT. I have an old Canon Pixma IP110 portable printer that needs a new battery, so I went looking on Amazon. Searches for my printer name and “battery” turned up the LK-72 battery from Canon, works with the newer Canon Pixma TR150 printer. So I asked ChatGPT if they were compatible, and it was confident they were. But when I looked at the product pages for the LK-72 or third-party compatible versions, none listed the IP110 in the compatibility matrix. So I pushed harder, and eventually realized that ChatGPT was being confused by the fact that Amazon and eBay product pages often list many other related products and since the printers were similar, they listed products like ink cartridges for the IP110. That’s when I thought to check the physical dimensions and look at the actual photos, which revealed that they’re different sizes and have different connector orientations. So, not compatible.

However, Perplexity and Gemini and even Google’s AI answer all got the answer right. Sometimes you’re winners, and sometimes you’re losers…

1 Like

And it drives me nuts. Amazon offers to auto-complete my search term for the exact item I’m seeking, and the resulting page does not contain the item that Amazon offered in its auto-complete. Argh.

Sorry for the digression. I’ll curtail the rest of my rant, which would otherwise be lengthy.

3 Likes

If Amazon were to offer a new state-of-the-art AmazonChatBot™ that presented you with exactly the product or the products meeting exactly the specs you specified and nothing else, now that would be a world-shaking AI development. :smiley:

Dave

5 Likes

Yes, please.

It is still the best introduction to working with AI I was able to find. I used both AI and traditional search to find alternatives, and where trials were offered, I followed along with the free content.

I had a hard time writing this sentence so I asked AI to help me: “I have to admit that I excluded some courses due to either the presentation style or the variety of English, which, although comprehensible, required significant cognitive effort to process.”

The great thing about ‘Prompt Engineering for ChatGPT’ is that everything you learn still works and it get you into the right headspace for working with AI.

This isn’t specifically related to that article or to agentic web browsers, but I did recently use Amazon’s “Rufus” AI to help me in my purchase of a new iPad. I found it quite helpful, though not perfect.

So my situation is that I had a very old 13" iPad Pro (2018) that basically can’t be upgraded any more. It has Lightning connector and other limitations, such as the Logitech keyboard case I used with it breaking (2nd one – Logitech replaced the first for free). Since I only use this iPad for reading in bed at night, I haven’t been too bothered by the limitations, but I have considered upgrading my other iPad and “downshifting” it to night duty and getting rid of this old one.

Recently Apple announced the iPad M5 and that intrigued me, especially with Amazon having it on sale for Black Friday. But one reason I haven’t upgraded in the past is that a new iPad meant upgrading all my accessories: my Apple Magic Keyboard, Pencil, etc. wouldn’t work with the new iPad. Buying all of those again would be an extra $400.

Since I don’t buy iPads often, it’s hard for me to remember which Pencils and cases and keyboards work with which. The Pencils all sound like the same thing, but are very different, and each only works with certain iPads. Cases are a horror show of compatibility issues as they depend on the iPad size, type, and year. I got bogged down in the weeds trying to find compatible accessories and trying to figure out exactly how to proceed with the upgrade.

I was about to just give up, but I decided to try using Rufus to clarify exactly which Pencils and accessories worked with which devices. It was very helpful and would provide me with exact Amazon links for the recommended item. Once or twice there was some confusion where the AI contradicted itself, and I was a little worried it might be lying to me. But I found that if you ask the same question in two different ways you’ll get a more accurate answer. (It really is no different from talking with a salesperson at Circuit City back in the day, where the sales guy will tell you anything to make a sale, and you have to call him on it. “Why do you say this VCR has four heads when the box says it only has two?”)

Anyway, I ended up making the plunge, buying a Magic Keyboard clone from a third party manufacturer for $99 instead of Apple’s $350 one. I got a new Pencil on sale, too. I ended up springing for the 13" iPad M5 as my current one was the 11" where I wasn’t entirely happy with the size. Everything came and worked just as advertised, and really love the screen real estate of the 13". I moved the 2021 11" down to serve as my night iPad, where it works just fine and still gives me access to Apple Intelligence and other newer features. The old 13" iPad I sent to Apple as a trade-in (they say I’ll get about $100 back).

BTW, the third party keyboard case works great – the same “floating” magnetic design and the only difference between it and Apple’s is that it connects via bluetooth instead of the special connector on the back, so this keyboard has to be charged separately (via USB-C). But it lasts for weeks on a charge and connects in less than 2 seconds, so that’s not a big deal.

I’m delighted with everything and rather impressed with the Rufus AI (your mileage may vary).

1 Like

If you have a few spare minutes and are interested, I’d be curious what you’d see if you asked ChatGPT, Perplexity, Claude, and Gemini the same basic questions. The issue is whether Rufus has some special Amazon knowledge that was helpful, or if any of them could have done it with access to the Web.

I haven’t had a chance to do that yet, but will sometime.

One fascinating thing I just noticed a few days ago is Rufus – Amazon’s chatbot – actually gave me purchase links for sites that were not Amazon!

I was looking for a quirky tee-shirt as a Christmas gift for a relative and it couldn’t find one like I wanted on Amazon and suggested some non-amazon sites.

Not really what I wanted, as I prefer the safety of Amazon for buying, but I was shocked Rufus did that. I had assumed it would be trained solely on Amazon products. I’m not sure why it does that: perhaps Amazon was worried they’d get anti-trust scrutiny if they only supported their own site? Or maybe Rufus is based on a general-purpose LLM and they couldn’t stop it from searching for things off-Amazon? Seems odd either way.

Here’s one I did with Google Gemini.

The task was to find what vintage Japanese drawing software was shown in the image.

To find it I uploaded the image to Google Gemini, asked the question, then had a short conversation with it during which I guided its assumptions, and pointed out the options for text outline and shadow. It then suspected it was Magic Palette, which I confirmed manually by doing a Google image search.

Chat transcript
https://g.co/gemini/share/a329328c3ba3

I then found the exact scan that had been cropped down to the image, in an old Japanese computer magazine on Internet Archive.

2 Likes