I don’t think any reasonable person is worried about any of that.
But there is a lot of concern that these tools reinforce and cater to some of the worst human behavior, just like social media before it (which is also “just a bunch of code” and algorithms).
Just watch Apple’s recent AI TV ads (here, here, here, here, and here). The key use case Apple itself boasts about essentially comes down to deception intended for lazy people and egoistical posers who have zero work ethic and care little about their family members and friends. Who wants that kind of behavior enforced? We have more than enough of that going around as is.
What’s missing here is emphasis on how these tools can be used to support good behavior and suppress bad impulses. But none of that is apparently sexy enough or being discussed, hence so much skepticism.
If even Apple can’t come up with better showcases, no wonder folks are pushing back. And that’s before I, as a scientist and journal editor myself, even get into yesterday’s Apple “scientist” in a lab coat having AI write her journal article for her. We actually require authors who publish in our journal sign that they don’t do that. That has become quite common in my field actually. I hope Apple is just being tone deaf here and not actually deluded.