Some Thoughts about Apps, Bugs and Testing

jweil · May 4, 2024, 11:57pm

After dealing with a tech issue involving bugs I came to a realization today.

Most anyone who utilizes technology today in the form of programs, apps, and hardware is most often a volunteer unpaid beta tester. Even more disconcerting is the fact that sometimes we pay the provider to do this and they may even profit from it.

Allow me to explain. Proper testing and SQA costs time and money. For many companies this is an area that often gets short changed for increased profits as it is somewhat hidden from the eyes of senior management and the customer or user (sorry Adam). The individuals that become aware of the consequences of this is tech support and the person using the products when issues arise from using it. This is why many companies have addresses and forms for victims of these issue to file a report so that they can catalog the issues, triage them, and eventually fix the ones they feel need fixing. So when you are using these products and find issues and file a report or call tech support about them to report them you are essentially doing what company employees that test products are paid to do. Except you are volunteering or paying for the privilege to do so. The way you are paying for this is utilizing your time to try to figure out what is going on before calling support which may have a support fee as well. Additionally if you purchased the product you are also paying for it through lost time, and filing reports, if you do so, for something that you have paid for that you assume was properly tested and will properly function.

As such when we discover tech issues and if we report them, we are essentially volunteer beta testers and sometimes paying for the privilege to do so.

So why does this and how does this happen? Because we allow this to happen.

When bugs occur we want them fixed so we report them without requesting compensation and put up with it.
Buried in most licenses is a statement that the product is not guaranteed to work properly and that the company is not responsible for and consequence of it failing to work properly. We accept that because we have no choice if we want to use the product and it seems to be standard operating procedure throughout the industry. Additionally what many do not realize is that when you purchase a product you likely are not purchasing the actual product but only the right to use it. That is in the licenses of most products buried in the fine print that almost know one bothers to read. If you are skeptical about this, I invite you to take most any product, particularly software and actually carefully read the license or warranty that you agreed to. In most all cases you will find this statement or similar one. That is one reason that most warranties are now titled ‘Limited Warranty’.

In such comments I like to discuss options and workarounds. Unfortunately with this issues they are few and far between as such tactics are now ingrained in our culture and society. The only things that come to mind is to not use or purchase the product which likely is not practical to do and to contact your government representatives about this to request new laws to restore consumer protections and rights and that existing laws that are applicable to be rigorously enforced.

However if others who are reading this have any other constructive suggestions to address this, I invite them to offer them in comments.

Ratsima · May 5, 2024, 12:23am

I’ve been a “volunteer” tester for Mac Mail ever since I purchased an M1 MBA about a year ago and continuing when I bought an M2 MBA late last year.

On both machines Mail periodically insists on using Port 25 for SMTP TLS authentication. This almost always fails because the port should be 587 which usually succeeds.

READ May 04 15:26:39.106 [kCFStreamSocketSecurityLevelTLSv1_2] 
-- host:p56-smtp.mail.me.com -- 
port:25 -- socket:0x6000014f1440 -- thread:0x600002e0f440
501 5.0.0 AUTH aborted

READ May 04 15:42:58.114 [kCFStreamSocketSecurityLevelTLSv1_2] 
-- host:p56-smtp.mail.me.com -- 
port:587 -- socket:0x60000143a880 -- thread:0x600002f899c0
235 2.7.0 Authentication successful

I’ve been dealing on and off with Apple support on this issue for nearly a year now. I’ve communicated by email with three different senior support people all of whom gave up and stopped responding to email. I have submitted Connection Doctor logs, Capture Data logs and Screen Recordings. I never received a response on any of the above.

I’v’e pretty much given up hoping that there might be a solution to this. Fortunately, there is a workaround. Use a VPN. When I use a VPN Mail starts using the correct port (587), the TLS authentication succeeds and the mail is sent. I have no idea why. Neither does Apple.

And, yeah, I’m a bit weary of being an unpaid tester.

As for a solution to the general problem? I don’t think there is one. Major bugs that affect users in general will be addressed. Minor bugs that affect only a few will be ignored.

That’s just the way it is.

Quantumpanda · May 5, 2024, 1:44am

The problem will not be addressed until and unless some major tech company decides to accept that profits are secondary to making things that work. And that will not happen with any company that has public stock, because most stockholders will fight any attempt to stop seeking ever-increasing profits.

This ultimately is a problem not with just the tech industry, but with capitalism as it is practiced in the US and globally by US-aligned companies. Social responsibility and customer satisfaction take a back seat to the ever-greater hunger for profit. It’s the same problem that is destroying American health care, ruining the housing market throughout the country, and preventing US workers from being compensated fairly. Under this model, profit, at the expense of all other considerations, must continue to increase.

The US government will not resolve this problem. Politicians are too dependent on corporate lobbyists for campaign funding (and other ancillary, sometimes unofficial, “funding”). While there are a number of elected officials who are genuinely motivated to make changes like this, there are not nearly enough of them in the right places to push them through.

If some large corporation in a major industry decides to buck the trend and repudiate the toxic form of capitalism the nation has fallen into, then and only then might things change, because if they continue to be successful despite rejecting the pure-profit-growth model, others may follow. But it will take a brave and persuasive CEO to persuade a board of directors to allow such changes.

There are smaller companies that are successfully pursuing such changes, but so far none that are large enough to make other multinational corporations take notice. If the example set by Dan Price with Gravity Payments in 2015 wasn’t enough to convince businesses that investing in your workers pays off many times over, it’s going to take someone at the FAANG level (or its equivalent in another industry) to make other companies consider doing something like this.

jweil · May 5, 2024, 2:49am

Unfortunately while I agree with most of what you stated, I must disagree about major bugs. A month after I received my MacPro 7.1 (2019) desktop in early 2020, it started crashing most every time I shut it down. Due to my background of 13 years a Apple, both as an employee and contractor, doing testing and support, I suspected the issue was macOS after about a month of this. Instead Apple Support continued to blame me for my Apps and hardware, including a disk drives installed internally that were purchased from Apple at the time I purchased the machine. Until September 2023 I spent over 400 hours of my time, and over 100 hours on the phone for support with endless erase and reinstalls, removing Apps, hardware, extensions, and menu bar items to no avail. This included Apple first replacing the guts and then the computer just to get rid of me. Also during this time I was unable to get Apple to conduct a detailed examination of the crash logs. Last Summer I updated to Ventura at Supports insistence. That made no difference. Then magically with a Ventura update last September the issue completely disappeared and has not returned. The bottom line of all of this is that Support refused to properly diagnose the issue, blamed me for the issue, wasted my time, failed to do a proper diagnosis, and took a shotgun approach to address it including replacing its guts including the graphics card, then a $5000 computer rather than taking the proper steps to properly diagnose the issue and fix macOS 4 years ago.

Ratsima · May 5, 2024, 3:04am

That’s pretty much my experience with Apple Support. Perhaps I didn’t explain myself well.

Quantumpanda · May 5, 2024, 3:12am

There are an awful lot of people in customer support, both line reps and leaders, who think that withholding assistance is somehow better for the company than trying to help. Part of this comes from the whole profit thing I mentioned above, but I think some of it is just straightforward sociopathy and/or psychopathy.

I’ve worked a wide variety of phone-based support jobs, both inside and outside IT, and I’ve encountered this attitude in every single such place I’ve worked except for one very small company where I was literally one of only two reps. (Technically, I encountered it there too, because the company’s sole developer was openly contemptuous of users, but he wasn’t directly part of the support communications with customers, so his attitude wasn’t shared with them.)

Social responsibility and compassion simply aren’t part of the skill sets being taught in by the majority of the US school system, at any level, and they’re not a priority for most businesses when looking for or training employees. If you don’t start with them or get them instilled by your parents, you’re probably not going to care about any of the customers you’re being paid to assist. It doesn’t help that a lot of customers (not necessarily a majority, but enough to cause a major amount of stress) seem to go out of their way to make it hard to care about their issues, which then carries over into how CSRs treat other customers.

silbey · May 5, 2024, 2:44pm

I think we forget that this is the case for all technology. What we buy is almost always flawed and doesn’t work in many of the ways that it’s supposed to. It’s not particularly new and it’s not unique to software and tech. Nearly 35 million cars were recalled last year by the NHTSA for defects of various sorts, and those were only the ones that rose to the level of government attention. My 2018 Honda has a number of features that don’t quite work right (eg, the HVAC system only has vents in the front of the car, so the person in back is usually too hot or too cold; the windshield washer sprayers go out of alignment frequently; the washer fluid reservoir is just less than you get in a bottle, always leaving me with leftovers). None of these rise to dangerous levels, but they’re not working to some degree. They’re bugs.

My stove had keypad to enter temperatures and the zero button broke fairly quickly. Replacing it would have cost about as much as the stove was new, so we started cooking things at odd temperatures (399, 351, etc). Again, I could work around it, but it’s not quite right.

Go back however far you want and there’s similar levels of defectiveness in our day to day technology. We’ve always been beta testers for products and we always will be. We just have the language to describe it now.

xdev · May 5, 2024, 3:29pm

There are two recent changes that make this worse, however. One is just the pace of development. What used to take years is now released in months – that leaves no time for testing.

Two is that as a society we’ve become accepting of these bugs. The people on this forum are more technical and we understand the bugs don’t have to be there, but average people don’t think that way. They think bugs are inevitable, so they don’t complain. Think of the kids who’ve had smartphones all their lives. They either ignore or workaround the bug, wait for the next update or replacement cycle (which isn’t far away), or just switch to another app or tech. It never occurs to them to complain or stop buying stuff (which would pressure the maker to release better products).

Look at the failed releases of Humane Pin and Rabbit R1. Both are incredibly buggy to the point of uselessness. Yet somehow both companies felt it was okay to release now? How is that possible? Does no one at either company even use the products? Are they so enamored with their products that they didn’t see the flaws? Are they numb to the problems because they tested it when it was really bad and now it seems much improved?

For products like gadgets that are relatively harmless if they don’t work, it gets really scary when things like cars – today computers on wheels – are released without proper testing and full of dangerous bugs that could cause fatalities. The future doesn’t look like it’ll be any better, and probably a whole lot worse.

silbey · May 5, 2024, 4:02pm

Neither of those things is remotely new. Products have always been rushed to market with not enough testing and people have tolerated the bugs that resulted. The 1965 Chevy Impala was developed quickly to hit the annual release timeline (sound familiar?) and testers failed to discover that the engine would occasionally impact the throttle linkage causing the car to race forward out of control. Chevy sold the car and people drove it for FOUR YEARS before the government issued a recall. Cars have always been “full of dangerous bugs that could cause fatalities.”

dave6 · May 5, 2024, 4:23pm

First off, I have to differ on the issue of whether bugs “have to be there”. There’s an old saying that every non-trivial software program has bugs, and I’ve never seen any evidence to the contrary. So the number of bugs can be reduced but the complete elimination of bugs is a pipe dream.

Never forget that QC costs money. If you’re dissatisfied with being an “unpaid beta tester”, remember that paid beta testers will be reflected in the cost of the software. (The idea that additional QC costs can be covered out of corporate profits without price increases is another pipe dream.)

How much more are you willing to pay for software from companies with more extensive QC processes? Are you willing to pay for an annual license to support continued testing and bug fixing against OS updates or other changes outside the developer’s control?

I suspect what we have for most off-the-shelf software is an equilibrium, where the level of QC is commensurate with the willingness of customers to pay for it.

I work in drug development, and we focus a lot on the concept of risk-based quality management. The decision to perform a particular level of QC is based on an assessment of the possible damage caused if defects are not detected and corrected. The level of QC on a system that cannot be easily updated or whose failure could have life-threatening consequences is going to need to be much higher than on a system where bugs annoy users and software updates can easily be pushed out.

This can apply to both systems within a product and the product as a whole. So in my Civic, I occasionally have to reboot the screen/ audio system to get CarPlay working right. But the car is 100% “drive-by-wire” and I’ve never seen a single issue with the functioning of the accelerator, brakes, steering, or other components involved in operating the car. I can almost guarantee that Honda has a more extensive QC program for this embedded driving software than for the dashboard screen software. (I remember hearing that most car screens are based on some sort of customized version of Android.)

Similarly, medical device software goes through extensive validation; I’m sure my CPAP is completely software-driven, and I’ve never encountered a software problem with it. But this is reflected in the price of the device.

Short version: QC costs money; There’s high quality QC when circumstances demand it; and for most software we purchase, including our operating systems, we’re getting a level of QC consistent with what we’re willing to pay for.

Dave

jweil · May 5, 2024, 7:08pm

Much of this I agree with but take exception to some of it. Given the complexity of software I concur that their will always be some bugs in it. But that should be limited to class 3 and 4 bugs, not class 1 and 2. This can result in loss of data and should be found and fixed before release. Additionally companies should focus of creating a reasonable stable of their product as opposed to adding new features to unstable products. Software release times should not be based on the clock, ready or not, or on a competitors expected release, but released when it has been properly tested and ready for release. Some just don’t get it in their efforts to secure short term profits that releasing unstable products full of serious bugs results in extreme pressure on tech support and frustrated customers who often can’t reach it in a timely manner thus increasing support costs, stressed and dissatisfied employees and fed up customers that may seek alternatives or resist any upgrades, especially paid ones if what they have is finally working for them. Then there are the additional issues that some companies, including Apple ,essentially lie to customers blaming them for the companies software or hardware issues, simply ignoring the issues for years, which is what happened to me, or treating them like mushrooms - i.e. keeping them in the dark and feeding them manure {<-trying to keep this politically correct and not having the reply deleted}.

The bottom line of all of this is that doing proper SQA and taking the time to do it correctly may be less costly in the long run. Some companies only figure this out when forced into bankruptcy or have their CEO terminated for releasing a defective products.

Need I mention Boeing?

Another Example: many years ago, at least 20, when LED’s first appeared, I replaced all my Christmas lights with just released LED lights from Phillips which were quite expensive at that time. Some of the bulbs started burning out during the first season and many continued to do so in the second and subsequent seasons. Phillips response to this was to send me a handful of replacement bulbs but denied anything was wrong with their product. Since then I switched shavers from Norelco (Phillips brand) (rotary) to Braun (traditional), replaced all my Phillips brands with competitors when they wore out, and have knowingly purchased a Phillips or Norelco branded product since then including light bulbs and Super-Automatic espresso machines costing over $1000. Given their behavior and releasing defective products without taking responsibility for them I refuse to support this behavior and profits by purchasing their products. This decision may have saved my life as in the last few years they have sold CPAP machines that had materials in it which broke down which injured or killed their customers. They were recently ordered to pay a Billion dollars to customer base who purchased these machines. Since I use CPAP my decision not to buy Phillips products may have saved my life as there are one of the most popular brands.

Paying attention to corporate culture and basing purchase decisions based on it can yield positive outcomes.

jweil · May 5, 2024, 7:24pm

One of the most recent examples of this is Boeing. I would not be surprised if the long term outcome was bankruptcy. Just because you have an almost virtual monopoly on a product, in this case, large commercial aircraft (Airbus is the only other competitor), does not necessary provide shelter for bad behavior resulting from the paradigm of maximizing short term profits as the mission statement of the company.

Apple, are you listening?

bwhite · May 5, 2024, 7:51pm

Since you don’t mention trying it, if you haven’t already, uncheck the box in your account’s Server Settings tab for “Automatically manage connection settings”.

I’ve found for myself and a number of clients over the years that simply unchecking (and keeping it unchecked) that box solves similar issues in Mail.

silbey · May 5, 2024, 8:48pm

Many more people died in Boeing plane crashes in their supposed golden age (ie, pre McDonnell Douglas merger) than have died since (it’s about 2.5 to 1). If that’s what maximizing short term profit does, I’ll take it.

kat634e · May 5, 2024, 9:04pm

A friend of mine who is not tech savvy ended up throwing away a 2017 iMac last year, after many months of stunningly poor performance: 10 minutes to boot up, every action taking minutes rather than seconds, etc.

I went over to have a look at it, and discovered that Apple had shipped that model with a 5400 rpm spinning drive, something I consider ridiculously cheap and shortsighted. Unfortunately, she had repeatedly called Apple Support, and was told EVERY TIME to upgrade the OS. Not one of them apparently had enough knowledge to recognize what installing one OS after another that requires an SSD to be really functional was not going to solve her problems. I suspected that her internal drive was corrupted or failing, but we would have needed to boot from an external drive (that she didn’t have) to run diagnostic software (that she didn’t have) to investigate the health of the drive.

I started to set up such a drive, but before I could arrange to give that a try, she had just junked the iMac in frustration. She uses an iPad instead and will probably never buy another Mac.

Ratsima · May 5, 2024, 9:50pm

Interesting you would suggest that. Some time ago I tried that and even manually set the SMTP port to 587, but Mail still insisted on using Port 25 in spite of the manual setting.

However, I can’t try it again. Here you can see that all of my mail servers are listed:

But when I click on “Edit SMTP Server List”, both Google and iCloud are missing. No idea how this happened.

Any idea how to get them back?

[edit] - I just recalled how this happened. One of the Apple senior support people suggested that I remove and reconfigure the iCloud server. I did that and it appears in the server list, but no longer in the edit server list.

jweil · May 5, 2024, 10:07pm

Not surprised. I rest my case!

Update: The current price on Amazon for a 1TB WD Blue 5400RPM Disk Drive is $40.00. A Black drive at 7200 RPM is a bit more as !TB Black drive at 5400RPM no longer seem to be widely available if at all. Considering Apple buys such drive at wholesale cost:

Apple; Shame on You!!!

MMTalker · May 5, 2024, 10:30pm

Lockheed Martin, Northrop Grumman, GE Avation, Bombardair, and maybe even SpaceX are significant competitors. Keep in mind that all of the above also earn very big bucks in delivering aerospace rockets. Here’s what Boeing is launching tomorrow:

And IIRC, these companies also make fighter jets and other military aircraft.

What I think is very interesting is that Boeing has Apps in Apple’s App Store.

xdev · May 5, 2024, 11:08pm

You’ve just proven my comment. My point is the same as yours: bugs don’t have to be there – they are there because us consumers won’t pay (in time or money) for them to be fixed. Most consumers just don’t care. They want the cheapest price, even if it isn’t good value. So we get buggy products.

xdev · May 5, 2024, 11:12pm

Not in the large passenger jet space. There is only Airbus and they are equally backordered (both companies have about 10 years of sales in the queue from what I’ve read) so there’s nowhere else for the airlines to go. It’s also more efficient for the airlines to standardize on one brand of aircraft (all Boeing or all Airbus), so it’s unlikely Boeing will lose any customers due to their quality control problems.