Koingo temporarily shutting down due to data loss

That’s all well and good…but as a developer at Koingo should have had the beta software on a separate machine or boot drive that allowed him/her (seems like a one person shop to me) because as a developer he should have known of the inherent risks of running a beta version of macOS. Using it on their production machine used to run the business is irresponsible and they’ve got nobody to blame except themselves for making a dumb decision.

5 Likes

I too received the letter. I’m surprised a CEO of a software development company, of all people, did not have a backup. iCloud is not a backup service (as far as I know.) It synchronizes your files across all devices, which means if a file is removed from iCloud on your machine, it also gets removed from the cloud storage. He seems surprised by it.

4 Likes

You are correct. It is not. It is a cloud storage service. But there’s a fine (but important) distinction between what cloud providers give you for "resiliency"for your data and protecting that data against annihilation.

Having worked in the data protection space, you would be surprised at the number of corporate employees that think that when you sync data to OneDrive/Box/Google Drive that the cloud provider is responsible for backing it up. Yes, they may have the availability of deleted items through a trash can-like construct. But that’s not a backup.

2 Likes

I found a copy of the letter online which answered some of the questions I had. To fill in some details for those who didn’t see the letter:

  • Josh, the CEO of Koingo, had a backup of local files. It’s not like he didn’t have any backups.
  • He mistakenly assumed that data in iCloud was safe and therefore didn’t need to be backed up.
  • A bug in Photos caused it to hang, perhaps corrupting the library.
  • When he relaunched Photos, it started over from scratch, wiping out the existing library in iCloud, which then propagated to all of his other devices.
  • He blames Apple for releasing software that can cause massive data loss.
  • He blames Apple for having no way to recover a lost photo library in Cloud.

It sounds like a backup of his local Photos library woulld have had only lower-resolution versions of many photos. Even so, a backup of the local library would have been better than nothing. I don’t know what he uses for backups, but it does sound like he explicitly went out the way not to back up the local library.

If it is as you say it reflects very badly on Josh hence Konigo. If I was a CEO and had made such crass mistakes running beta software I would have kept very quiet about it.

Anyone serious about backups knows that iCloud is not a backup. Apple Photos backup strategies usually advise having at least one local copy of the Photos library which has full size originals (not “optimised”) and hence can have a local backup…quite irrespective of the beta aspect.

5 Likes

Now that I’ve heard a few voices here, and for the most part we seem to be on the same page, let me share the deeper issue here that’s concerning me.

By all reasonable standards, this is a person we would expect to have understanding of how these things work. You don’t become the CEO of a small software company that’s been in business for nearly 30 years by being ignorant about computers. (Okay, maybe some people do, but they generally don’t stay in business for long after that.)

If someone like Josh can get something so straightforward and fundamental so very wrong, is it any wonder “ordinary” users make the same mistake, over and over again?

Is there something inherently wrong with how we (collective industry- and community-wide “we”) try to educate people about data safety, data security, and similar issues? Is there something wrong with our expectations of responsibility for these situations, on the part of both users and developers?

Should we be expecting and demanding better from the industry than what we’re getting?

I don’t have answers to any of these questions. But I think it’s long overdue for the community to re-evaluate our expectations, what we’re willing to put up with from computer companies, and what responsibility and/or liability the companies should be expected to take on when things go wrong because users don’t understand.

6 Likes

I think some of the blame lies with sites like 9to5mac, which endlessly promotes each beta release, and usually provides instructions about how to obtain it. Beta software is not for the general user, but some sites sure make it seem like it is. (No, the CEO of a software company is not a general user, but he may have gotten lured into acting like one by the tone of these stories.)

2 Likes

I know nobody reads these, but iCloud’s terms of service explicitly say:

You are responsible for backing up, to your own computer or other device, any important documents, images or other Content that you store or access via the Service. Apple shall use reasonable skill and due care in providing the Service, but Apple does not guarantee or warrant that any Content you may store or access through the Service will not be subject to inadvertent damage, corruption or loss.

3 Likes

There is also this.

Please note that since the public beta software has not yet been commercially released by Apple, it may contain errors or inaccuracies and may not function as well as commercially released software. Be sure to back up your iPhone or iPad and your Mac using Time Machine before installing beta software. Since Apple TV and HomePod mini data and purchases are stored in the cloud, there’s no need to back up your Apple TV or HomePod mini. Install the beta software only on non-production devices that are not business critical. We strongly recommend installing on a secondary system or device, or on a secondary partition on your Mac.

5 Likes

I suspect Josh knew perfectly well about the poor choices he made, but he grew complacent. If I’m right, it’s a good example of the fact that human nature is probably the biggest obstacle. In the past, I recall reading about a poll that revealed that the majority of users knew they should be backing up their data, but hardly anyone did. People know they shouldn’t be using their dog’s name for a password, but they still do anyway. Many here frequently express a dislike of tracking, yet they still use social media. Every time, you hear rationalizations. “I have to use FaceBook because…” or “I can’t remember more complicated passwords. Who’s going to know anyway?” Ordinary users make the same mistakes repeatedly because they don’t care enough not to make them until something really bad happens.

I guess that’s kind of my point. Telling people what they “should” do doesn’t help them do it, thanks to human psychology. But advising “best practices” is the default whenever this kind of thing happens to someone. Which leads to my question: Are we doing this wrong?

It’s sort of like the re-evaluation of a basic assumption of economics that’s been taking place. For a long time, economists have operated under the assumption that consumers are rational. But it turns out that they’re anything but (which marketers figured out long before economists did). Newer economic theories take the irrationality of human behavior into account.

Advising “best practices” on data security is a similar assumption of rationality. The reality is that the biggest danger to your data is yourself—or, more specifically, the human penchant for doing the wrong thing regardless of whether one “knows better”. The best backup strategies are the ones that happen without user intervention, because they don’t depend on the user repeatedly doing anything. Time Machine’s success is a testament to the value of defaulting to “don’t make the user remember to do something the right way”.

The more I think about this, the closer I’m coming to concluding that leaving things like data security up to the users’ rationality will never be successful. I know that there are people in the industry who feel the same way, but they’re not being heard by the people who make the money decisions.

Which brings me back to, should we, as a community, be demanding better than this? Or, perhaps more to the point, why aren’t we demanding better than this? Why are we accepting this as status quo?

4 Likes

Don’t get me started. I’ll be in full “old man yelling at clouds” mode in no time.

I think a lot of the problem comes from software development practices now commonly in place that favor rapid, frequent releases of new “features” (often very poorly documented, even within the development team) over almost everything else. The truth is that most consumer software these days is the equivalent of beta software by design.

Remember when Gmail was officially “beta” for years? In practice, it still is, but we accept it as the “modern” way of doing things. I don’t think it will get better unless/until ad-driven/subscription-driven business models go away. Yeah, I’m not optimistic about that.

1 Like

Since there is no free lunch, should those of us who religiously employ backup strategies share in the increased cost of Apple et al. or some universal entity backing up for those who don’t do it on their own? If the industry takes over this enormous task for everyone, it would inevitably be reflected in the increased cost of hardware and software for everyone, would it not?

We all share in the costs when our friends and coworkers, and businesses we patronize, fail to properly secure their data, in terms of inconvenience, emotional impact, frustration, and sometimes actual time and money lost. The costs are not paid only by the person who failed to do the backup.

Case in point: The Koingo incident that started this thread. Josh made a mistake in his backup strategy, and because of this, he’s temporarily shut down the entire company. His customers are paying for his mistake by not getting the development effort that would otherwise be being put into his products right now. His employees, if he has any, have to redirect their work to a task that may be futile. And his family has lost the actual pictures.

No one is an island, and “cost” does not equal “money”. The hyper-capitalist focus on only monetary costs is part of the problem here.

3 Likes

Not that I think Apple is at all to blame for Koingo’s misfortune, but there is one thing that Apple could do better with regard to iCloud. They could better educate users about iCloud: what it’s for, what it isn’t for, how best to integrate it into one’s digital world.

As for backup, why is Time Machine still not integrated with iCloud?

Apple allows you to restore up to 30 days of data from iCloud Drive, contacts, calendars (and maybe reminders and notes?) on the iCloud web site. It’s a shame that they don’t have the same for photos (or even a week of data). It’s probably a hard problem, but because of optimized storage and large libraries, it would be a useful feature that would have fixed this issue with minimal loss.

The ‘Deleted Items’ album essentially provides this. I can only imagine that in the Koingo issue, the library was corrupted such that Photos has no memory of the data ever existing. In that case, there’s nothing to restore (and would be the same if you had a corruption of contacts for example).

Go onto iCloud.com and look at data recovery. They have specific days that you can recover contacts. I have two entries for contacts; I have 10 daily entries for calendars that seem to happen daily at 7:59 pm. (And now looking I see that it’s bookmarks rather than reminders that can also be restored.)

I argue that they could do the same with iCloud Photo Library. Not individual photos and videos in deleted items - the full library. Even three back entries might be enough to recover from a disaster like this, especially now that optimized storage is a thing.

1 Like

As for backup, why is Time Machine still not integrated with iCloud?

Generally speaking, iCloud has specific folders for specific steps still…which is why Apple doesn’t call it a backup I believe.

Because no one can prevent people from being stupid idiots. Time Machine works very well on autopilot, but only if you turn it on. Only if the Time Machine backup drive stays connected.