Recovery disasters

Or, ‘when one backup isn’t enough’.

This week, I tried to update my MacBook Pro’s OS from Yosemite to High Sierra. This update failed: after the update, the Mac would hang during startup. No problem, I thought. I’m well prepared for this:

  • the MacBook contains a primary SSD and a second bootable harddrive (also Yosemite)
  • I maintain 2 sets of Time Machine backups, one on a network volume (hosted by another Mac), one on a harddisk in a USB enclosure. I even test these on occasion by restoring individual files.

Here’s my experiences over the next 2.5 days, as a ‘should be straightforward’ recovery procedure turned up a bunch of pitfalls.

The first step was to boot into the secondary HD. This worked, but the High Sierra upgrade had converted the SSD to the new APFS file system, and Yosemite can’t read that. This left my SSD entirely inaccessible.

How to handle this? Maybe download the High Sierra installer again? I tried that, but App Store downloads require an Apple ID. Oops, that password lives in my 1Password vault which is stored on the SSD, and I didn’t have a paper backup of that vault. I did have Time Machine, so maybe I can recover the 1Password vault and open it.

So, next step: access the Time Machine backup on my network drive. Oops, that backup is encrypted. Again, this password lives in 1Password and in the Keychain, no paper backups. TM backup inaccessible. This was getting scary.

The following day, I remembered the second TM backup, which might still be unencrypted. That turned out to be correct, so I could start recovering. Booted into the Recovery partition, restore from backup. This meant erasing the drive, reformatting with HFS+. I rushed this step, not considering the consequences. You see: my backups weren’t complete. I had excluded some folders from the backup in an effort to reduce the size of the backup. Most of these were expendable (various cache folders), but I had also excluded ~/Pictures, which contains my Photos database.

Anyway, restore from second TM backup, which was 1 month old. This gave me access to Keychain and 1Password, so I could access the network TM backup. Wrote down the passwords for the next time.

Another restore attempt using the network TM backup (which was up-to-date). Oops, this does not show up in Recovery mode.

Booted into the secondary partition, accessed the TM backup by opening its disk image, and I copied everything to the SSD, overwriting the 1 month-old files with newer ones.

After several hours of copying, a restart showed that the OS on the SSD would hang during startup. So that doesn’t work, and it’s unworkable to go through 500,000 files and decide which ones can be safely overwritten and which ones can’t.

Luckily I have lots of external HDs lying around, so I copied the TM bundle to one of them, and was able to access that from Recovery. One more recovery step (4 hours of copying), and I had a usable system again. It gave me one more heart-stopping moment when Mail opened to a “I need to import your old mail” screen instead of opening my mailbox, but after that import everything was there.

In the end, it looks like I’ve lost ~4 years worth of photos, and a few files I’ve recently downloaded. Everything else is intact.
The cost to get to this point: 2.5 days, and stress levels that rose to the downright unhealthy.

Lessons learned:

  1. Passwords need a backup that is accessible when the system is down.
  2. Encrypted backups are both safe and dangerous.
  3. Always make a full backup before installing a new OS.
  4. The gain from reducing my Time Machine backup size was more than wiped out by the loss of data induced by it.
  5. Having a second, entirely independent computer on hand was a lifesaver, as I could look things up while the MBP was inoperable.
  6. Similarly, my habit of not throwing away external harddisks meant I had several of them to play with, copy files to, install recovery tools if necessary, etc.

In hindsight, I should have done these things differently:

  1. Instead of erasing the SSD, I should have installed High Sierra on an external HD, and booted from there to get access to the SSD.
  2. I didn’t take enough time to consider the consequences of erasing the SSD. One click and it’s all gone.

Harro de Jong

2 Likes

Thank you for your courage and humility to public share this story, Harro. This is a good reminder that even experienced Mac users can overlook one or two minor, yet crucial, details that can turn a minor computer hiccup into a very stressful event.

Your “lessons learned” section is spot-on, particularly the item about making a full backup before proceeding with an OS upgrade. Two things I’d like to add to that list, are:

  1. Create that full backup in the form of a bootable clone, and boot from that clone before proceeding with the update, so you know that it actually does work as intended. That gives you a truly up-to-date and complete snapshot of what’s on your computer. And it being bootable should reduce the reliance on a second computer to near-zero.

  2. Purchase your password manager’s companion mobile app, install it on your phone, and set up synching. Even if your Mac should become completely unusable, you’ll at least have access to all of your login credentials, as well as (hopefully!) your key personal data, such as bank accounts, etc. If you’re one of those people who drag their mobile phone everywhere they go, having that data along with you can also be a low-effort, yet integral part of disaster preparedness.

3 Likes

As well as Time Machine I use Carbon Copy Cloner once a macOS update appears to be working correctly. In theory I can then boot from the clone, restore more recent files from Time Machine and be on my way. Your experience suggests theory and practice don’t always align so thank you for the alert.

Incidentally, it was similar recovery issues that led me to abandon Windows and switch to Mac in 2003!
http://users.tpg.com.au/users/aoaug/ms_dig.html#nightmare

2 Likes

Oh, ouch. I’m so sorry. I was on the edge of my seat reading; and my heart ached when you confirmed the loss of photos, despite an otherwise seemingly thorough backup strategy.

[edit: sorry, I didn’t grok where it was that you were missing your iCloud password and couldn’t reinstall for lack thereof. Yikes. That’s one of four passwords that is always in my head, regardless of encrypted vaults. Bank, macOS login, iOS login, and AppleID. ]

I’m still left wondering why, if you did try, that you couldn’t try again to install (reinstall) High Sierra after the first boot into the new system failed. Did I misread that after the APFS conversion appeared to bork that the SSD was not even seen in Recovery? Or was it that you just immediately went to Yosemite on HDD?

Again, my sincere condolences, and thanks for sharing your pain to highlight to others the pitfalls and possible of even apparent due diligence.

And I agree with others, nothing short of a full bootable clone makes me feel safe prior to a major upgrade. But the encrypted password vault is one I’d not have thought of had you described your strategy in advance; I wouldn’t have realized from your description that you were facing a risk factor that could’ve just as easily come from a dead SSD. And since my Keychain and 1Password vaults also sync to iOS and other macOS, my own ingrained comfort would’ve made me overlook this critical assumption.

2 Likes

Sorry, just reread it again (three times), and what I derived was that your photos weren’t being backed up anywhere, at all? Since you eventually gained access to both Time Machine volumes, both external (unencrypted) and network (“full”, encrypted), isn’t the real lesson here to make at least one full backup of your data, regardless of upgrade issues versus dead or stolen Mac? It’s not that two backups aren’t enough; it’s that at least one of them has to be complete, is it not?

Again, sorry if I’m misreading (I’m not altogether sharp in my current situation). And, again, I’m beyond sorry for your loss of photos, whatever the reason.

1 Like

Purchase your password manager’s companion mobile app, install it on your phone, and set up synching.

My phone is actually what got things started: I wanted to update my iPhone to a newer iOS version. This failed: after the update the iPhone would hang during bootup. Apple’s support article told me to update my Mac. So in this case, the companion app wouldn’t have helped. I’m still going to convert my 1Password from standalone to the subscription/cloud version though.

I didn’t grok where it was that you were missing your iCloud password and couldn’t reinstall for lack thereof.

Well, my initial attempt failed. Then I created a new Apple ID so I could download the installer again, but when I ran the installer from the HD it didn’t see the SSD.

Did I misread that after the APFS conversion appeared to bork that the SSD was not even seen in Recovery?

I don’t remember if I tried that. I was getting frantic, and trying lots of things quickly (also a mistake, by the way).
When you select ‘Reinstall Mac OS’ from Recovery, I think it prompts you for your Apple ID because it has to download the installer.
In Recovery, I could restore a Time Machine backup to the SSD, but that process starts with reformatting the SSD.

your photos weren’t being backed up anywhere, at all?

Yes. I have some older backups (the most recent one is an iPhoto database from 2015). Around 2015 I started using Time Machine, and I noticed I was backing up 200 Mb/hour minimum, which I thought would fill up my backup disk quickly, which is why I excluded some folders. In retrospect, TM deletes most of the redundant files later on (it doesn’t keep every hourly backup).

That’s one of four passwords that is always in my head,

When I started using 1Password, I also switched to using its password generator for almost everything including my Apple ID and the Time Machine backup encryption, so all my passwords are autogenerated gibberish that’s impossible to remember. Now, I’ve written those down and they’re going into my physical vault.

Harro de Jong

1 Like

I was getting frantic, and trying lots of things quickly (also a mistake, by the way).

This is a good thing to remember in any “lost data” scenario: don’t panic.

Remain calm. Many times what you’re seeing is a fluke or temporary mistake. Sometimes the data is still around. Rushing to restore or fixes can actually make things worse (as you discovered).

At any rate, you’re not thinking clearly during a panic, so it’s the wrong thing to do. Walk away, take a breather, and try and think through all the scenarios.

I’ve done this a few times and often being calm will remind me of things I wouldn’t have remembered in a panic: that I had some data on Dropbox or last year’s version of the file on an old hard drive, or something else that helped.

Sorry this happened, but it’s a good reminder!

2 Likes

So sorry to hear this story, Harro, and thanks for sharing it and your hard-won lessons. A few points of agreement and perhaps expansion (and absolutely none of these are intended as criticism!):

  • One aspect of backups is having another Mac available. Having all your data won’t do you much good if there’s no Mac to run it on. And having another Mac available makes it easier to troubleshoot things, look up passwords, verify backup drives, and so on. The big honking problem here is that you have to be able to afford another Mac. I generally do this via the classic desktop/laptop approach—I wouldn’t want to use my MacBook Air for everything, but I can do a lot on it if my iMac were to fail.

  • It is NEVER worth excluding things from backups for space reasons unless you are 100% certain that they’re backed up in other ways or that you’re 100% OK with losing all that data. One of the reasons I recommend iCloud Photos is that while not a backup per se, it provides an independent copy of all your photos. I have even had to restore my photos by erasing a library and letting them all download again because the library had developed corruption that prevented it from being backed up.

  • The specific corollary to the above general statement is that photos are precious—back them up in multiple independent ways.

  • I’m not sure if it played a role in your experience, but I generally recommend staying more up to date in part because upgrading from a 5-year-old version of macOS to a 2-year-old version will come with many more gotchas than moving from the n-1 version to the n-current version. That’s the trouble with waiting—nothing forces you to upgrade, but the upgrades can be much more painful when you do decide to make the jump.

  • As @xdev said, remaining calm is so, so important. I’ve been where you were many a time, and my reaction to making a mistake it try to rectify it as quickly as possible, and I’ve often moved too quickly and made things worse. Now the only time I move quickly after realizing I’ve made a mistake is when I think it may be possible to prevent the spread of the problem by acting fast. Most of the time, however, it’s worth taking some time to think through what has happened, come up with an approach to address it, and if possible, run it by an expert friend to make sure you’re not missing anything.

Hopefully others can learn from your experience and the suggestions in this thread.

1 Like

My thought was if you could see your SSD using Disk Utility in Recovery Mode, to attempt First Aid, then another attempt to upgrade, before giving up and wiping it. But, frantic frustration, and assumption of a complete Time Machine might’ve led me down the same path. Ugh. Sorry.

F

I have used Stellar Software Photo Recovery software a few times to recover files accidentally deleted from SD cards (relatives who have just returned from a big holiday and “deleted all” instead of one photo!).

Stellar also have Mac data recovery software that (fortunately) I haven’t tried:

https://www.stellarinfo.com/data-recovery-mac.php

Not free ($79) but it may be worth considering for some situations. The trick is to not panic and try to leave the SD card/Mac unchanged until you are able to run the recovery software because your precious deleted/corrupt data could be overwritten.