Apple infrastructure problems causing app launch problems

Josh and I hit this just as it started, since we both restarted our iMacs about the same time. Apps were launching extremely slowly, when they launched at all, and the entire system was sludge.

On Twitter, Jeff Johnson discovered the problem and posted about it. It’s related to an Apple server that handles revoked certificates.

This command in Terminal will edit your /etc/hosts file to block connections to the problematic Apple server:

echo 0.0.0.0 ocsp.apple.com | sudo tee -a /etc/hosts

(Some are saying that 127.0.0.1 is better, but it shouldn’t matter much since you REALLY do not want to leave that line in /etc/hosts—it MUST come out later! Does someone have a Terminal command that will do that for those less comfortable with editing Unix files?)

There’s some speculation that Apple is having major infrastructure problems, perhaps associated with the Big Sur release since developer.apple.com is down, and the Apple System Status page shows a few outages.

Obviously, the fact that an Apple infrastructure problem can cause major issues for Macs all over the place is hugely troubling.

The problem onset for me around 12:40 PM (Pacific time). It affected all three of my Macs simultaneously, and within 10 minutes, I had calls from two clients, in different parts of the US, reporting the same issues—apps excruciatingly slow to open, with some, notably MS Word, not opening at all. The problem affected only Macs, not iOS devices or PC’s.

The problem would disappear when a Mac was disconnected from Internet and recur immediately upon reconnecting.

Fortunately, after an hour, the problem seems to have resolved for everyone I’ve been in contact with.

(I wonder how many thousands of hours of productivity Mac users lost during that hour.)

Yep, I can confirm that the problem has gone away now too. I removed the line from my /etc/hosts/ and rebooted and everything works properly again.

So it was temporary, but troubling.

Just wasted half an hour or so trying to find the problem. :rage: Apple.

Here’s a shell command to remove the /etc/hosts line:

sudo sed -e '/^0\.0\.0\.0 ocsp\.apple\.com$/d' -i '' /etc/hosts

I’ve tried that on Mojave. sed is a standard Unix command from long ago, so I highly doubt it’s changed in Catalina or Big Sur, but just as an FYI I haven’t tested it on either of those (or earlier than Mojave.)

1 Like

I will keep that command on hand, just in case! :crazy_face:

Apropos, perhaps. :slight_smile:

2 Likes

Thank you for posting this information. While I saw it way after the problem was resolved it explained why I, my son, and many of my coworkers were having problems today. Different problems I saw were probably caused by it, including one person who recorded an interview with a VOIP client on her Mac and never saw the recording until after Apple fixed its problem.

As of the release of Big Sur, Apple now has complete control over what apps you are allowed to run on your Mac. Little Snitch cannot prevent it anymore.

First off, this problem affected at least Mojave and Catalina as well as Big Sur. It was technically Big Sur’s fault, given that so many people downloading Big Sur caused problems for an Apple CDN (content delivery network), which in turn caused the problems with connectivity to ocsp.apple.com.

Second, apps were not prevented from launching. They launched after the connection to ocsp.apple.com timed out. That’s why disconnecting from the Internet entirely, or blocking ocsp.apple.com via /etc/hosts worked.

So I’m curious how you think Little Snitch’s inability to block traffic from a set of Apple-specific apps in Big Sur means that Apple has complete control where it didn’t before? I don’t see the connection.

2 Likes

Also, this only happened with apps originally installed via the Apple Store, correct? I didn’t see it for myself (I either was not using my Mac at the time, and/or only using apps that don’t check in with oscp.apple.com)

Not for me, it seemed to affect every app. While trying to figure out what was going on, I rebooted, and it took a huge amount of time to launch all the stuff that launches when I log in and put icons in the menubar, and most of those aren’t app store installed apps (Quicksilver, Default Folder, Busycal, a couple of Objective-See’s apps…) Firefox took so long it finally gave up (sort of. When I tried to shutdown it told me it couldn’t because Firefox was blocking it. Sigh.)

My guess (and it’s just that), is that when launching an app, macOS was verifying the signature, which meant checking to see if the developer’s signing certificate was revoked, which meant hitting ocsp.apple.com and waiting for that request to time out. So it would have been every app signed by a developer. I didn’t think of that until after I saw Adam’s original post, otherwise I could have tested it by launching one of the unsigned apps I use, but didn’t in time.

1 Like

ah, that makes sense. Thanks for the clarification. I guess I went to lunch or something during the time Apple disabled EVERY APP INSTALLED! Yikes, they better shore up that infrastructure so this never happens again.

No. This check happens for every single app that is launched on your Mac, always. Even unsigned apps are still checked, I believe.

The point is that Apple can simply stop any app from working via this service. They recently actually did this to my printer driver. Suddenly we could no longer print. Reinstalling the driver didn’t fix it. Incredibly frustrating waste of time. My wife had just downloaded something, which made us think it was a malware problem, sending us down the wrong avenue looking for solutions.

The specific way this infrastructure problem presented did hit multiple versions of the OS, but the fact is that up until Big Sur, you could keep your computer from running this check using something like Little Snitch. Now you can no longer do so. Meaning, you must now rely on Apple’s good will and judgment to not remotely disable the program you bought and installed and use daily but that your government just decided it doesn’t like, just for example. Given Apple’s strong stance against letting governments tell them what to do, I’m sure this won’t ever be a problem and nobody should be concerned about it. Oh wait, they caved to China’s demands? Yeah, perhaps Apple having absolute say over what programs we can run on our computers is a bad idea.

Edited to add: For further reading on this: Jeffrey Paul: Your Computer Isn't Yours

1 Like

It sounds like Adam’s solution would be a workaround if Apple did block some app. If the app is running, it keeps running, yes? In the case of the printer driver, could @jtbayly have disconnected from the internet and then printed? I wasn’t affected by this problem and I hope Apple fixes it so it never happens again, but I would like to be prepared.

By the way, @ace, my SO and I thought the video you posted was hilarious as well as ironic, being from Apple.

That wasn’t Apple’s doing. HP unintentionally revoked its own certificate.

Not using Little Snitch, but there are lots of other ways to block traffic. I suspect editing /etc/hosts would still work, for instance, as would a Pi-hole or any other network management device. Nothing has really changed here.

I’ve seen that, but I’d also like to see some corroboration that Apple’s OCSP traffic is unencrypted. It should be using OSCP stapling, as I note in the article I just published. Regardless, @rmogull is going to look into it on the weekend to see if these claims are a true concern.

2 Likes

Editing etc/hosts, assuming you still can, is not the same as having a firewall you can control. An external device is now necessary to have a firewall. On your own computer. Nothing’s changed?

And how exactly do you recommend setting up a pihole at Starbucks?

Apple had to approve their request. Apple took the final action. They are both at fault, but Apple demonstrated that it holds the power to disable whatever apps it wants on my computer. It’s precisely to prevent this sort of thing that you would use Little Snitch. Editing the hosts file breaks all sorts of things. You can’t leave it that way. It’s a temporary workaround. A firewall is designed to allow you to control the traffic to and from your device so that you can protect it from whatever threats you decide are problematic. Apple has decided we are not allowed to have working firewalls on our Macs anymore at exactly the same time that they have demonstrated the actual reasons one might want a working firewall.

I look forward to reading Rich’s analysis.

1 Like

Not to my mind. You can still block traffic to a particular host if you like. The way you might have to do it is different, that’s all.

I’m not sure what the difference is between blocking all traffic to ocsp.apple.com with Little Snitch and using /etc/hosts to cause all connections to the site to fail. Both would seem to have the same effect, and if there’s a long-term problem with one, I assume the other would suffer from it as well.

I’m curious, given that Apple has been able to revoke the certificate for any app it wants for years now, how long have you been using Little Snitch to protect against abuses of that power?

I only found out Apple was doing this earlier this year.

If you’ve used Little Snitch you know how much more powerful and fine-grained it is than editing etc/hosts. Trying to say it’s the same is like saying that nothing has changed when your Tesla is remotely disabled but there are still horses somewhere you can ride.

1 Like

I have used Little Snitch in the past, but I disabled it. I like the idea, I like the Objective Development folks a great deal, and I’m glad it’s available for those who want to use it, but I don’t have time in my life to evaluate network traffic at that level. Nor do I want to get into second-guessing Apple with what might or might not be a legitimate certificate revocation.

Yesterday, when I needed to block ocsp.apple.com, editing /etc/hosts took less time than it would have taken to download Little Snitch, much less to buy, install, and configure it.

Not the same, no. Cheaper? Yes. Easier? Yes. Faster? Yes. Less flexible, though. But you’re right, not the same

Hmmm. I doubt it. Most certificate models allow the owner of a certificate to revoke it with no overt action by the infrastructure operators. The idea is that, should the owner of a certificate discover that it has been compromised, they can kill it as quickly and easily as possible. It’s unlikely that Apple does (or even can do) any review of revocation requests. No culpability here except following best practices for certificate infrastructure.

Install an OpenVPN server on the Pi-Hole. My primary iPhone always uses my Pi-Hole (except for the annoying fact that you can’t AFAIK automatically start a VPN after a reboot).

1 Like