MrMacintosh has found a cure for some of the mysterious failures that can occur when the progress bar gets stuck trying to upgrade to Big Sur and will sit there indefinitely. The problem is a large accumulation of hidden files apparently generated by Spotlight. The MrMacintosh site has details with ways to check for the problem before you encounter and ways to recover if you’ve already been affected.
Anything over 20,000 of those hidden files can cause problems; MrMacintosh has a way to check for them. It found 959631 on my desktop, which I have not tried to upgrade, and that is not the highest reported.
Just a note that this issue is more common for those updating from Mojave and Catalina than it is with Big Sur where the issue with mdworker empty folders reportedly no longer occurs. But those empty folders can carry forward during migration from an older macOS, so those that have done so should check to see if the problem remains and clear out those folders in accordance with the instructions Mr Macintosh published.
I should have made it clear that the MacMini desktop with 959631 of the files is still on Mojave. I was thinking of upgrading to Big Sur but put that off when attempting to upgrade my 2017 Macbook Air from Mojave to Big Sur froze, apparently because it did not have enough free space.
The Mrmacintosh link details how to delete the files so you can upgrade and how to recover if the bug bit you. I followed the instructions to delete the files from the MacMini and it caused no problems, although I have not yet tried to update it.
The linked article tells the reader to run several commands preceded by sudo, but then says “It is ok if you delete everything in the /private/var/folders directory.” The given command to accomplish that task, shown below, is not preceded by sudo. Should it be? rm -Rf /private/var/folders/*
As I understand it, the /var/folders directory (which is really a symlink to /private/var/folders) is a set of per-user storage locations for app data, caches and temporary files. The directory names appear to be some kind of hash value, but they correspond to distinct users, including a variety of private system users (all within the zz directory).
For example, on my Mac (replacing user names with placeholder names):
It appears to contain system files. There are directories named after system apps, frameworks and services. Some are not accessible. Many are binary files (including SQLite databases), while some are plist/XML data files.
> ls -l /var/folders/c6/fvrtg96j0jx10s3778p4kgjw000101/0
ls: com.apple.LaunchServices.dv: Operation not permitted
ls: com.apple.SharedWebCredentials: Operation not permitted
ls: com.apple.exchangesync: Operation not permitted
ls: com.apple.lockoutagent: Operation not permitted
ls: com.apple.nsurlsessiond: Operation not permitted
ls: dmd: Operation not permitted
total 0
drwxr-xr-x 2 user1 staff 64 Sep 27 18:31 com.apple.AppSSOAgent
drwxr-xr-x@ 3 user1 staff 96 Oct 19 2020 com.apple.Safari
drwxr-xr-x@ 3 user1 staff 96 Oct 18 2020 com.apple.ScreenTimeAgent
drwxr-x--- 2 user1 staff 64 Oct 18 2020 com.apple.Spotlight
drwxr-xr-x 2 user1 staff 64 Oct 18 2020 com.apple.bird
drwxr-xr-x 3 user1 staff 96 Sep 27 18:41 com.apple.corespeechd
drwxr-xr-x 2 user1 staff 64 Sep 27 18:30 com.apple.dmd
drwx------ 3 user1 staff 96 Oct 18 2020 com.apple.dock.launchpad
drwxr-xr-x 2 user1 staff 64 Oct 18 2020 com.apple.icloud.searchpartyd
drwx------ 4 user1 staff 128 Oct 18 2020 com.apple.notificationcenter
drwx------ 3 user1 staff 96 Nov 5 09:27 com.apple.pluginkit
drwxr-xr-x 3 user1 staff 96 Oct 18 2020 com.apple.progressd
drwxr-xr-x@ 3 user1 staff 96 Oct 18 2020 com.apple.routined
C is the “Darwin User Cache Directory”. The system configuration variable DARWIN_USER_CACHE_DIR always contains the directory for the current user:
It contains cache files. In there, you will see directories named after apps and system services, each one containing cache data for the app.
> ls -l /var/folders/c6/fvrtg96j0jx10s3778p4kgjw000101/C
ls: com.apple.QuickLook.thumbnailcache: Operation not permitted
ls: com.apple.WebKit.GPU.Sandbox: Operation not permitted
ls: com.apple.WebKit.Networking.Sandbox: Operation not permitted
ls: com.apple.WebKit.WebContent.Sandbox: Operation not permitted
total 2368
drwx------ 2 user1 staff 64 Sep 27 18:40 ${DaemonNameOrIdentifierHere}/
drwx------ 2 user1 staff 64 Oct 18 2020 AudioComponentRegistrar/
drwxr-xr-x 3 user1 staff 96 Dec 10 2020 com.DigiDNA.iMazing2Mac/
drwx------@ 4 user1 staff 128 Nov 23 2020 com.TryBest.Paint/
drwxr-xr-x 3 user1 staff 96 Feb 9 2021 com.adobe.ARMDC/
drwxr-xr-x 3 user1 staff 96 Oct 15 17:17 com.adobe.AdobeRdrCEFHelperGPU/
drwxr-xr-x 4 user1 staff 128 Sep 27 18:43 com.adobe.Elements Auto Creations 2021/
drwxr-xr-x 3 user1 staff 96 Oct 23 2020 com.adobe.Elements2021/
drwxr-xr-x 3 user1 staff 96 Oct 23 2020 com.adobe.ElementsOrganizer2021/
...
T is the “Darwin User Temporary Directory”. The system configuration variable DARWIN_USER_TEMP_DIR and the environment variable TMPDIR always contain the directory for the current user:
It contains temporary files, again on a per-app basis. I’m not clear about how temporary files differ from cache files. I think cache files are expected to persist over time, while temporary files can be considered cruft after the app that created them quits.
Cleanup At Startup is typically empty. As I understand it, macOS wipes the contents after each reboot. I think it is meant for temporary files generated by system installers. On my system, the directory is empty and was last modified on the date I upgraded from Catalina to Big Sur.
X is empty and hasn’t been modified since the date I upgraded from Catalina to Big Sur. I have no clue what it is supposed to be used for.
In general, I think you can safely delete everything under /var/folders. You can definitely delete the contents of the C and T directories. I would recommend rebooting afterward, in case a running app is using one of the files, but every app should be able to recreate its caches and temporary files the next time it runs.
Thanks, David. My general housekeeping instincts would nudge me in that direction anyway, but I’m interested whether you are saying that the presence of those files might hinder the data transfer process to the replacement computer?
I would like to think that Migration Assistant will make no attempt to transfer these files, since they’re all supposed to be cache/temporary files generated by the locally-running applications.
You can, but it will slow your computer for awhile as some will have to be rebuilt. The only files that we found causing issues were the empty folders listed from terminated mdworker processes.
I wrote the “what is /var/folders” years ago specifically because deleting everything in it sometimes caused machines to not boot correctly. Chances are you’ll be ok but it can be risky. There’s also files there now that are protected by sips or some other mechanism. So it might be a bit difficult to delete everything.
FWIW, I followed the instructions to delete the contents of /var/folders/ – 959631 files – and have seen no obvious issues on my 2018 MacMini still running Mojave before or after rebooting the machine. I’m still wary of upgrading to Big Sur before running another test on my laptop.
It’s rather disturbing how long these issues have lingered without obvious action by Apple.
Years ago I did some pruning and cleaning of these directories and files – and had problems on reboot. Because I had done a clone/backup immediately prior, I was able to restore.
I no longer recall which version of macOS I was running at the time.
Mr. Macintosh revised his instructions today to remove the recommendation to remove everything in /var/folders for reasons similar to what you found. He also modified the instructions for fixing the problem adding preferred commands to speed up the cleanup process by removing the entire com.apple.mdworker.bundle and com.apple.metadata.mdworker folders rather than doing a find for each individual empty folder which has been observed to take 20 minutes in extreme cases.
Just wanted to thank y’all for bring the private/var/folders/ issue to my attention inasmuch as I was just planning to upgrade from Mojave to Big Sur. I followed the guidelines ala Mr. Macintosh and found I had more than 400K of those buggers lurking there. I removed them using his method and the upgrade went without a hitch
After removing them and still in Mojave, I test drove things for awhile and didn’t notice any problems
And, while I was poking around Mr. Macintosh I came across his “macOS Big Sur Full Installer Database. Download Directly from Apple! – Mr. Macintosh” page at https://mrmacintosh.com/macos-big-sur-full-installer-database-download-directly-from-apple/ which I used to get the Big Sur Installer
… I had previously been working with Apple Support to get Big Sur, in vain; even got bumped up to a ‘senior’ support tech who sent the issue to ‘engineering’ - after about 10 days - nada. The only thing they offered was a link to Big Sur in the App Store, which on “Get”, loaded System Preferences > Software Update, but, alas it always reported it couldn’t find it. Whereupon I found this thread and Mr. Macintosh…
The upgrade seems to be working just fine, thank you very much
I successfully upgraded my 2017 MacBook Air to Big Sur after opening up enough memory space (36 gig) for the installer to work. My only complaint so far is that the font size in the Finder seems so small that I am having trouble reading it.
You can adjust the Finder text size. Open a folder and select View → Show View Options (or just type CMD-J). Select the size you want. Click “Use as Defaults” if you want the setting to apply to all folders (that haven’t been explicitly customized).
I spoke too soon – the upgrade appears to have scrambled the mail that was in Apple Mail. Nothing important was lost because I don’t keep important email files on that machine – but it would be a huge problem if I try to upgrade my desktop because I have important mail going back well over a decade. I also am having problems trying to log access the Air from my MacMini desktop running Mojave. The only good news is that the tiny type has vanished.
How was the mail scrambled? Did you lose messages or parts of their content? Were mailboxes lost or changed? Or something else? Also, POP or IMAP (and if IMAP, was content on the servers changed)?
It took a while to find out where the email went. The potential show-stopper is that all my POP mail, which comes from my personal web site, has disappeared, and so has the email for that account. That’s where I get most of my mail. I prefer POP mail because I manage my mail from my desktop and only use the laptop when I’m out, which isn’t often these days. My web account also has only 250 megabyte storage, which would make IMAP impractical.
I do have IMAP Google and iCloud accounts, and the upgrade hid those by shifting them to OnMyMac accounts at the bottom of the page. From a quick look, it seems that Big Sur did rearrange email in those accounts, keeping mail from this year in the accounts at the top of the page, and copying those messages to other folders on OnMyMac which contain older messages.
My main reasons for upgrading the laptop to Big Sur were to check for potential problems that might affect me if I updated the desktop from Mojave, and to set up a computer that could run TurboTax. If I can’t find a workaround, I will stick with Mojave for the near term.