Explanation for some Big Sur upgrade problems

MrMacintosh has found a cure for some of the mysterious failures that can occur when the progress bar gets stuck trying to upgrade to Big Sur and will sit there indefinitely. The problem is a large accumulation of hidden files apparently generated by Spotlight. The MrMacintosh site has details with ways to check for the problem before you encounter and ways to recover if you’ve already been affected.

Anything over 20,000 of those hidden files can cause problems; MrMacintosh has a way to check for them. It found 959631 on my desktop, which I have not tried to upgrade, and that is not the highest reported.

2 Likes

Just a note that this issue is more common for those updating from Mojave and Catalina than it is with Big Sur where the issue with mdworker empty folders reportedly no longer occurs. But those empty folders can carry forward during migration from an older macOS, so those that have done so should check to see if the problem remains and clear out those folders in accordance with the instructions Mr Macintosh published.

I have 2,177,525. Glad I’m replacing this computer and not trying to upgrade! :anguished:

You can just delete those files. It won’t break anything. That’s actually the recommendation for anyone about to upgrade.

3 Likes

I should have made it clear that the MacMini desktop with 959631 of the files is still on Mojave. I was thinking of upgrading to Big Sur but put that off when attempting to upgrade my 2017 Macbook Air from Mojave to Big Sur froze, apparently because it did not have enough free space.

The Mrmacintosh link details how to delete the files so you can upgrade and how to recover if the bug bit you. I followed the instructions to delete the files from the MacMini and it caused no problems, although I have not yet tried to update it.

The linked article tells the reader to run several commands preceded by sudo, but then says “It is ok if you delete everything in the /private/var/folders directory.” The given command to accomplish that task, shown below, is not preceded by sudo. Should it be?
rm -Rf /private/var/folders/*

As I understand it, the /var/folders directory (which is really a symlink to /private/var/folders) is a set of per-user storage locations for app data, caches and temporary files. The directory names appear to be some kind of hash value, but they correspond to distinct users, including a variety of private system users (all within the zz directory).

For example, on my Mac (replacing user names with placeholder names):

> ls -l /var/folders
total 0
drwxr-xr-x@  3 root  wheel    96 Oct 18  2020 c6
drwxr-xr-x@  3 root  wheel    96 Oct 18  2020 dd
drwxr-xr-x@  3 root  wheel    96 Oct 18  2020 ww
drwxr-xr-x@  3 root  wheel    96 Oct 18  2020 yx
drwxr-xr-x@  3 root  wheel    96 Oct 18  2020 zv
drwxr-xr-x@ 35 root  wheel  1120 Sep 27 18:30 zz

> ls -l /var/folders/*
/var/folders/c6:
total 0
drwxr-xr-x@ 7 user1  staff  224 Sep 27 18:30 fvrtg96j0jx10s3778p4kgjw000101

/var/folders/dd:
total 0
drwxr-xr-x@ 6 user2  staff  192 Sep 27 18:32 54bxs1g12td3y4md40lk2xjh0000gs

/var/folders/ww:
total 0
drwxr-xr-x@ 6 admin  staff  192 Sep 27 18:43 p_tsnbcs3dj_3cq6zbdrmm880000gt

/var/folders/yx:
total 0
drwxr-xr-x@ 6 macports  macports  192 Sep 27 18:39 m3d21nm91wnd4c0k2yjk57200000gn

/var/folders/zv:
total 0
drwxr-xr-x@ 6 user3  staff  192 Sep 27 18:32 92_sf7yj6sbfsqc66wthwjhh000102

/var/folders/zz:
total 0
drwxr-xr-x@ 7 root                    wheel                   224 Sep 27 18:30 zyxvpxvq6csfxvn_n0000000000000
drwxr-xr-x@ 5 _lp                     _lp                     160 Jan 17  2021 zyxvpxvq6csfxvn_n000003800000t
drwxr-xr-x@ 5 _appleevents            _appleevents            160 Sep 25  2020 zyxvpxvq6csfxvn_n000006w00001q
drwxr-xr-x@ 5 _mdnsresponder          _mdnsresponder          160 Sep 25  2020 zyxvpxvq6csfxvn_n0000084000021
drwxr-xr-x@ 5 _windowserver           _windowserver           160 Sep 25  2020 zyxvpxvq6csfxvn_n00000b000002r
drwxr-xr-x@ 6 _spotlight              _spotlight              192 Sep 27 18:31 zyxvpxvq6csfxvn_n00000b400002s
drwxr-xr-x@ 5 _securityagent          _securityagent          160 Oct 18  2020 zyxvpxvq6csfxvn_n00000bh00002w
drwxr-xr-x@ 5 _atsserver              _atsserver              160 Sep 25  2020 zyxvpxvq6csfxvn_n00000c4000031
...

Every one of those per-user directories has the same next-level directories:

> ls -l /var/folders/c6/fvrtg96j0jx10s3778p4kgjw000101/
total 0
drwxr-xr-x@  21 user1  staff    672 Sep 27 18:40 0
drwx------@ 304 user1  staff   9728 Nov  1 17:26 C
drwxr-xr-x    2 user1  staff     64 Sep 27 18:30 Cleanup At Startup
drwx------@ 448 user1  staff  14336 Nov  5 10:36 T
drwxr-xr-x    2 user1  staff     64 Sep 27 18:30 X

I don’t know everything about these directories, but here’s what I do know:

  • 0 is the “Darwin User Directory”. The system configuration variable DARWIN_USER_DIR always contains the directory for the current user:

    > getconf DARWIN_USER_DIR
    /var/folders/c6/fvrtg96j0jx10s3778p4kgjw000101/0/
    

    It appears to contain system files. There are directories named after system apps, frameworks and services. Some are not accessible. Many are binary files (including SQLite databases), while some are plist/XML data files.

    > ls -l /var/folders/c6/fvrtg96j0jx10s3778p4kgjw000101/0
    ls: com.apple.LaunchServices.dv: Operation not permitted
    ls: com.apple.SharedWebCredentials: Operation not permitted
    ls: com.apple.exchangesync: Operation not permitted
    ls: com.apple.lockoutagent: Operation not permitted
    ls: com.apple.nsurlsessiond: Operation not permitted
    ls: dmd: Operation not permitted
    total 0
    drwxr-xr-x  2 user1  staff   64 Sep 27 18:31 com.apple.AppSSOAgent
    drwxr-xr-x@ 3 user1  staff   96 Oct 19  2020 com.apple.Safari
    drwxr-xr-x@ 3 user1  staff   96 Oct 18  2020 com.apple.ScreenTimeAgent
    drwxr-x---  2 user1  staff   64 Oct 18  2020 com.apple.Spotlight
    drwxr-xr-x  2 user1  staff   64 Oct 18  2020 com.apple.bird
    drwxr-xr-x  3 user1  staff   96 Sep 27 18:41 com.apple.corespeechd
    drwxr-xr-x  2 user1  staff   64 Sep 27 18:30 com.apple.dmd
    drwx------  3 user1  staff   96 Oct 18  2020 com.apple.dock.launchpad
    drwxr-xr-x  2 user1  staff   64 Oct 18  2020 com.apple.icloud.searchpartyd
    drwx------  4 user1  staff  128 Oct 18  2020 com.apple.notificationcenter
    drwx------  3 user1  staff   96 Nov  5 09:27 com.apple.pluginkit
    drwxr-xr-x  3 user1  staff   96 Oct 18  2020 com.apple.progressd
    drwxr-xr-x@ 3 user1  staff   96 Oct 18  2020 com.apple.routined
    
  • C is the “Darwin User Cache Directory”. The system configuration variable DARWIN_USER_CACHE_DIR always contains the directory for the current user:

    > getconf DARWIN_USER_CACHE_DIR
    /var/folders/c6/fvrtg96j0jx10s3778p4kgjw000101/C/
    

    It contains cache files. In there, you will see directories named after apps and system services, each one containing cache data for the app.

    > ls -l /var/folders/c6/fvrtg96j0jx10s3778p4kgjw000101/C
    ls: com.apple.QuickLook.thumbnailcache: Operation not permitted
    ls: com.apple.WebKit.GPU.Sandbox: Operation not permitted
    ls: com.apple.WebKit.Networking.Sandbox: Operation not permitted
    ls: com.apple.WebKit.WebContent.Sandbox: Operation not permitted
    total 2368
    drwx------     2 user1  staff       64 Sep 27 18:40 ${DaemonNameOrIdentifierHere}/
    drwx------     2 user1  staff       64 Oct 18  2020 AudioComponentRegistrar/
    drwxr-xr-x     3 user1  staff       96 Dec 10  2020 com.DigiDNA.iMazing2Mac/
    drwx------@    4 user1  staff      128 Nov 23  2020 com.TryBest.Paint/
    drwxr-xr-x     3 user1  staff       96 Feb  9  2021 com.adobe.ARMDC/
    drwxr-xr-x     3 user1  staff       96 Oct 15 17:17 com.adobe.AdobeRdrCEFHelperGPU/
    drwxr-xr-x     4 user1  staff      128 Sep 27 18:43 com.adobe.Elements Auto Creations 2021/
    drwxr-xr-x     3 user1  staff       96 Oct 23  2020 com.adobe.Elements2021/
    drwxr-xr-x     3 user1  staff       96 Oct 23  2020 com.adobe.ElementsOrganizer2021/
    ...
    
  • T is the “Darwin User Temporary Directory”. The system configuration variable DARWIN_USER_TEMP_DIR and the environment variable TMPDIR always contain the directory for the current user:

    > getconf DARWIN_USER_TEMP_DIR
    /var/folders/c6/fvrtg96j0jx10s3778p4kgjw000101/T/
    
    > echo $TMPDIR
    /var/folders/c6/fvrtg96j0jx10s3778p4kgjw000101/T/
    

    It contains temporary files, again on a per-app basis. I’m not clear about how temporary files differ from cache files. I think cache files are expected to persist over time, while temporary files can be considered cruft after the app that created them quits.

  • Cleanup At Startup is typically empty. As I understand it, macOS wipes the contents after each reboot. I think it is meant for temporary files generated by system installers. On my system, the directory is empty and was last modified on the date I upgraded from Catalina to Big Sur.

  • X is empty and hasn’t been modified since the date I upgraded from Catalina to Big Sur. I have no clue what it is supposed to be used for.

In general, I think you can safely delete everything under /var/folders. You can definitely delete the contents of the C and T directories. I would recommend rebooting afterward, in case a running app is using one of the files, but every app should be able to recreate its caches and temporary files the next time it runs.

See also:

4 Likes

Thanks, David. My general housekeeping instincts would nudge me in that direction anyway, but I’m interested whether you are saying that the presence of those files might hinder the data transfer process to the replacement computer?

I would like to think that Migration Assistant will make no attempt to transfer these files, since they’re all supposed to be cache/temporary files generated by the locally-running applications.

1 Like

You can, but it will slow your computer for awhile as some will have to be rebuilt. The only files that we found causing issues were the empty folders listed from terminated mdworker processes.

I wrote the “what is /var/folders” years ago specifically because deleting everything in it sometimes caused machines to not boot correctly. Chances are you’ll be ok but it can be risky. There’s also files there now that are protected by sips or some other mechanism. So it might be a bit difficult to delete everything.

FWIW, I followed the instructions to delete the contents of /var/folders/ – 959631 files – and have seen no obvious issues on my 2018 MacMini still running Mojave before or after rebooting the machine. I’m still wary of upgrading to Big Sur before running another test on my laptop.

It’s rather disturbing how long these issues have lingered without obvious action by Apple.

Years ago I did some pruning and cleaning of these directories and files – and had problems on reboot. Because I had done a clone/backup immediately prior, I was able to restore.

I no longer recall which version of macOS I was running at the time.

Mr. Macintosh revised his instructions today to remove the recommendation to remove everything in /var/folders for reasons similar to what you found. He also modified the instructions for fixing the problem adding preferred commands to speed up the cleanup process by removing the entire com.apple.mdworker.bundle and com.apple.metadata.mdworker folders rather than doing a find for each individual empty folder which has been observed to take 20 minutes in extreme cases.

2 Likes

Just wanted to thank y’all for bring the private/var/folders/ issue to my attention inasmuch as I was just planning to upgrade from Mojave to Big Sur. I followed the guidelines ala Mr. Macintosh and found I had more than 400K of those buggers lurking there. I removed them using his method and the upgrade went without a hitch

After removing them and still in Mojave, I test drove things for awhile and didn’t notice any problems

And, while I was poking around Mr. Macintosh I came across his “macOS Big Sur Full Installer Database. Download Directly from Apple! – Mr. Macintosh” page at https://mrmacintosh.com/macos-big-sur-full-installer-database-download-directly-from-apple/ which I used to get the Big Sur Installer
… I had previously been working with Apple Support to get Big Sur, in vain; even got bumped up to a ‘senior’ support tech who sent the issue to ‘engineering’ - after about 10 days - nada. The only thing they offered was a link to Big Sur in the App Store, which on “Get”, loaded System Preferences > Software Update, but, alas it always reported it couldn’t find it. Whereupon I found this thread and Mr. Macintosh…

The upgrade seems to be working just fine, thank you very much

Thanks again

I successfully upgraded my 2017 MacBook Air to Big Sur after opening up enough memory space (36 gig) for the installer to work. My only complaint so far is that the font size in the Finder seems so small that I am having trouble reading it.

You can adjust the Finder text size. Open a folder and select View → Show View Options (or just type CMD-J). Select the size you want. Click “Use as Defaults” if you want the setting to apply to all folders (that haven’t been explicitly customized).

1 Like

I spoke too soon – the upgrade appears to have scrambled the mail that was in Apple Mail. Nothing important was lost because I don’t keep important email files on that machine – but it would be a huge problem if I try to upgrade my desktop because I have important mail going back well over a decade. I also am having problems trying to log access the Air from my MacMini desktop running Mojave. The only good news is that the tiny type has vanished.

How was the mail scrambled? Did you lose messages or parts of their content? Were mailboxes lost or changed? Or something else? Also, POP or IMAP (and if IMAP, was content on the servers changed)?

1 Like

It took a while to find out where the email went. The potential show-stopper is that all my POP mail, which comes from my personal web site, has disappeared, and so has the email for that account. That’s where I get most of my mail. I prefer POP mail because I manage my mail from my desktop and only use the laptop when I’m out, which isn’t often these days. My web account also has only 250 megabyte storage, which would make IMAP impractical.

I do have IMAP Google and iCloud accounts, and the upgrade hid those by shifting them to OnMyMac accounts at the bottom of the page. From a quick look, it seems that Big Sur did rearrange email in those accounts, keeping mail from this year in the accounts at the top of the page, and copying those messages to other folders on OnMyMac which contain older messages.

My main reasons for upgrading the laptop to Big Sur were to check for potential problems that might affect me if I updated the desktop from Mojave, and to set up a computer that could run TurboTax. If I can’t find a workaround, I will stick with Mojave for the near term.