Class 1 MacBug: Ventura reinstall may delete all drive data without warning

Background: Three libraries types exist in MacOS Ventura: System Library; User Libraries: and the Root or just the Library. This pertains to the Root or “Library”. This is the Library that lives in the root directory of a drive, the directory that appears when you first open up a drive in Finder. This library affects all the users and can get corrupted. Sometime System Recovery will fix it but sometimes not. Unfortunately Apple provides no tools and little to no support to isolate and fix corrupted files within it other than perhaps the shotgun method of making a backup; erase and reinstalling everything on the drive. Additionally this library folder can not be moved or deleted. However the contents can be deleted leaving an empty Library folder. However if this is done the drive may refuse to boot or boot incorrectly. So far the only way to fix a corrupted Library is to delete the contents from another drive; boot into recovery; and reinstall MacOS.

Class 1 Bug: However if the Library contents is deleted; the computer booted into recovery and MacOS is reinstalled, what happens during the reinstall without warning or information is that all the existing user folders and the application folder are deleted. So the result is the same as if the drive was erased and installed with a fresh copy of MacOS. This issue has been reported to Apple Support. This is different than a normal MacOS reinstall, as with it, the user and application folders remain untouched. What makes this a serious bug is that the user is not warned that without the Library contents in place that the entire drive will be essentially erased and all your data and applications will be wiped clean from the drive.

Conclusions: For most users, if leaving the Library untouched and simply Reinstalling MacOS from recovery fails to fix a corrupted Library, it is probably best to do a backup using time machine, erase, and reinstall of MacOS; then use system migration to restore the drive. For the advanced user, time can be saved by backing up each user file and the application folder in their entireties to another drive; do; either delete the Library folder contents or erase the drive; reinstall MacOS; boot from another bootable drive; copy back the user folders and application folder contents to the newly created drive; then boot from it giving the users the same identical names as they had before which will having them default to using the existing user folders containing all the preferences and setup previously present when they were copied. This will reduce the setup time when the drive is restored. Note their as several critical files kept in the User folder that can be lost if the User Library is not properly backed up. This includes any mail stored locally on the Mac, Notes stored locally, and Reminders. However if they exist on iCloud as part of your Apple User ID, they will be restored correctly.

This issue also points out the need to always keep frequent, good, and viable backups.

1 Like

Either stupid me or migraine fog. Simpler way to resolve this issue. After first installing MacOS or if you are sure your Library is not corrupt, simply do a Finder copy to a different drive or flash drive of the Library (root Library) as a backup. Then if it gets the Library in use gets corrupted:

  • Erase its contents
  • Copy and paste the backup Library into the active one
  • Immediately reboot into System Recovery
  • Reinstall MacOS

The reason to reinstall MacOS is to link the contents of the recovered Library to the system and insure the file permissions of the Library are correct.

After doing this you may have to reinstall device drivers such a printers or other Apps as device drivers an a few Apps use this Library for some of their files. However this is much faster and easier than doing an erase and restore and if you did you would need to do the same reinstallation anyway. That said I have not actually tested this but I have no reason to think it would not work and have successfully backed up my Library folder to another drive. However I strongly urge that you have backups of your data, Apps, and user libraries for the first time you try the restore just in case.

1 Like

The implication of this strategy is that the root Library folder is not update as you add applications or files or folders

– otherwise how does replacing a recent but corrupt root Library with a non-corrupt but “historic” (i.e., not recent) backup copy enable you to reinstall OS and everything is ok.

I am not saying your second approach is incorrect.
I just don’t understand how it results in a functioning up-to-date computer system.
Especially if you have not tested it.

Thanks

Bob

1 Like

Sorry about the confusion. If Apps utilize the root Library they usually add files to it during installation. So if the root Library file contents is replaced with a virgin functional copy, those previously installed files may not be present if any Apps were installed since the root Library was created. If so, such Apps may not work correctly. So the solution to this issue is to simply reinstall those Apps. That should repopulate the root Library with the correct files and fix the issue - i.e. a functioning up-to-date- computer system. So how do you know which ones to reinstall? If the App worked fine or properly launched before the replacement of the root Library and does not after, that would be an indication it is the App that needs to be reinstalled. Advanced users could examine the contents of each folder of the root Library before removing it to locate App names in the files in the folders which would indicate which Apps may need to be reinstalled. Often hardware drivers such as printers and scanners reside in the root Library and are most likely to need reinstallation.

I’m not sure that I completely followed the entire situation described in this thread, but I didn’t see any mention of the Sealed System Volume (SSV) or the Data volume and how they interact with each other to present a single “fused” volume to the user. If you are running into library corruption issues, it seems important to figure out whether the corruption is occurring in the SSV (extraordinarily unlikely) or the Data volume (highly possible). For example, third party apps only add libraries to the Data volume. Once that is sorted, the options for solutions become clearer.

The root Libary, sometimes refered to as the Normal Library or just Library is not in the secure area or the System folder. It is at the basic or root level of a drive, when you first open a drive along with the User and Applications folder. Apps can place files in this library. What generally is contained in this Library is things common to all users such a printer drivers, input devices, antivirus files, utilitiy files such as iStat, cloud files, such as Dropbox, added fonts common to all users, and launch agents.

This Library is on the Data volume but managed to a degree by MacOS as some common system file are located here such as the default User Picture. Also folder and file permission in the Library are also primarly managed by MacOS but 3rd party App can set or change permissions when installed.

1 Like

I really HATE that Apple have taken away the ability to easily clone a drive back to a machine. This, along with target disk mode, made recovery from a ‘disaster’ quick and simple (provided of course you had appropriate backups).

1 Like

Is that correct? As I understand it with the sealed/locked/signed/read only system volume…the thing that appears to be at /Library on the single partition that’s displayed in Finder is an amalgamation of /Library on the system volume and /Library on the Data volume just like the hard drive in Finder is an amalgam of the 2 partitions…so a reinstall would replace only the sealed system volume Library and thus the /Library op seen would be an amalgam of the system and Data volume ones. A similar thing is done with /Applications and also sometimes things get installed in ~/Applications.

I could also be completely hosed on my understanding though…and installers do weird things…for instance Topaz installers put applications either in /Applications or in /Topaz under there…and moving all of them into Topaz or Applications results in the moved ones no longer working. I know the system volume is a security thing and understand the amalgamation process so to not confuse users…but it seems to operate in ways that aren’t fully explained.

Me too…I know it is better for security but it’s really just the iOS-ification of macOS just like we’ve seen in the UI for things like System Preferences which IMO is a step back in usability…searching for what you want is the only good way to figure out where things are hidden now.

jweil,

Thanks. That makes complete sense.

An obvious implication is that – while macOS is going smoothly – it might be reasonable to backup the (root) Library folder “periodically” – every 6 months more or less, depending on how often one adds new program – just to minimize the number of apps one has to “reinstall” – which can be a time-consuming process. Obviously to the extent that corruption of the (root) Library could be subtle and inapparent for a while, it would seem prudent to save the backups – just in case.

One other question – if you know –
It would appear the (root) Library is “recreated” with (at least some) macOS updates. I have been using my current computer for over a year and the folder has a “created” date of 3 weeks ago (1 April 2023 – hopefully my current installation is not an April Fool’s joke ;-))).

There are 70 subfolders
– 62 have a create date of 1 Apr '23
– only 14 of those have been modified since
– of the 8 created before 1 Apr '23 – none of those was modified on 1 Apr '23
– 7 of those dates pre-date when I got the computer
– and the remaining 1 has a create date appox when I got the computer

My instinct would be that a recently created folder would be unlikely corrupted.
Do you think that a reasonable assumption?
If so checking occasionally for a new create date would make it a reasonable time to make the backup.

Of course – yours is the first post I at least have seen about the consequences of a corrupted root Library folder. Even if it is very uncommon, the knowledge of the consequences is valuable – definitely thank you for that. But it may not make it worth spending too much effort to anticipate dealing with the problem. That would suggest your “do the backup after you install a new macOS” is the most time-efficient advice ;-))

Thanks

Bob

In theory yes but in practice not necessarily. While the folders create date may indicate what you are suggesting, the files with in it may be different. Frankly I am not sure the issue is MacOS itself but other forces may be at work creating the corruption such as installers of 3rd party software that have bugs or Apps that are able to write to the files inside the root Library and are doing so incorrectly and possibly changing some file permissions. That may be the reason that reinstalling MacOS is not always able to fix a corrupted root Library as it does not know about 3rd party files and why isolating the corrupting is so difficult…

As far as you’re suggesting about installing a new MacOS, that is exactly what I am doing now that I have isolate some of my issue to a corrupt root Library. I feel this is the best way of handling the issue as after the fact the root Library could become corrupted but not manifest itself until some future date so backing up during that period would not necessarily fix the problems. By having a backup immediately after a clean install before installing any Apps or drivers you are assured of having a clean backup.

What is a “root library” and how does it relate to the SSV and Data volume as mentioned by @josehill? And what Apple references can you point me to about it? All I know is what I’ve read in Howard Oakley’s extensive postings, and Apple.

1 Like

I described this in an earlier reply:

Addendum: The root Library is created by the MacOS installer and it populated by some system related file that are in common usage by all users. This includes such things as Midi files, Some Apple preferences, Scripts, Security files, User Pictures used for creating new users, Web Server files, System Migration information, System Configuration files, etc. The rest of the files consist of files from 3rd party Apps, printer drives, and scanner drivers. Startup Items are also sometimes located here.

@jweil I think you are looking at the situation incorrectly.

There is no “Root” library. The Library that appears at root level is split between the SSV Library on the System volume, and third party stuff from the -Data volume, cleverly amalgamated to make it appear as a single Library. You can see what is in the SSV Library alone in Finder at /System/Library which only contains Apple stuff, no third party.

The same is true of the “Root” Applications folder. It is a clever amalgamation of the SSV Apple-only applications (viewable at /System/Applications) on the System Volume, and third party apps on the -Data volume.

The SSV is unwritable and verified extremely thoroughly every boot and if it fails the machine won’t boot. Corruption can only happen in -Data volume. Your regular backups of the -Data volume with TM or whatever, backup up whatever can be corrupted. TM only backs up the -Data volume.

1 Like

This ‘library’ seems to be an Apple stepchild as it is often referred to in different ways including Apple support. Recently a support agent told me it was referred to as just ‘Library’ or ‘Normal Library’. Just calling it ‘Library’ seems a bit confusing to me as there is a minumum of 3 ‘libraries’ on every bootable drive. In any case it seems to be symantics on what it is called. While it may be a jumbled mess of stuff from various locations, I choose to refer to it as the ‘root library’ as it is at the root level of a drive in a common folder named “Library”. If you have a better name for it that would uniquely identify it I would be interested.

Since I am not an Apple developer computer engineer, I can only assume your description of it is correct. That said frankly it makes no difference to me in the real world. What I do know is that it can get corrupted and Apple provides no way of fixing it other than an erase the drive and reinstall MacOS onto the drive. I also know that from a user prespective it looks like a single library containe both Apple stuff and files that support added 3rd party Apps. So for me my fixit suggestions work for me and I find it personally outrageous that if some of its contents get deleted that the reinstalling MacOS from Recovery ‘fix’ wipes out the entire drive without warning.

No matter what a “support agent” calls the SSV and Data volume, I’ll go with the guidance provided by Howard Oakley in multiple articles and as necessary, Apple support (though I trust Howard more).

OK. All I can say is that I have never seen deleting the Library contents recommended as a repair/recovery method, and I am not at all surprised that data is lost if this is done as a precursor to reinstalling macOS since the SSV and Silicon came along. Apologies if I have misunderstood exactly what you did.

I am also a fan of Howard Oakley. During my two and half years of testing and using M1 and M2 Macs his writings have been a great help to me in understanding the huge implications on repair, recover and reinstalling macOS since Apple Silicon and the SSV. The old ways of doing things don’t apply. Here is a link to one relevant article. There are many others.

3 Likes

One of the reasons may be is due that it is something that Apple Support refuses to support it and does not provide any tools to address any issues with it. It is also, from a users prospective, poorly documented and to support it would take a lot of time for the typical user. As such Apple’s shotgun procedure to fix it is to have the customer back everything up, and then do an erase and install of their drive. This may be great for Apple’s profit line but bad news for the user given how much time it takes with the additional risk of loosing data.

One of the key reasons that a user might lose data that few realize until it happens is that some user personal data is buried in the user library such as contacts, notes, email, and cloud server information that may be stored locally on their machine. Attempting to restore some of this stuff after an erase and reinstall can be quite dicy and Apple offers little to no support to recover it. This is especially true if Time Machine is being used. Apples position is to not store any of this information locally but store it on iCloud so it can be recovered from the network. Of course in order to do all of this, a user may likely have to subscribe for additional iCloud storage space, a fact that I am sure has not gone unnoticed by Apple’s beancounters whose mantra is to increase profitability in any way they can get away with it. I suspect the last thing Apple wishes a user to do is store this stuff on a hard drive at a fraction the cost of iCloud storage space and ignoring the fact that the stuff can’t be recovered unless you have a working internet connectioin at the moment. Since I refuse to subscribe to what I regard as Apple’s preditory behavior, I keep most of such data locally on my drives and have plenty of room left over from the complementary iCloud space and save $120 per year. A 1 tb hard drive is under $50 as a one time purchase with a life span of many years. Occasionally interchanging 2 of them with off site storage such as with a trusted family member, friend, or safe deposit box. This means you are not paying forever to store your data and if you Mac burns up in a fire from an earthquake with the internet going down for weeks or gets stolen, you can likely easily recover your data just as soon as you can get your hands on a replacement computer, with minimal loss or no loss of data, without being dependent on a good internet connection to do it.

I invite you to try a little experiment: Call Apple Support and ask them precisely where are the notes, reminders, calendar, contacts and mail actually stored. It is likely to be an interesting discussion and highly likely it will need to be escalated to a senior advisior. Hopefully you will get a better answer than ’ in the User Library’

The “user library” is located at /Users/username/Library. The Library that gets fused between SSV and Data partition is /System/Library. I can pretty much guarantee NO user settings or calendar or anything like that is stored in the latter. If it’s user data you are looking for, you need to resort to your backups. TM will do, but of course so will any SD clone. But there is no use in trying to fuzz with /System or the parts it gets from the Data partition or going to System Recovery or re-install. None of that is appropriate. Entirely different ballgame.

1 Like

I do not need to call Apple Support to discover where local copies of those data types exist…they are all in the ~/User Library and can be restored from a Time Machine or other back up of the -Data volume when necessary. Yes some of the locations are deep in the Library, and I agree Apple documentation about restoring these is not good.

Apple wants us to do things their way and I have learned over the years that in general it is usually the best way. Fighting it with complicated alternatives tends to lead to trouble. This has become more and more the case as the OS has evolved and particularly since SSV and Silicon. When I am setting up a new install I have no hesitation in signing into iCloud and letting it download all my data. I don’t have any reason for digging out the local copies on backups unless something has gone wrong like accidental deletion.

2 Likes