Due to all the cautionary words here and elsewhere I put off my Catalina upgrade until a couple of months ago. Mostly seemed to go OK except for one really odd thing on which I’d value the TidBITS Talk team wisdom:
My file structure dates back to pre-OSX days. When OSX came along I saw no reason to change and it’s been working fine until Catalina. I use two top-level folders on my hard disk for personal and business documents. Almost everything important to me is in these two folders. On completing the upgrade they had disappeared. After panic and investigation (won’t bore you with the details) I found they had not actually gone but were relocated into a second instance of my hard disk.
The path to these folders now looks like this: MacHD > System > Volumes > MacHD (again!) > my precious data folders. (The Applications folder appears in both instances of MacHD, and both also have Systems, Library and Users folders but I have not checked to see of the contents are identical.)
After the initial bafflement I got used to this and simply worked around it. Until I found that Backblaze was no longer backing up my precious data folders. Turns out that Backblaze will not back up anything in the System folder, including these folders. Moving them back to the true top level doesn’t seem to work because they get relocated again, I think as a result of system updates (hazy about this because the last time was a couple of months ago and I’ve forgotten the details).
Is this file structure normal for Catalina? If not, any suggestions on how to fix it? I considered a clean install but after re-reading Joe Kissell’s Upgrading to Catalina book I’m dubious it would fix anything.
I can presumably work around this (and get Backblaze working again) by moving my data folders into my Home folder. I haven’t done that partly because I like having my key data at the top level of my hard disk, and partly because I’m troubled that something has gone wrong with my Catalina setup.
My advice is to put your data where Apple wants user data to be, in the Home folder. I’m surprised you haven’t experienced problems before, certainly other people have.
This is really Big Brother-ism. There is something to be said for having a separate system volume but it is t any of Apples business where we store our data. If we want a folder at the root of the non system data volume…the OS should let us make one and assign permissions as we want to. The idea that we should do this only the way Apple says we should is a bad idea.
I ended up putting those folders in /users/shared and aliasing them to my Home…and also putting them in the left pane of finder windows. Not ideal… it with Apple’s insistence that they and only they own the root of the volume it was the best I could figure out.
Since forever, at least days of my beige G3, I have always kept my data on separate partition or volume from system. Home folder only contains certain app-generated support files. Don’t use Photos. Still doing this,running Mojave. Data/files safe from System problems, OS update issues etc.
I’m sorry to say it, but yes, you should put your data in the Home folder and move on. Storing data in weird locations as you’ve been doing is a recipe for confusion and eventual disaster. I’m surprised you haven’t had more problems before Catalina. Catalina just takes it to the next level by putting the system on a separate read-only volume and merging that with the read-write data volume behind the scenes.
We all had our preferred methods of partitioning disks (System, Apps, Documents, for instance) in the days before Mac OS X, many years ago, but it became clear quickly that ignoring Apple’s default folder structure was only going to end in heartache. Too many apps and utilities assume that certain data is in certain places.
At this point, all documents are on an external drive, and if I ever upgrade to Catalina, I assume (hope) they can remain there, using home documents folder only for app-generated stuff like bookmarks, avatar images, odd Adobe stuff.
I don’t think that should be a major problem, but I’m not sure what you gain by keeping everything on an external drive, other than slower performance (assuming you have an internal SSD). Generally speaking, the only reason to put things on an external drive is if there isn’t sufficient internal storage for it.
You also end up with a more complex backup situation, since you’ll need to back up both the internal drive and the external drive, and in the event of problems, restore both. That could be problematic in the event that you don’t have a backup set of hardware that exactly duplicates your main set.
So what about installers that create stuff like ~/Applications? Or user compiled code? Where should that go to remain as independent of the OS’ structure as possible?
One more backup isn’t a problem. I already have redundant backups for that drive and photo archives. I ranched decades off grid, a day’s drive from any tech support, so this “belt and suspenders” routine is second nature, even though I now live in “civilization.” I do keep my Lightroom catalogue (not its photos) in iMac’s Home documents folder for optimal performance, but that catalogue, too, is automatically backed up externally. Old dog, new tricks… if it ain’t broke, why fix it … Obviously not for everyone
The question is, if something does break, will you be able to restore reasonably? I thought my backup strategy was pretty good until the internal SSD on my iMac failed. Then I discovered that having a bootable duplicate on an external hard drive connected via USB 3.0 was worthless because it was too slow to boot the iMac and work reasonably. Times change, and what worked before may not work again, particularly in an emergency situation.
That is a standard install location for apps meant for a single user rather than all users of a machine. I believe that I used to have some personal Automator applications that would install into ~/Applications as well. (I don’t use any of those anymore.)
And now that I look I see that I have Chrome Remote Desktop installed in ~/Applications on my account, which must be back from when I was documenting how to install that for some friends of mine.
Sorry, but some of us disagree. Catalina’s system volume is bad enough…but at least it makes some sense. Putting data on a different volume is fine…but when Apple makes a volume named Macintosh HD-Data…the user should be able to create and use whatever folder structure they like on that volume. Apple thinks that common data should go (presumably) in /users/shared on that volume…but if a user wants it at the root…and it isn’t overwriting any of the folders Apple wants at the root…then Apple should allow that and leave the user created folders alone. Yeah…we can make it work if it’s all in users/shared…but there is no disadvantage to Apple for users to have it the way the user wants it. Better Apple focus on other trivial stuff…like stability and whatever bug is causing the Mail issues.
It is user data and the user should be able to have it however they want…absent am actual security problem or other issue.
You don’t tug on Superman’s cape
You don’t spit into the wind
You don’t pull the mask off that old Lone Ranger
And you don’t mess around with Jim
Obviously, you’re welcome to do whatever you want, but it’s going against the assumptions that Apple and many software developers will have about where data will be stored, such that you could run into problems or even lose data.
Personally, I’d prefer to tilt at other windmills since this one has been in place since 2001 and won’t be changing.
Thanks to all for the feedback. I guess I’m going to have to fall into line and put all my stuff in the Home folder. Despite Adam’s comment that this has been in place since OS X arrived in 2001 I must have missed every message about it over the last couple of decades.
One more question, which is going to make me sound like an absolute novice despite starting my Mac experience with a Classic in about 1987: Can my data files can live anywhere in the Home folder or do I have to put them in the Documents folder for everything to work properly?
(Since I don’t consider myself a novice it’s pretty frustrating to find myself in this situation of asking stupid questions. Use of MacOS has always seemed pretty obvious, until now.)
No…it hasn’t been around since 2001. I have had folders at the root of my drive ever since I ran an Apple ][GS for my use and the os has never cared. At some point…I had to start using admin credentials to create and set everyone R/W permissions on them but after that they just worked.
As I said…I can understand the reasons for making the system volume read only…but the Data volume should not be and as far as I can tell has no need for this as long as you don’t try to say create a folder named users. Creating and setting perms on a folder named Junk should be allowed…and moving it to relocated items is simply not necessary. I’ve yet to see either Apple or any post explain just why this isn’t supposed to be allowed.
You can create whatever folder structure you want within your Home folder and it’ll work as you expect. You may live without some security protections that you’ve likely already been living without or don’t care about.
By default, the root of the Home folder is readable by all user accounts but the default folder structure (Desktop, Documents, Downloads, etc.) only permit access by the owner of the Home folder. When you create a folder at the root of the Home folder, it inherits those world-readable permissions by default. You can change there permissions for the folders you create, they don’t have to remain world-readable and subsequent system permission fixes won’t revert them.
You may not have any other actual users that you need to protect your files from but file permissions can be another layer of security. Say some malware gets on your machine, allowing it to exploit a system account like _locationd, the malware can access anything that account can so it can read those world-readable files and folders from the root of the Home folder but not the contents of Desktop, Documents, Pictures, etc.
A newer layer of security, that I’m a little fuzzy on, is macOS letting you limit file system access for individual applications you run. To access Desktop, Documents, or Downloads, and only those folders, they need your permission. You can give an application permission to read the Downloads folder without letting it access Documents. In the Security & Privacy preferences, this is the “Files and Folders” setting under Privacy.
How might “Files and Folders” be related to not storing your documents within ~/Documents/? Say you try a little menu bar application that reminds you went tea time is but unbeknownst to the developer, someone snuck malware into their code that tries to exfiltrate data. The application shouldn’t need to access files at all but it silently can except for those in Desktop, Documents, or Downloads; for those folders, it needs your explicit permission, even though the application is run under your user account. Again, I’m fuzzy on how the “Files and Folders” feature is meant to work, it’s possible I’m mistaken about its relevance to storing files in ~/personal/ vs. ~/Documents/personal/.
Sorry! Yeah, it was a big deal in the early days of Mac OS X, since the closest Apple had ever come to enforcing a directory structure before was the System Folder. I remember all sorts of complaining because Apple’s default folders didn’t necessarily match all that well with how people did things. It’s possible that this even stemmed in part from Mac OS X’s Unix heritage, since Unix has directory structure conventions with /var, /usr, /etc, and so on.
Obviously, lots of things outside the recommended hierarchy have been possible, but usually at a cost of extra effort in setting permissions, fighting with app expectations, and difficulty troubleshooting.
Not at all. It adopted the Unix mentality, which is significantly older than anything Microsoft developed.
Any multi-user system (and macOS is multi-user, even if you’re the only person who ever uses it) must keep each user’s files separate. The home directory is the mechanism it (and all other Unix-like systems) uses.
