Apple’s File Provider Forces Mac Cloud Storage Changes

[ddmiller (Doug Miller)]

@Sebby

and Syncthing for syncing point-to-point between computers whether on-net or off (using a bag of tricks including NAT hole punching and relays). These are command line tools,

I’m a big fan of Syncthing. It’s what I use to sync my desktop and download and local documents folders between all of my Macs. Sure, it has a command line interface if you want, but you can do everything easily from a local webpage on each client. It does take a while to reconnect on Intel clients sometimes, but on Apple Silicon it’s up and running in no time.

[Sebby (Sabahattin Gucukoglu)]

I was a bit confused, wasn’t sure if we were talking about the same thing. Then I realised that just because I used MacPorts and hadn’t visited the SyncThing site in ages didn’t mean there wasn’t now a non-command-line way. And, sure enough, there’s now a really nice native Mac app. That’s awesome. Now I can get more people using this, no more excuses.

[mikebhm]

@ace

You should be fine using iCloud Drive this way. Making a file online-only is just the same as with Google Drive—Control-click the file or folder in question and choose Remove Download. I don’t know why Dropbox doesn’t present those commands.

Two points.

1. Dropbox has the same “Make online only” and “Make available offline” by right clicking the item:

2. iCloud Drive’s “Remove Download” and “Download Now” are not working consistently for me. I have successfully uploaded a 68GB test folder to my iCloud Drive. After it had uploaded (watched pie chart and progress bar) I right clicked “Remove Download”. The symbol next to the folder changed to the cloud-with-arrow and the right click option changed to “Download now” both of which as expected for an on line only item.

However after four days “Get Info” shows only 11GB of it is on line only and 57GB is on disk. It has been stuck at this for three of the four days. I repeated the exercise and it stopped at the same point 57.21GB on disk:

Can anyone else who thinks they have on-line folders in iCloud Drive, please do “Get info” to see how much is still on disk?

A test on a much smaller folder worked correctly. Get Info showed it was iCloud only.

Incidentally Making the same 68GB test folder to be on-line only in Dropbox worked correctly within a few hours.

Thanks

[dave1 (Dave K)]

@mikebhm

However after four days “Get Info” shows only 11GB of it is on line only and 57GB is on disk. It has been stuck at this for three of the four days

I feel your pain. I mentioned in another thread that this is why I ditched iCloud Drive for mass storage in favor of Dropbox. iCloud can get stuck for days. Worse, it can make your only copy of the file inaccessible even for reading when it’s in this stuck state.

Now, if the File Provider API is the cause of this stuckness, we’re all screwed. Apple is replacing what works with what doesn’t work. And when it comes to reliable protection of and access to your files, nothing in computing is more important.

[jeremyn]

Google Drive’s switch from FUSE based functionality to Apple File Providor has completely destroyed our shared video workflow costing us hundreds of thousands of dollars in lost time and attempts to find new solutions. Our entire company is run off of Google Drive including hundreds of TB of video footage. Suddenly all of our video timelines are broken, design files are broken, DVD & Blu-ray projects are failing to load due to paths changing. Basically links everywhere are messed up. In the past we could all use the same paths, now every path is unique to the user’s computer due to being buried inside everyone’s unique ~/Library folder.

And to make matters worse, even when I do fix all the paths, opening a video project in Davinci Resolve results in sometimes HOURS of waiting while all the related files cache locally, assuming their boot drive doesn’t fill up in the mean time and completely crash the system.

With the FUSE base system Google Drive could just load the frames of exactly where the playhead was sitting rather than forcing a full download of each and every video and audio layer touched by the playhead. Utterly unusable now.

I’m diehard Apple fan going back to the 1970s, my License Plate on my car was MCNTOSH for over a decade including the dark days of the late '90s, but I swear the software division of Apple isn’t even the same company anymore. Their priorities are all messed up. They don’t give a &^# about true video professionals who actually push their amazing hardware in the way the hardware designers intended.

[jzw (Jolin)]

@Simon

I’d like a big master switch somewhere that allows me to turn everything to offline-only all the time.

Offline-only is easy: just move everything into your Documents folder and cancel your account with whatever cloud sync provider you were using

@deemery

In particular, removing “save as” as a choice without a modifier key pisses me off each time I want to put a file into a -specific location of MY CHOOSING- within a file hierarchy.

You can easily restore the old behaviour and get Save As back in all File menus (in fact, it’s so seamless I forgot that Apple had changed this, as once you make the change it sticks through upgrades, etc.):

Putting Save As back on the File menu by default

You can restore Save As… to its rightful place (both in the menu and on your keyboard) by using the Shortcuts tab of the Keyboard system preference. Simply add the following App Shortcut for All Applications: Save As… ⇧⌘S [image] This will put Save As… back in the menu without holding down option (or any other key) and allow you to use the ‘standard’ keyboard combo for it as well. Duplicate will still exist in the menu underneath Save As… for those times when you want it. (You c…

[david19 (David Weintraub)]

Saving Cloud Files to External Drives

What if you change the location directory of $HOME/Library/CloudStorage/xxxxx where xxxxx is a folder to a symlink to an external drive?

$ cd ~/Library/CloudStorage
$ cp -R DropBox /Volumes/external/DropBox
$ mv DropBox DropBox.old
$ ln -s /Volumes/external/DropBox DropBox

And if it all works…

$rm -rf ~/Library/CloudStorage DropBox.old

And if it doesn’t work…

$ cd ~/Library/CloudStorage
$ rm DropBox  #Symlink
$ mv DropBox.old DropBox
$ rm -rf /Volumes/external/DropBox

It’s been a while since I did Unix shell scripting magic, so there might be syntax errors. But it seems to me this might work. Directory symlinks are normally treated as directories they’re pointing to in Unix except if the process verifies it’s a symlink and not a file. cp command can detect symlinks depending upon the option and so can find.

I did this a few years ago to change around my Desktop Folder. My Desktop Folder was a symlink to point to various Desktop folders I could switch around. The idea is when I was working on a particular project, I could run a shell script to switch my desktop to be the files in that project. I had some app that allowed me to create a desktop menu of shell scripts, and used that to change around the Desktop folder.

Unfortunately, MacOS found the setup too confusing whenever I switched the desktop folder, and I ended up giving it up.

[ace (Adam Engst)]

@david19

What if you change the location directory of $HOME/Library/CloudStorage/xxxxx where xxxxx is a folder to a symlink to an external drive?

Well, that’s the question. The theory is good, but someone needs to test it for real. Since Dropbox has been iffy with symlinks in the past, that’s not something for the faint of heart.

[james.cutler (James R Cutler)]

Another View of "Apple … forces … storage changes. " This is not a new problem for user who depend on external drives to hold media files. Apple has done this to several apps they supply. Here is my take on use of an external volume to provide enough storage for media in ~/Library. It could also be used to accommodate a large Dropbox folder within ~/Library.

iTunes managed multiple media types from a single interface with excellent metadata management and location independence for the iTunes folder. (soft links). When Apple dismembered iTunes, it moved Audiobooks, epubs, and pdfs to ~/Library/… This meant, as my library grew, I ran out of space on my DATA volume. My solution was to move my Home directory to an external volume on an external drive. This requires only one edit to System Settings(Preferences) to change the location of the Home directory. This must be done after the old Home directory has been cloned to the new location. I used Carbon Copy Cloner because it preserves essentially everything, including UID, permissions, and softlinks to external Movies, Music, and Pictures folders.

Caveats below do assume modest competence in Terminal and macOS user management. If you lack this, find some who has the competence or solve the space problem by throwing more money at Apple.
Caveats include:

1. Use a separate admin account, not the user account being migrated…
2. Make backups first, multiple if possible.
3. Create the external volume and new Home Directory before changing any System Setting.
4. Verify the copy (using CCC) if you can.
5. Change the user’s Home directory location using Advanced Settings.
6. Restart macOS.
7. Login the migrated user. [Magic Happens]

After some time, Delete old Home directory (rm -rf /Users/oldhomediretory)

Regarding updates while main user account is on external volume:

1. Log in to the separate Admin account
2. Dismount the external volume containing the user Home directory.
3. Physically disconnect the external drive.
4. Perform update
5. After declaring success, connect the external drive.
6. Restart macOS and login into the regular user account.

In my experience, major upgrades are almost guaranteed to mess up if this sequence is not followed. Application updates have been no problem.

[chirano (Conrad Hirano)]

@david19

$rm -rf ~/Library/CloudStorage DropBox.old

You need a slash instead of a space between CloudStorage and Dropbox.old, or else you’ll wipe out the whole CloudStorage folder.

[macsolu]

@ace ,

What’s your opinion (this is also open for comments from the readers) about a professional design agency storing and actively working on Photoshop, Illustrator, and InDesign files exclusively on Google Drive? Yes – each designer would, ahead of time, first choose to make their file offline / downloaded locally, then remember to change the file’s status to online-only upon the completion of the project.

I’m interested in feedback about the pitfalls and possible troubles of depending on such an approach. Does anyone know of a design firm that has chosen to operate in this fashion?

Also, I’ve heard that the app entitled Arq will allow for syncing files & folders out on Google Drive with an in-house file server running macOS Server software. Anyone have experience with Arq and Google Drive?

[dave1 (Dave K)]

@ace

One thing that does seem to be lacking in the File Provider world is a setting to make everything offline all the time, disk space permitting.

I can’t comment on the File Provider API implementation since it hasn’t rolled out to me yet. But traditional Dropbox (which I’m running) certainly makes that quite easy. Just select “Make available Offline”:

at which point it downloads, you get a solid green circle badge, and it stays offline until you change your mind.

Is this functionality gone under File Provider API?

[ace (Adam Engst)]

@dave1

But traditional Dropbox (which I’m running) certainly makes that quite easy. Just select “Make available Offline”:

Yes, that’s the same in the new world order, as I note in the article. But there’s no setting in Dropbox to say that everything should be offline, all the time. You’ll have to make things offline manually. It should stick, but I’m not sure that’s guaranteed.

[dave1 (Dave K)]

@ace

But there’s no setting in Dropbox to say that everything should be offline, all the time.

I’m confused. Here below you can see that you can go all the way up to the top folder and say Make Available Offline (all the time), and you are saying it’s the same in the new world order, but you’re also saying there’s no such setting:

[Simon]

So are you saying that is a recursive command that applied once will then affect any ~/Dropbox/* contents from here on out?

[dave1 (Dave K)]

@Simon

So are you saying that is a recursive command that applied once will then affect any ~/Dropbox/* contents from here on out?

It’s absolutely recursive. You can open the subfolders and their subfolders and see that they all have the green circle badge. And when you “force offline” like this, I don’t think it will ever remove them locally without your permission.

[xdev (Marc Z)]

@dave1

You can open the subfolders and their subfolders and see that they all have the green circle badge.

Yes, but when you add new files from another source (not that Mac) will those files automatically be on disk or only in-cloud? I suspect the latter, so you keep having to manually tell new files to download.

What bothers me about this kind of thing is I use cloud-syncing like Dropbox and One Drive as a way to move files around between devices. It’s frustrating when I go to the other device and the files aren’t actually there and I have to do extra steps to force them to download and then wait for that download to happen. I want it to be automatic like in the old days.

[jajvj1 (Jeffrey Jones)]

I do some photo editing for a community newspaper. Photos that they want me to look at are placed in a folder shared via Dropbox. I set the folder to “Make available offline”. Yes, anything they put in that folder is automatically downloaded to my computer.

[s1dorner (Steve Dorner)]

One result of all this has changed my own workflow significantly. It used to be that if you changed a file on a Google drive (via Finder overwrite or application save), the new contents became a new version of the old Google drive file. Crucially, the link to the file on Google Drive didn’t change.

Now, such overwrites are new files with new Drive links, meaning any references to the old file (from other documents or web pages or emails or …) give the old contents. Not pleasant.

The workaround is (queue the irony here) to update files only by dropping them into Drive in a web browser.

It took me a while to figure out that this was going on. Hope I save someone else some time.

[richard.peach]

Thank you for this article. I ran into problems with Dropbox due to a crash of my iMac (Catalina) & so an upgrade to a Mac mini (Ventura) via my Time Machine back-up. Saved this article for future reference