Use ChronoSync to Move Large Folders

Originally published at: Use ChronoSync to Move Large Folders - TidBITS

Moving large folders with the macOS Finder can become an exercise in frustration if it doesn’t gracefully handle errors. In such situations, ChronoSync Express from Econ Technologies can move your data more successfully.

1 Like

Great to know, thanks, Josh. This reminds me to review the rest of CS’s features again. I got my copy of the full version back in 2005 for $20, and Econ has just updated it for Monterey, as promised, por nada.

1 Like

Great article. One question: doesn’t Carbon Copy Cloner do basically the same thing or is there some advantage using ChronoSync Express?

1 Like

I was wondering the same since you can choose a folder, not just a schedule or full clone.

I have done this for many years with rsync in the terminal. rsync comes with MacOS.

Copy this text into a new terminal window

/usr/bin/rsync -avE -h --exclude=".*" --stats --verbose

After copying hit space once (or twice does not matter how many)

Drag the “Source Folder” from Finder to the terminal. This will add the path to the folder to the terminal text. Add /*.

Hit space.

Drag the “Destination Folder” to the terminal. Add /.

Hit Return (Or Enter).

Example (on one line): /usr/bin/rsync -avE -h --exclude=".*" --stats --verbose /Users/paalb/Vacation/* /Volumes/BigDisk/2022_Vacation/

You will get a log of all files, look through it and check for errors.

To get all details about the command, write this in the terminal

man rsync

Hit Return (Or Enter).

This has been beneficial several times in the past when I had disk crash where many of the files still were accessible and Finder just refuses to copy.

6 Likes

I don’t see why not. These apps are fundamentally GUI layers over rsync.

2 Likes

@jk2gs
Yes, I use CCC for the same purpose. I have a number of CS (and CCC) licenses, but I can use the CCC trial to quickly get a copy done if a licensed version isn’t installed.

I do appreciate @paalb noting the proper syntax for rsync. I’ve found myself having to work on a couple of Linux systems at the command line and finding the proper syntax has been a bit of experimentation. Eventually I get it right, but I’m going to copy these into a BBEdit note for future reference.

Cheers,
Jon

2 Likes

If you think this is bad on the Mac, try it on Windows. Windows Explorer is extremely slow copying and especially moving files. Windows UI is also limits file names (including drive letters and directories) to just 255 characters even though the underlying NT file system can handle 36,000 characters. It’s not
uncommon to move files to a subdirectory and find out you can’t access files due to that 255 character limit.

Copying files on Windows will usually take two to three times longer than on a Mac, and the more files, the slower it is. Usually on a Mac, if you’re copying 100 files, and you get an error, you’ll find out in a few minutes. You might not know in Windows for a half hour or more.

I use to run Cygwin on Windows just for the power of command line tools such as rsync. Now Windows Subsystem for Linux finally gives you the tools that have been available on the Mac since OS X 10.1.

Still, if the Mac has the full power of real Unix underneath its UX, why not use something like rsyc when copying entire directories from one place to another?

2 Likes

It’s even stranger than that

The original WIN32 API has a 260 character path limit (256 for the path, plus 3 for the drive-prefix (C:\) plus a terminating NULL character). The Unicode version of the file APIs have a much bigger limit (32K bytes). So apps may or may not see a limit, depending on which API they used. Yikes! :roll_eyes:

Since Windows 10 release 1607, however, the non-Unicode APIs (and Explorer) have been updated to support the same 32K path limit that the Unicode APIs use. But this is disabled by system configuration for backward compatibility. You can set a registry key or use the Group Policy Editor to enable support for long paths.

I’m not sure what the “backward compatibility” rationale is. The only thing I can think of is that maybe there are mainstream apps that use a hard-coded 260 byte buffer and may not be checking for overruns, assuming that the documented limit would never change. But whatever Microsoft’s rationale, I routinely set this registry key to eliminate the problem.

See also:

1 Like

This is precisely why Microsoft engineers wrote robocopy for internal use. It made its way to a free addon power tools kit. But it’s built-in to Windows nowadays. It’s a command line scriptable tool that can multithread to speed up copying many small files. It has logging and will skip problematic files.

1 Like

Our company had a Microsoft technical support person on site. We were developing in C# and constantly ran into this issue due to long classpaths. He and I got into an argument over this. I complained it was an issue. He told me path names can be 32K and I was making the whole thing up.

I wrote a small shell script in Cygwin that purposefully created an extra long path (Cygwin wasn’t affected) and showed him Windows could not access the file. Not Windows Explorer. Not Microsoft Office. Not Visual Studio. Not even Terminal unless you preceded the PATH name with \\?\, and then only some of the commands.

He was shocked. He knew that the underlying NTFS file system supported these long names since Windows NT was released and FAT file partitions were deprecated. There was even a Win32 API for accessing and opening these long paths, but not one of Microsoft’s programs didn’t use it.

A few days later, he got back to me. It was all done for backwards compatibility reasons. That was about two decades ago.

I’m glad that finally Windows 10 has the mechanism to finally fix this issue. Maybe, looking at Windows 10, they cleaned up all the old Windows 95 legacy stuff, so that all Microsoft’s programs can handle it. Still, there might be third party programs that can’t.

1 Like

Perfect timing for my setup! I needed to transfer all my movies from a 4GB to new 6GB (hard drive 7200rpm, of course) and it would have been impossible using Finder. (I would usually end up copying ten folders [actual movie, IMDB link, original .MKV transfer, etc.] or so; wait until those 10 were copied; do another 10…)

I’ll give this a try. Not all 500 movies at once, of course… I’ll try some trial syncs, see what happens, and proceed from there.

I appreciate the 7-day free trial, too, and never-ending free updates. (“No one is left behind!”)

I’m thinking Transmit5 using FTP/SFTP would also do the trick. Also a 7 day free trial.

Another workaround for this, is to zip your files/folders first, then transfer, then unzip.