In the thread I created on SNB linked previously, some responses mention other threads there in which others are looking at running SMB v4 and other changes that could be make it easier for an end user to run Time Machine in such a configuration. I hope that if these experimentations are successful, then Merlin will include them. I’m less familiar with other alternative firmware, but indeed if one manages to get it to a state where Time Machine can add the share in its GUI, hopefully they’ll share the work.
1 Like
I thought i was being clever and did a ssh 'btrfs send <btrfs_snapshot>' | btrfs receive <btrfs_snapshot> between my two family network TimeMachine servers running LInux as described above, so that I could seed backups on the portable server.
There were no problems with the new portable server serving the btrfs snapshot (re-snapshotted to be rw) and Time Machine on the client recognised the snapshot and offered to inherit it. But for some reason, for the new server, the client won’t pass verification or allow backups to start.
The worrying thing for me isn’t that there is a problem per se, but that I can’t easily work out where the issue is. The TimeMachine “option click” choice of Verify doesn’t report any logs on failure. And after googling about this for some time, I’m not confident that fsck_apfs can reliably fix an APFS sparseimage over the network. Since TM is running fine to the source of the clone, I know there is almost certainly something wrong in the image or local metadata; I just can’t find it.
There are 6 serious issues with networked APFS volumes listed at the Electric Light Company’s " How should you check an APFS backup store?", not to mention the article from the same source " Time Machine to APFS: When a network backup goes wrong.
I was delighted with my btrfs snapshots of APFS TM network backups being so lean and thought this would be a brilliant way of hedging against another APFS sparseimage corruption. However, if I can’t recover from these snapshots there is no point. And looking around, tools for recovering APFS network volumes don’t seem very good.
I’m wondering now if a home-grown rsync solution is the way to go. I believe that is what Carbon Copy Cloner uses. With rsync checksumming and btrfs snapshots I know i can retrieve the data, and easily sync the backups themselves between servers. Retrieval would certainly be more painful for family members though.
I’d be grateful for any comments on the reliability of APFS TM volumes and/or thoughts on the suitability of rsync, at least for file backups rather than system restores.
I used rsync to back up the Linux servers I maintained, and could re-create complete systems with just those backups, though it required a certain amount of esoteric knowledge. I never had a problem and I had a high degree of confidence that I could recover from any filesystem-related horrors.
I was happy enough with it that I used a similar script to back up my macOS system that I used along side Time Machine, CCC, and SuperDuper backups (yeah, I’m that guy). In all my many years using post-OS9 Macs, I never once had to do a full system restore, but I did feel that I had a system secure against file data loss. I had few incidents where TM would, out of the blue, present an error like “unable to use this backup: erase this disk and start over” which really decreased my confidence in TM. I also, several times, went looking for a lost file on TM only to discover that it was one of the many types/locations of files that TM apparently does not back up. OTOH, I never needed a file that wasn’t available in my rsync or CCC or SuperDuper backups.
Thanks for your response @ron. Yeah, I’ve been responsible for backing up 10s of Linux servers and 100s of TBs of data with rsync. We used to use hardlink copies for snapshots. Using snapshotting filesystems on the rsync target, like btrfs or zfs, makes things much easier these days.
I’ve compared backing up my linux machine with rsync to a family member’s experience with time machine. Both machines have the same ~400GB of data and roughly the same amount of weekly data churn. rsync + btrfs snapshot is about 3 times faster backing up to the same nvme external drives. My snapshots are to btrfs on luks encrypted drives too, so it’s not the encryption making it slow. I’m surprised Time Machine is considered fit for purpose.
It would be good to know if you still using rsync to backup your Mac. If you are instead using CCC, is that because of its capability of restoring a whole machine?
I’m editing this to add that I’ve just discovered that rsync has been replaced in Sequoia 15.4 with openrsync, a BSD licenced variant with only a subset of rsync’s features.
A year or so ago, disturbed by a number of the goings on at Apple, I decided to try putting my money where my open-source mouth is and switched to Linux as my daily driver. Up until then, I was still backing up with rsync and Carbon Copy Cloner.
1 Like
The problem for us all is that Time Capsules were a really elegant solution, easily managed, that handled a surprising number of use cases. There simply is no available alternative that matches their simplicity and sturdiness.
I guess something ike TrueNAS with ZFS snapshots and TimeMachine support might be a good solution. Perhaps TidBITS could recommend a setup?! But my worry, based on my discoveries noted here, is how robust APFS sparse image files are for network backup. That wouldn’t be such an issue if block or extent based snapshotting solved an APFS corruption issue but my experience suggests otherwise (see below).
1 Like
After reading this article, I ordered a USB3 case and connected it to my Asus ZenWiFi XT8 router along with a 3Tb HDD I had lying around from a long-dormant 2010 Mac Pro. I formatted the drive as HFS within the Asus web interface. Then, also within the Asus web interface, I turned on Time Machine. I went back to my Mac, opened up the Time Machine preference pane, and when I added a new drive it was able to see the newly formatted HDD connected to my router. It looks like it will take around 4 hours for the initial backup. I don’t know for sure if it’s using SMB instead of AFS, but I imagine it’s likely to be SMB. I suppose I’ll find out for sure once they completely remove AFS support from Mac OS X. I’ve still got a Time Capsule (with a somewhat recently updated HDD inside), and I’m still backing up to that as well. But it’s nice to now have a second alternative up and running for when the time comes that OS X can’t use the Time Capsule anymore.
1 Like
I don’t know if your router can be configured to use AFP instead of SMB, but since Apple has been discouraging AFP for awhile, my guess is that it is using SMB only. If you’re interested, you should be able to figure out if your router is running AFP by using a Bonjour-capable network analysis tool.
A good one to try is Discovery - DNS-SD Browser, available for MacOS and iOS. If you launch it, it will scan your local network for Bonjour services. Time Machine requires that AFP and/or SMB “advertise” themselves via Bonjour.
You’ll almost certainly see an entry for “_smb._tcp.” or similar for your router, indicating that SMB services are active. If your router is also running AFP, you should see something like “_afpovertcp._tcp.” associated with your router’s address. My guess is that only the SMB entry will be there.
The Time Machine support of Asus routers (currently) uses AFP. You can know for sure by looking at the System Log, you will see mentions of AFP there.
I realise this discussion thread is really about Time Capsule and the imminent loss of AFP network protocol support. As noted previously I’m a Linux user trying to support Macs for the most demanding of users: family members. My note here is about the use of APFS (previously HFS) sparseimages by TimeMachine, and whether that is a suitable format for backup. Excuse the further somewhat off-topic post.
My approach has been to develop a secure network backup for Mac users in my family using reliable and well-know software such as Samba (which has SMB 3 support) and rsync file synchronisation, using the smb shares as a TimeMachine target. A key point of the design has been to mitigate APFS sparseimages created by TimeMachine becoming corrupted and trying to work around that by snapshotting APFS sparseimages.
My tests (which are perhaps flawed) show that I can’t restore a snapshotted APFS sparseimage after cloning it using btrfs snapshots or rsync (with the extended attributes flag). TimeMachine is unable to complete the verification of the snapshots used in my tests. The Mac hdiutil tool fails to attach or verify the networked image, making it impossible to repair with fsck_apfs. This may be because of some basic problem in how I’ve setup things, of course.
However the Eclectic Light Company’s post on sparse bundles suggests some fundamental flaws with APFS sparseimages that are likely to make them unsuitable for backup, with the author concluding:
Despite their widespread use and advantages, macOS features supporting the use of APFS-format sparse bundles are at present riddled with bugs (I count 6 above) and have serious shortcomings. Their inability to self-compact is a major failing which makes their use clumsy at best, and Disk Utility’s lack of support for basic maintenance functions through the last 2.5 years reflects badly on Apple’s engineering priorities. Disk Utility remains of early beta-test quality and far from complete.
Unless things have changed since that post in 2020, perhaps Mac users should shy away from TimeMachine (for at least data if not system backup) and consider other tools such as rclone or Carbon Copy Cloner instead.
Ivan here, author of the original 2020 article trying to come up with a good Time Machine replacement. I’ve been meaning to pitch Adam on an updated version of that article, since, unsuprisingly, some bits are out of date. But, I’m not exactly sure what it would say, since my conclusion on modern macOS is to avoid Time Machine over Wi-Fi, always.
The following conclusion is what I’ve landed on regarding Time Machine backups over Wi-Fi, with or without Time Capsule: their existence inside an APFS bundle makes them not as reliable as a backup needs to be; and adds a layer of abstraction I don’t want.
So, don’t use Time Machine over Wi-Fi on any device. Instead, use a network copying app, and a backup host with a snapshotting file system. An example (not saying best choice, just an example) would be using a Synology NAS formatted with Btrfs, and Carbon Copy Cloner or ChronoSync on a schedule to just copy files to that NAS. If you need to go back in time, in theory Btrfs should have your back, according to the thinning policy you specify in the NAS.
But that’s really all pretty cumbersome. While pricier, the answer that makes me the happiest is: Mac mini, headless. (Or any other Mac you have lying around.) Forget that it can serve Time Machine; I don’t care. I get to copy to an actual APFS volume with CCC, and CCC on the receiving side will automatically make an APFS snapshot when complete. And you can specify a thinning policy for those snapshots in CCC, too. And you can use external SSD’s for easy flexibility. And it’s just all APFS! No tricks! No abstraction!
Right now, CCC network copies are just calling rsync under the hood, so they don’t have the bells and whistles of local copies, but Mike Bombich has suggested to me that a future version may have a copying engine for network drives with similar capabilities as local drive copies (and better speeds).
If you need to administer the headless mini, screen sharing awaits. And, if you need to restore a bunch of stuff pronto without network bottlenecks and complications, you can just detach the backup drive from the mini, and plug it directly into your own Mac. It’s APFS! No foreign file system nonsense!
(Speaking of which…It’s a digression, but I wouldn’t necessarily trust HFS+ formatted drives supported by non-Apple routers and NAS products, as I saw mentioned. Those are depending on a Linux implementation of HFS+…which is maybe fine, but I’m certainly not trusting my backups to it. And, worse, there’s no support for journaling when writing, which is key to preventing file system corruption in case of something like an unclean restart.)
One concern about this setup is security: if you have FileVault enabled on the mini, then, if it restarts for any reason, you have to physically log in to it. If you’ve got a keyboard attached and are in proximity, fine, but those are two things I don’t want to count on.
What I propose: the external backup drive(s) can be encrypted via the usual macOS method for doing so (control-click > Encrypt). Don’t ever save the password in the keychain. Don’t use FileVault for the startup drive, and don’t keep anything sensitive on it. If the Mac restarts for whatever reason – some sort of email notification would probably be a good idea, thought I don’t have a ready method for it – you can remote into it, and remount the backup drive(s) by entering their encryption password(s).
Alternatively, Apple silicon Macs appear to be reasonably secure even with FileVault disabled, as long as Find My is enabled. When Find My is enabled, macOS requires that an admin password be known to get to Recovery mode. And, unlike Intel Macs, they don’t offer Target Disk Mode without access to Recovery mode. So I think that, without an admin password, there’s no way of getting the data off an Apple silicon Mac when Find My is enabled, even if FileVault is disabled. That means that if you want to save the drive passwords in the keychain, they’ll auto-mount when the computer restarts, but if someone were to walk off with your system, they’d still be unable to get into it, or get the data off the drive, without knowing your admin password (or the drive password, if you choose to use a different one).
8 Likes
My old Linux-powered 2012 Mac Mini NAS ran Netatalk 1.x to serve AFP, purely because it could store all that silly Mac-specific metadata in files and databases, and that was good enough for me, especially since the speed advantages of SMB weren’t that much greater with carefully tweaked buffers, and Samba was (and is!) a tangled mess of obsolete nonsense that needs a specific extension to support Apple’s SMB client at acceptable speeds, which imposed the same requirements on the host filesystem for extended metadata.
Then the netatalk people released the 2.x series, dropping support for filesystem-based metadata, increasing the dependencies required massively (to include, for instance, Tracker for search), and dropping a load of legacy support for AppleTalk and authentication methods long-dead. I was very surprised by this, since it was already pretty clear, by then, that AFP was going away. And so it has. I wonder if they felt it was worth it? The 1.x series wasn’t beautiful, but it worked.
None of which much matters, of course, because as you rightly point out, Time Machine over the network is just a novel form of Russian Roulette with your data. Until CCC figures out snapshotted remote copies, I suggest a network backup tool like Arq or Duplicati (or whatever), that has no requirements on the target filesystem beyond that it stores and retrieves objects, which will preserve metadata and be efficient during backup runs. This obviously isn’t quite as transparent or flexible as whole-Data-volume backup performed by CCC, so here’s to hoping that in future it is possible to do such a backup over the network.
And, yes, just get a Mac Mini for a storage server that keeps your data on APFS volumes, if your budget and demands allow. You know it makes sense.
2 Likes
A little off topic, maybe, but since a few people mentioned that they are using ASUS routers:
I realize that this exploit is not directly related to AirPort/Time Capsule security, but I would be extremely nervous about exposing an AirPort router or Time Capsule to the Internet or even enabling wireless functionality if others can get within range. There is very little information on the net regarding AirPort vulnerabilities, but I note that the AirPort devices run a BSD networking stack and other BSD components that have not been updated in several years. Maybe AirPorts aren’t considered big enough targets to go after, but maybe not.
2 Likes
And this post from Bruce Schneier’s website a few years ago shows home router security is, unfortunately, not a recent problem:
“Most routers are designed offshore, by third parties, and then private labeled and sold by the vendors you’ve heard of. Engineering teams come together, design and build the router, and then disperse. There’s often no one around to write patches, and most of the time router firmware isn’t even patchable. The way to update your home router is to throw it away and buy a new one.”
1 Like
Home router security discussion continues at Home router security.
1 Like
I was interested to see the conclusion by @IvanExpert about coming up with a good Time Machine replacement. Apropros of that, I’m writing to report that I’ve successfully concluded my experiments described above to setup backup of Macs in my family to two small Linux servers using TimeMachine over SMB. These are both now working well. One is a 2014 Mac Mini retrofitted with a 2TB nvme drive, the other a Mele device similarly equipped. Each server cost <£250 all in. It’s nice and easy to work with nvme drives.
I had significant difficulties using the TimeMachine interface for adding network targets but I was able to do it without trouble using the tmutil command. The backups are now performing well over Wifi with sync over 1:1 ethernet cable nice and fast for bulkier transfers.
I’ve set the servers to snapshot the APFS sparsebundles weekly. Interestingly, presumably due to the way APFS bands work, these snapshots are very space efficient.
The CCC approach is very interesting. I personally use rsync to backup my Linux machine to a luks-encrypted btrfs volume on the same backup devices – it runs at least twice at fast as TimeMachine for the same volume of data. However I believe CCC network backups will not allow full machine restores. Also, despite @IvanExpert’s thoughtful advice of using a Mac filesystem for backups I’d prefer the bitrot protection, compression and snapshotting capabilities of btrfs since APFS sparsebundles seem prone to corruption.
Anyhow, If anyone would like me to write up a step-by-step guide of how to setup a Linux server for TimeMachine backups, I’m happy to do that.
2 Likes
Thanks for the heads up with this article. Many interesting post here, but often pretty technical. What I really want is just to be able to continue using Time Machine wirelessly by connecting an SSD to a router. I’m able to do that reliably now with an (older?) TP-link Archer C9 (that I got for free from a neighbor and migrated to after my Time Capsule failed a few years ago), but after updating to OS 15.6.1 today I got the message in Time Machine that AFP will no longer be supported in the future, so I guess the Archer uses AFP (not sure how to check that).
I did some checking now on the internet - like with the Linksys Velop mentioned by Gordon early on - but it’s really hard to determine if Time Machine backups with samba are supported. I searched the entire Linksys website, failed to find any mention at all of Time Machine.
Is there any available router right now going forward for this simple low tech solution?
I’m posting an update on this strategy of snapshotting TimeMachine backups on a Samba network share in case anyone hits this in future.
For context:
- The APFS file structure is documented here on the Eclectic Light company article " How robust are APFS clone and sparse files?"
- Mike Bombich’s advice is “If you want a reliable, fast, feature-rich, simple to use backup for a Mac, then I recommend taking networking 100% out of the picture.” and he points to several important caveats to backing up to a NAS.
Mac network backups don’t allow you to restore a whole Mac AND the poorly documented APFS file system format seems prone to corruption due perhaps to network interruptions or someone sleeping/stopping their Mac during a backup.
The best approach seems to be, for normal users:
- use CCC for network backups
- always ensure you have a local storage backup solution
The notes below are if you wish to play dangerously with your data (I strongly recommend local Mac backups if you are going to try this).
So how does my Debian miniserver with a Samba network sharepoint on a snapshotting btrfs filesystem stack up as a TimeMachine target?
- backup performance over fast WiFi is fine.
- backup completions can be monitored via the
com.apple.TimeMachine.SnapshotHistory.plistmade as part of the TimeMachine backup process and automated emails can be sent accordingly. - regular snapshots can be taken using
btrfs subvolume snapshotwhich are, interestingly, file extent aware and seem space efficient.
The reason for this post is to report that this setup just suffered its first APFS corruption issue. I was able to rollback to the previous btrfs snapshot and continue the backing up process. That worked as inteded (whew!).
I’m interested to see in future if I can move a btrfs snapshotted timemachine backup to a local Mac disk and restore from that. Since I’ve standardised on nvme drives for all my family’s storage, I can in theory sync an image from a btrfs volume to a local apfs-formatted drive using AFPS for Linux. I expect this won’t work, but it might be fun to try.
I assume you’re thinking about mounting the image and a physical APFS drive and using Linux to copy the contents of one to the other (maybe with a cp -R or rsync command or some equivalent)?
Yeah, that sounds really iffy to me, given the fact that the Linux APFS driver is reverse-engineered (since Apple hasn’t published sufficient documentation) and fairly new (so there are likely bugs that haven’t yet been discovered). And I don’t know if the standard Linux file utilities will be able to copy all of the metadata for an APFS volume (especially for things like version history, snapshots and copy-on-write clones).
But if you have an APFS disk image file (or sparse-bundle collection of files), you should be able to use Disk Utility on a Mac. Select the destination volume, then click “Restore” from the toolbar. Tell it to restore from the disk image. The destination volume will be wiped and replaced with the contents of the image. I’d give that a try (maybe with a thumb drive of sufficient size) to see if it works as expected.
For the sake of completeness, based on alert notifications in macOS 26, “macOSFuture” is confirmed to be macOS 27.
2 Likes