The utility of Disk Warrior in today's APFS world

TidBITS Talk
1

I may be wandering off-topic a bit, but saying you’re running Disk Warrior against your backup disk makes raises some warning flags.

It’s no secret in the Mac community that HFS+ is brittle, HFS+ Time Machine disks even more so. It’s this fragility that necessitates the existence of tools like Disk Warrior to “fix things”.

Having to run Disk Warrior against your backup disks introduces the possibility of your backup disk losing data. Which in turn makes me concerned about the reliability of the backups.

Which leads me to my belief that HFS+ is a problem waiting to happen, especially since it’s not obvious when a problem has been introduced in HFS+, and that it seems to go undetected. .

This situation is is contrast to the snapshots used by APFS formatted Time Machine disks. Those snapshots seem to be much more robust than the HFS+ hacks. And even with HDD media, APFS works well for Time Machine with acceptable performance.

My questions whenever anyone says “I need Disk Warrior to support APFS” is: “Why? What do you need Disk Warrior to do that Apple’s APFS fsck/Disk First Aid doesn’t already do?”. I haven’t heard a really good answer to date. Directory corruption doesn’t seem to be around much any more with APFS.

1 Like
2

I need Disk Warrior to support APFS so I can use it to repair APFS disks that have problems.

The strength of DW has always been that it does not try to fix problems. Instead, it creates a correct logical image of the disk, based on information DW obtains directly from the disk. Thus the problems DW solves need not involve directories. This approach simplifies recovery of function by bypassing two challenges of traditional disk structure repair: finding what is malfunctioning, and correcting the malfunction. For the user, the question of what caused the malfunction is generally of less interest than regaining use of the disk. The details of the cause of the problem should be of interest to the developers, including Apple, who created the software that had a problem. To this end DW lists differences between the correctly functioning logical structure it has just created, and the malfunctioning one which will be replaced. Hopefully this will allow developers and/or Apple to modify their software so future versions will be less likely to trash out the function of a disk.

Directory corruption doesn’t seem to be around much any more with APFS

Hmm… maybe.

Is another way to say this, “No errors in APFS have been found that cannot be found and corrected by Disk First Aid”? Full details of the function of APFS remain unknown outside of Apple. The point of DW for APFS would be to have independent verification of the logical representation of the disk. Please consider the possibility that the reason few or no errors in APFS-formatted disks have been reported that Disk First Aid cannot find, is Apple has made it very difficult or impossible to search for them.

The AlSoft (developer of Disk Warrior) web site says

The next major release of DiskWarrior (DiskWarrior 6.0) will include the ability to rebuild APFS disks. Apple released a majority of the APFS format documentation in June of 2020 . Our developers are now using this documentation to update DiskWarrior in order to safely rebuild Apple File System (APFS) disks.

Being able to rebuild APFS disks will mean more than confirming the APFS structure. Hopefully, DW will be able to build a logical representation of all the information on the disk. This probably includes security features, which now seem likely to be blocking disk function due to incorrect implementation. My guess is these security features are described in the unreleased parts of the APFS documentation. Withholding documentation of security features might be at least partially for security. Another reason might be that this part of APFS appears to be in rapid flux, as it is modified to enhance security. Many changes were listed for the macOS update 15.4.1 → 15.5. Two were in the firmware that controls the T2 security enclave chip in my MacPro7,1. Another was the build number of APFS. After upgrading to 15.5, my Promise R4i RAID and two external USB HFS+ disks became marked read only. It seems the files are there and in good condition, but APFS has mistakenly marked them as read-only.

I need Disk Warrior to support APFS so I can use it to repair APFS disks that have problems, such as were created for me by the recent macOS upgrade, and others can use it to fix problems cause by various other point upgrades in macOS 15.

3

They’ve been saying this for a long time now. It can be debated whether Apple’s level of documentation (which is generally recognized to be incomplete) or the technical difficulties of using that information to “safely rebuild” a file system are to blame for the delays.

I do understand that you have had some issues with upgrades. I also wonder how widespread those issues are. The answer to that question will gauge how viable Disk Warrior will continue to be as a product.

4

I appreciate your thoughtful questions. Replying has led me to think through the problem more thoroughly.

Before this discussion of DW was split off, the topic was “Sequoia issue with USB hub: “allow accessories to connect” missing”. Various posts confirmed that macOS Sequoia had problems, ongoing and recurring, with drives connected by USB. Anecdotally, such issues seem fairly widespread, but maybe not persistent. However prevalent, that is what the original thread was about. We can all hope that whenever malfunctions appear in macOS, Apple will immediately correct them, making third party repair applications irrelevant. Good luck with that. I’ll go you one better. My hope is that as Apple continues enhancing macOS the code will be tested sufficiently that essentially no new bugs are revealed on final release. A nice idea but not very likely.

Even if our wishes for a bug-free macOS did come true, there still might be a market for third party tools.

I don’t have the reference handy, but recently I read a review comparing HFS+ and APFS for various kinds of uses. Each had certain advantages and disadvantages, but if I recall correctly (and I read correctly in the first place), both accumulated problems that were not easy remediable, which detracted more and more from function. If I am remembering this correctly, a full recreation by DW of the logical representation of the disk might be useful after things bogged down. (I know this paragraph is way too sketchy, but maybe someone reading can supply the reference, or other relevant information.)

5

Continuing working on several disks I have more info on what works. When a USB disk is marked read-only, and thus mostly excluded from use by macOS, if the disk is formatted HFS+ a new logical representation can be created for it by Disk Warrior. The disk can then be used as normally.

Normal EXCEPT I just discovered I had been running macOS with macOS security off. I turned security back on and found all the disks where function had been restored, were once again marked read-only. So I turned security off again. The disks remained read-only. It seems a new feature of macOS security evaluates disks (probably more generally file systems) when they try to interact, and places a persistent mark on them to indicate the extent of function they will be allowed. DW has been rebuilding and rebuilding.

The workaround this leaves me with is to turn off all security and leave it off. Then let DW rebuild each disk that has been marked read-only.

Apple’s security enhancement has made it necessary for me to disable all security to be able to operate my computer. Ironic, but unfortunately not so unusual with Apple these days.

6

What security did you turn off?

7

When running in Repair mode a Security Settings menu is available. One set of choices are something like Secure, Medium Secure, and No Security. The other choice is between Allow booting from eternal disks, or only from the internal disk. Choosing No Security and Boot from External Disks, and subsequently rebuilding disks with Disk Warrior, lets my disks be used normally. This is the first way I have found to be able to use these disks since installing macOS 15.5.

I do not like the idea of turning off all security. I also do not like the fact of losing function of my computer for six weeks because of a System “upgrade”. This seems to be a no win situation.

Any advice or suggestions are welcome.

8

Honestly if I get issues with external disks I replace them with something new. I suppose another idea is to run Steve Gibson’s SpinRite, though that’s a Windows app, it can often work miracles, particularly on spinning disks.

1 Like
9

I’d like to see DW for APFS. DW has saved us on a number of occasions over the years so any additional repair option would be welcome - of course if you don’t want it, don’t use it.

I’m not a fan of APFS. I’ve had considerable issues with it - especially around shared volumes. It’s mystical space reporting is annoying and its slowness and lagging can be terrible (these may both be Finder issues but they don’t appear with my HFS+ volumes). Yes, HFS+ can become corrupt but DW has been invaluable in rescuing data and systems that Disk Repair simply couldn’t.

Given this, I’d be happy to see more options for data recovery of APFS - and hope not to need them.

10

Doug, thanks for the suggestion, but I think you have misunderstood the problem. As far as I can tell, none of the disks have a hardware problem, nor a problem with software function that I can detect. The problem appears to be a bug in macOS 15.5, introduced in the update from 15.4.1. A repair tool for Windows, even a miracle worker, will not be of use finding bugs in macOS.

11

Are we talking about the Promise Pegasus disk storage? Does this device require a third-party driver? If so, a chat with the manufacturer might be in order.

12

See posts 6 and 11 of the original thread.

The Promise Pegasus R41 RAID controller does require a driver, as well as a user interface provided by the Promise Utility. When I first installed macOS 15.5.5 my Mac became wildly unstable, including frequent kernel panics. After stability was restored by reinstalling macOS in Internet Recovery mode, I discovered the R4i had become marked read-only by some method that I could not alter. After extensively reporting my problems to Apple Feedback, my next step was to search the Promise web site for info, before contacting tech support. I discovered I was two versions behind on the RAID drive controller. One update had been released in 2022, then another in November 2024. I had also missed updates to the Promise Utility software that provides a user interface to the system. Much subsequent back-and-forth with Promise tech support, including escalation to level 2, plus perusing various support requests on the web site, did not solve the problem but did provide relevant data.

The technology of Extensions which enable third party software is a part of macOS that has been in flux in recent years, which means in recent versions of macOS. For a long time the extension which controls the R4i RAID has been part of the standard macOS distribution. This made it part of the immutable system software, and very secure. An update in 2022 changed this driver, and deleted the earlier driver version. This is the Promise R4i driver which is used by Intel Macs.

Keeping up with the times, Promise developed R4i RAID controller firmware, and Promise Pegasus software, with code optimized for Apple Silicon. The new driver was released in November 2024. This new R4i firmware is also a new type of extension, maybe introduced by Apple in macOS 15, I am not sure. This type of extension is the kind that can be (must be?) turned on or off in macOS Settings. Promise tech support says the new driver is optimized for Apple Silicon, but should be backwards compatible with Intel systems. Even though the new driver should work in Intel systems, its development effort focussed on Apple Silicon, so Promise recommends users with Intel systems use the older driver.

My discussions with Promise tech support extend over the time since macOS 15.5 was released. The current status, for the last couple of weeks or so, has been to wait for a response from Promise level 2 support.

On the Promise web site users of various systems report connection issues, many associated with macOS 15. It was in one of these discussions that rebuilding the directory with Disk Warrior was recommended.

So yes, as suggested, discussions with Promise tech support have been informative, even though the problem remains.

13

Ouch (and that’s an understatement). I hope that the Promise folks are in contact with Apple on this one. It seems as if macOS has introduced some kind of incompatiblity with the Promise drivers.

I’m trying to wrap my head around why a directory rebuild would make things work again.

14

Maybe I can save you from a headache. You are barking up the wrong tree. There is probably nothing wrong with the directory.

Disk Warrior’s virtue is not trying to figure out the problem. Instead it creates a correct logical representation of the disk and replaces the existing representation. I suspect one thing blocking creation of DW 6 is Apple has not been able to provide a clear description of what a correct logical representation looks like in APFS. This also suggests another answer to you asking why DW 6 is needed. The clear description of what APFS should look like, which is a prerequisite for DW 6, would also be of great benefit to Apple.

The bug is an Apple security feature introduced in 15.5 causes disk systems to be mounted as read-only. A DW rebuild allows the disk to mount normally, as read-write. But … wait, what? DW wrote its new structure to the disk Apple had marked as read-only. My guess is DW did not rely on standard Apple APIs. Instead it used either deprecated APIs (that have worked for many years) or, more likely, bare metal programming based on AlSoft’s detailed knowledge of hardware. Whatever the details, DW created a correct logical representation of the disk, which includes more than corrected directories, wrote it onto the disk, and then mounted it as read-write.

It is in these last two steps that DW fixes the problem. DW overrides Apple marking the disk as read-only, and writes to the disk. Then DW mounts the disk as read-write.

Which brings us back to the title of this redirected thread. I wonder where a disk utility that goes to bare metal and bypasses most Apple software fits with Apple’s security.

15

I doubt DiskWarrior is doing anything “bare metal” - it’s an application that’s either installed on a running macOS implementation or it’s being installed in a customized version of a bootable macOS Recovery-esque operating system. My guess is that it’s likely using OS-provided raw disk I/O read writes (something akin to POSIX open/read/wriite API calls) through device drivers available in the operating system. After all, in any UNIX-like operating system, a disk device can be opened almost like any other file and read/written to.

The key thing that DW has to do is to use the knowledge on how the file system metadata is laid out to directly read the disk sectors that make up the file system and build what it thinks is a “correct logical representation” out of them. The HFS+ internal data structures were relatively stable and known. Much less so for APFS.

I will agree that complete documentation of all on-disk structures that make up APFS containers and volumes/file systems is a pre-requisite for doing this reconstruction “correctly”. For whatever reason, what’s published to date by Apple doesn’t seem to be enough. Why this is happening is up for conjecture (you can argue that Apple wants to keep it proprietary. You could also argue that Apple considers APFS internals as a work-in-progress or moving target)… .

1 Like
16

Just to be clear, Spinrite is a disk repair tool that runs on windows but can fix disk sector issues with any disk file system, particularly spinning disk drives, as it does so at the sector level - it is file system agnostic. It can also be helpful with SSDs. It’s been around for a long, long time, going back to the PC XT IIRC. You can poke around at the web site I linked for a lot more info.

I think it’s possible to run on Intel Macs though I’ve never done it.

1 Like
17

I have progress to report. The problem with the Pegasus R4i remains. In a couple of days on the phone with Apple tech support I learned a useful new method for troubleshooting. Disk Warrior remains an important part of the workaround which lets me work while waiting for the bug(s) to be fixed. I have no more information on what DW does, nor how.

The new troubleshooting method is to create a new partition on the startup disk, install macOS on it, and run from the newly installed system. Somehow through the magic of APFS the new partition seems to take no space. The system newly installed on the new partition can be run with nothing carried over from the previously used system. This technique is comparable to the well-known method of creating a new user for testing issues, except rather than a new user space, it provides a completely new system. Testing in such a pristine macOS system the Pegasus R4i displayed the same read-only issue, suggesting the problem is not in macOS, but probably in the Promise Pegasus software. Promise/Pegasus system extensions which provide the firmware that controls the hardware RAID had been modified, but various combinations did not seem to solve the problem. What I recently found was three additions to frameworks: Pegasus API, Pegasus Kit and Pegasus Persistence, all Version 1.0, and all added to Frameworks on the day macOS 15.5 was released. I have not yet discussed this with Promise tech support.

I also discovered a reason my other HFS+ -formatted USB disks were read-only. Reading that others were having problems with USB disks, I hurriedly connected these disks that had been out of service a while. Finding what I was looking for, that they were mounted read-only, I did not consider them more closely. Now I realize they were working properly. They are read-only because they are Time Machine archive disks. Finder is not allowed to modify TM actives. The disk that has the archive currently used can be read and written to by TM as expected. Oops, my bad.

I received a “clarification” on the consequences of use of beta software. In addition to the many warnings that untested code can mess things up, I believe I was told that release versions can never be installed over any beta versions. If a beta version has been installed, a release version can be installed only after erasing the disk. Or more accurately, after erasing the partition.

I think this means the release macOS version, and a beta version, can be run on the same startup disk, each in its own partition. I know I was told that other versions of macOS can be run from separate partitions on one startup disk. The one constraint is that the earliest version of macOS that can be run is the version with which that Mac hardware was originally released. The Apple tech support person told me this allows him to run application software that has not been updated to run on the current macOS.

Summarizing, the problem leading to the pegasus R4i being mounted as read-only is likely due to faulty framework code added by Promise to macOS 15.5. Until this bug is fixed, the workaround is to run macOS with no security, and rebuild and replace the Pegasus directory with Disk Warrior.

18

With traditional disk partitioning, you create one or more partitions and each partition holds exactly one volume (HFS+ or FAT, typically).

In the APFS world, there’s another layer of indirection. The top-level partition table typically contains one device-spanning “APFS container”. Within that container, you can create many APFS volumes. All volumes in a single container share the same pool of free space. So creating new empty volumes consumes an insignificant amount of space.

If you would have create a second APFS container, that would require resizing (or maybe deleting and re-creating) the existing container. And volumes in one container would not share free space with volumes in another container.

You should be able to do this. Each will end up in a separate set of volumes (system, data, recovery, etc.) within a shared APFS container partition.

When you boot into one, the other’s volumes should appear as external volumes on your desktop.

Do not attempt to set up a common home directory shared by both systems. That runs a risk of one corrupting the other.

Any apps that you want to use on both systems should be installed separately on each, unless you are absolutely certain that it is safe to set up a shared installation - which I would only consider for stuff I developed myself.

If you want to share documents between the two, I recommend you create yet another APFS volume in the container and store documents there. Both systems should be able to mount it.

Yes. This has been the case since the dawn of Mac hardware. Apple does not support any OS running on hardware newer than that OS’s release. And OS installers will check for this and will refuse to install on too-old or too-new hardware. Sometimes it may be possible to hack around this, but it is rarely a good idea and is likely to fail anyway, because new hardware often has a different system architecture or may require device drivers that old versions of macOS don’t have.

You can sometimes sneak around this using VMs, but that’s a whole 'nother discussion and isn’t at all the same thing.