Auditing Free Drive Space: Where Have All the Gigabytes Gone?

It won’t be freed up over time. It will be freed when you start running low/out of free space and macOS needs to make room for something.

IMO, if you are running so low on space that macOS starts purging content, then you probably should have started deleting stuff quite a while ago.

There’s plenty of space, hundreds of Gb. Just like to keep track of what’s happening.

Having deprived us of progress bars, Apple now nominates “free space” as the last true source of randomness.

1 Like

Instead of “Absolutely Perplexing Fluctuating Space” I’d uncertainly suggest you call it “Approximately Probable Fluctuating Space”.:wink:

2 Likes

I didn’t actually assume the F really stood for Fluctuating, but that is me.

These issues are in play for me constantly, not necessary because of snapshots, though those too, but because I’m more than happy to use and abuse my favorite APFS capability, copy-on-write, which enables me to, say, think nothing about duplicating a VM or a sparse bundle or an iMovie library or a Photos library before making changes to it, since those copies take up no additional space. It’s so convenient! I can operate safely without doubling the space I’m using! But it also means that disk math is impossible, because of course it appears that I have hundreds more GB in use than I actually do.

The part that feels frustrating to me is that, in theory, a utility could analyze APFS data structures and theoretically figure this stuff out – I mean, after all, the computer has to know at some level what’s a lightweight duplicate or snapshot versus a block not otherwise represented. But I’m unaware of any tool, even seven years later, that works beneath the APFS file level, other than, of course, fsck_apfs. I don’t know if this is because Apple hasn’t released sufficient technical detail about APFS to make it possible, or whether it’s just because it’s so complex that no one wants to make it, but it’s a bummer. We could use it.

Also problematic is that if you make use of these APFS features, they’re part of the largely non-copyable container they’re in, meaning what fits on one computer may well not fit on another drive of the same size. This was a problem for me when cloning one computer that had about 1.9 TB used on a 2 TB drive – Migration Assistant just refused. I ended up using the Legacy Clone feature of Carbon Copy Cloner, which utilizes a somewhat unreliable Apple-provided underlying container copying mechanism, but it’s hardly something I’d want to depend on. I’d at least think Migration Assistant ought to have the capability of figuring out what is real space occupied when moving from machine to machine, but, no.

As for whether it’s possible to intentionally purge purgeable space – DaisyDisk does this, and in general has long been a tremendously excellent tool for managing disk space. It’s right up there with Carbon Copy Cloner as far as essential Mac tools go for me. I don’t think it has any way of knowing about what’s an APFS copy-on-write clone and what isn’t, but it at least helps me figure out how much space is hidden (meaning, snapshots and other stuff outside the immediate file system), and what the big fish are – or at least big fish candidates – when I’m looking to clear some space up.

3 Likes

A post was split to a new topic: Pining for the days of upgradable storage

Yeah, as soon as you get into any kind of behavior like this, the APFS Uncertainty Principle kicks into play big time. I was just revisiting an article Howard Oakley wrote about disk images, and how you can create a non-sparse image only to have it converted to a sparse image, but depending on what you do with it, it can revert to being non-sparse again.

I imagine that an APFS audit tool could be created, but it might be of relatively limited utility, given that the Fluctuating part of APFS would render whatever number it reports quite variable. It might also take a fair amount of time to run, as it evaluates every sparse image and duplicated block and snapshot to see how much data is really in use, such that it might not be accurate even by the time it finished.

2 Likes

Ah, I missed this one from Howard. Thanks for invoking it – fascinating reading.

I imagine that an APFS audit tool could be created, but it might be of relatively limited utility, given that the Fluctuating part of APFS would render whatever number it reports quite variable. It might also take a fair amount of time to run, as it evaluates every sparse image and duplicated block and snapshot to see how much data is really in use, such that it might not be accurate even by the time it finished.

YMMV, but I’d be more than happy to let something like this run all night to yield only ballpark accuracy! My question is whether Apple has even released enough information about APFS to make such an audit possible. Or to allow a program like Carbon Copy Cloner to theoretically recreate the internal APFS relationships on a target volume, so that that it ends up with roughly the same space used as the source volume, without having to unreliably block-copy the entire container. (And if not, couldn’t at least Migration Assistant do that? Or an expansion of “diskutil apfs” or “cp”?)

What about a tool to analyze your drive to utterly maximize disk space by identifying redundant true copies of things and turning them into lightweight APFS clones, or sparseify anything that can be sparseified? It would just be cool if the doors were open for devs to give power users tools to do powerful things, rather than one’s file storage being an opaque, unknowable, dynamic organism. Though, of course, Apple has been trending away from that for quite some time in most domains. (Sometimes you get something new and surprising though, like Shortcuts.)

I don’t know if I really am capable of going full zen on my disk space, as much as I’d like to; regardless, I don’t have much choice but to accept the mystery.

Yeah, this drives me nuts! After being a Mac Consultant for over 33 years, I’ve become increasingly critical toward Apple’s continual disregard for ease of use and understanding for their non tech users.

2 Likes

This got critical for me on a previous Mac that had a lot of media files on it, when I tried to install an OS update and was told “not enough free space.” That caused me to do a dive (not as deep as Adam or David C.) and learn about local Time Machine backups, and how to purge them. So this can be more than just an ‘academic question’

And to The Mac Doctor: I’ve always been very pissed off at Apple’s “an error has occurred” (and there’s not a damn thing you can do about it) attitude towards error logging, etc. I’m not sure which is worse, “An error has occurred” or “Error 4231” with NO WAY to find the meaning for that particular error code. Sometimes I have been able to spend some time (hours) digging through console logs, etc, to figure out what went wrong. As Mac OS gets increasingly more complex (a lot of that due to paranoid security), things are much more likely to break in strange (and often not repeatable/Heisenbug) ways.

Thanks for this coverage! It’s enough to validate that I’m not crazy or an idiot since I can’t figure out what appears to be simple math… calculating free drive space.

Besides COW, another way that storage is oversubscribed is Time Machine’s use of hard links, if I recall correctly. Hard links are additional directory entries in a filesystem pointing to the same inode. The data will only free up when its link count drops to zero. Until then, some ways of counting consumed space may double report such storage.

But I’m probably off topic now :sweat_smile:

I think that the use of APFS snapshots in Time Machine is instead of hard links. Can’t remember where I read that, but I’m pretty sure hard links were the best that could be done on HFS+, but snapshots are a more robust and elegant solution for Time Machine incremental backups.

Aha thanks for that! Yea I learned what I learned from a white paper when it first came out I think.

More off topic, but I’m not a fan of APFS. It seems to be nothing but trouble. Drives I cannot boot from or backup, all kinds of ghost images floating around in utilities that display drives, corrupt file systems that cannot be repaired…

We explored ZFS ages ago and abandoned it, right? I wonder why that was… ZFS has emerged as the standard elsewhere, such as for Proxmox virtualization.

1 Like

Correct. Time Machine on HFS+ (often called “TMH”) is based on massive amounts of hard-links in order to avoid duplicating content that is unmodified from one backup to the next.

Time Machine on APFS (aka “TMA”) is based on snapshots. So the underlying file system handles the deduplication of content. In general, it is faster and more robust than TMH, but has the disadvantage that there is no way to purge a single file from a set of backups (which you could do with TMH by deleting a file from every backup).

3 Likes

I recently reinstalled Ventura and all my apps and data on a new Mac mini. While I am confident that there is around 300Gb of free space left I cannot get consistent figures. About This Mac, Finder, Disk Utility, and System Settings all give different results varying by as much as 250Gb. Further the breakdown by app type within System Settings produces nonsensical numbers (e.g., it tells me I have 13Gb in TV but the folder is empty).
Another example. Finder tells me Xcode is 22Gb but it also tells me that the package contents amount to 8Gb while System Settings > General > Storage > Applications says it’s 12Gb.
This is ludicrous.

I did some similar experiments on my Ventura system, and offer the following observations:

  • “Purgeable space” is to be considered space in use - and nobody but Apple can justify why it’s reported in the way it is. The Purgeable figure includes Time Machine local snapshots, but also seems to include other caches that macOS can purge if needed. If you subtract the Purgeable figure from the Available space in the “get info” area of your hard drive, that figure should match what you see as free in Disk Utility. In my case I have Available space as 154.89 GB and Purgeable as 6.49 GB. Subtracting the two gives me a figure of 148.2 GB which matches what Disk Utility is telling me is Free.
  • The “Used” value of 88.9 GB in Get Info for Macintosh HD matches what Disk Utility tells me is used for the Macintosh HD volume.
  • The hidden APFS volumes which include Preboot, VM (virtual memory backing store) total about 5 GB and 2.1 GB repectively on my Mac. This space comes out of Macintosh HD but like snapshots, isn’t reflected in the Used space of Macintosh HD. It’s reflected in Disk Utility when viewing the Macintosh HD volume as “Other Volumes”.
  • The use of sparse files and “cloud offloaded, on-demand” file services means you can’t take the Size figure at face value to determine how much disk space a file/application is using. The “Size” in a Finder window is a representation of the maximum file size (i.e. “the file address of the last byte written”). If you look in the “Get Info”, you see a “on disk” figure, which may be lower if the file is actually sparse or offloaded to cloud. (The Finder should really have a “Size on Disk” display in addition to “Size”.
  • The Storage panel in System Settings does appear to have inconsistencies. It for some reason over-estimates file sizes. However, if you drill down into Applications, I found that the size reported for Xcode is the “Size on Disk” figure reported in the Get Info box for Xcode, and not the Size of the app as reported in the Finder window for the Applications folder.
  • The bar in the bottom of a Finder window that shows available space is showing the same value as Available in the Macintosh HD Get Info box. Which includes purgeable space.

If I wanted to figure out how much space I had available on my disk, I would take the Available figure in the Get Info box and subtract out the Purgeable space. I shouldn’t have to do that - the Finder needs to do a better job of making this known.

In dumbing down the Finder experience, Apple has masked over valuable information that they shouldn’t have, IMO.

If I subtract “purgeable” from “available” space as shown in Get Info for Macintosh HD I also get the Free space shown in Disk Utility.
I suspect that Ventura classifies my audiobooks in the Books app as purgeable as it is possible to shift this to Apple Cloud. However, I have over 250Gb of audiobooks and have no intention of paying Apple for that much storage when I have already paid them for the SSD on my Mac mini.
Curiously the free space shown by Finder flips during the day between the high “Free” number shown in Get Info and a number closer to that shown in Disk Utility. However, it returns to the high number in the morning.
By the way, I asked a Genius Bar advisor about this earlier this week and they said they do not use Finder but use DaisyDisk instead. Another vote of confidence.

1 Like

Media you purchased from Apple’s on-line stores (Music, Movies, Books, etc.) is purgeable because (in theory) you can always re-download it. And this content doesn’t count toward your iCloud storage, because any re-downloading comes from the store’s servers.

Your own content (not purchased from Apple), on the other hand should not be purgeable unless you are already storing it in iCloud somewhere, since you would need that storage to re-download it.

I wonder if that’s official or just what that particular tech (or his store manager) uses.

FWIW, if I want to track down what’s consuming all my space, my go-to app is Disk Inventory X, based on the open source kDirStat for Linux and its Windows port WinDirStat.

1 Like

The audiobooks are from my own sources not Apple so should not be classified as purgeable. However, there’s no other combination of folders that adds up anywhere close to the purgeable number.

DaisyDisk was a personal choice of the advisor.

I’m prepared to leave this entire topic as an Apple mystery. Maybe it will be cleaned up in a future update or macOS 14 and maybe pigs will fly.

Just cleaning up some old mail and found this additional article from Howard Oakley, which basically suggests that the Finder is wildly inaccurate when it comes to free space calculations.