Explanation for some Big Sur upgrade problems

I found a work-around for the loss of access to my POP email after “upgrading” to Big Sur on the Apple discussion boards: How can I get POP email from my personal … - Apple Community? The trick is to set up a new account and enter a bogus email address so it throws up a message that offers you the option of POP or IMAP protocols. After some fiddling around and looking for information from my hosting firm, I was able to get a connection and download the POP mail (which I keep around for a week before deleting). Older POP mail not on the server may be lost, so you may need to restore it using a Time Machine backup.

I am rather annoyed about the silly games Apple is playing trying to “encourage” users to transfer to the IMAP protocol by making it hard to continue with existing pop accounts. I also am annoyed by Joe Kissell’s failure to warn that that the Big Sur upgrade can hose POP accounts in his Take Control of Big Sur book. Email is a vital service for most of us, and otherwise he has been good about warning of other upgrade problems. It took time to find the work-around, and more time to get it to work properly. Worse, it looks like at least some archived POP mail may have been lost. I have copies on my desktop so I won’t bother to try a Time Machine recovery, but be careful with a Big Sur upgrade.

This did prompt me to check with my hosting company about converting from POP to IMAP, but they warned it would be difficult and that my accumulation of 30+Gig of email archives is too much for their highest-level service plan. The laptop is for travel, and I don’t use a smartphone, so I don’t need IMAP to make all of my email accessible from multiple devices, and I don’t like Apple trying to limit my options.

When you say not on the server, do you mean mail On My Mac, or somewhere else?

I have Apple Mail set to delete all POP mail from the hosting company’s server one week after it is read, which gives me time to go back and get mail if anything goes wrong. All that I see in the “Inbox” for that account and a copy of that inbox in “On My Mac” are that one week of mail that was still sitting on my hosting company’s server, plus a single email from 2018. I have not used Mail on the laptop Inbox for some time, and don’t remember if I cleaned out the old POP mail before the upgrade. A few odd messages showed up in numbered folders in On My Mac, so the upgrade process did seem to leave some odd bits and pieces laying around.

It seems to me that one workaround would be to switch to IMAP but then use Apple Mail rules to direct all incoming emails to appropriate mailboxes within the On My Mac account.

1 Like

Please forgive me if I am not understanding, but I think the copy of your Inbox may actually not be in On My Mac, i.e., not locally stored. You can verify this by clicking the down-pointing caret that hides On My Mac and see if the Inbox copy is still displayed.

I use POP as well for most of my accounts and have used the “trick” you mentioned which you also have to do if you want POP on an iPhone/iPad. If you have a backup of your mail library, you should be able to copy it to your Users/Library/Mail folder (which will replace that one) and then when you run Mail, it will import all of that data. You will lose whatever you have now so in case it doesn’t work as expected, copy your current Mail folder for a backup.

As for the POP “trick,” this isn’t new as I recall doing it before. IMAP seems to be more prevalent in general.

You’re right that the copy of the InBox is not really in On My Mac – actually, there are copies of each of my accounts below On My Mac as well as in the InBox itself. The display is somewhat different than in the Mojave Apple Mail, which only showed 1 to 3 folders for my Google accounts and a little-used Verizon account and nothing for my POP or (little-used) iCloud account. In BigSur five accounts (but not Verizon) are shown with at least Inbox, Drafts, Sent and Trash, and each of the Google accounts have at least seven internal folders/segments. That’s an odd difference.

I’m doing the work-around to avoid switching to IMAP, which would cause multiple problems: my ISP does not offer enough storage to hold all my email archives, and the conversion process is rather complex, so it would have required much more work on my part. I also prefer to keep incoming in the InBox and shift them to On My Mac early the next year in an annual file.

Someone more knowledgeable may chime in, but as far as I can tell (I have them too), those are IMAP folders, which mirror the folders on the server of your email service provider.

The drawback to that strategy is that the Inbox files are not being stored locally, so they are not being backed up by cloning or Time Machine, and if your HDD or SSD fails, you will lose any messages in them.

Something doesn’t add up here. You said that you have one of your mail clients delete pop mail on the server after a week, but your host says that you have 30GB in your pop account. I suspect that you aren’t really getting 30GB of mail in a week, but that the host’s server isn’t truly deleting your old mail when one of your mail clients tells it to. This is not an uncommon thing with pop.

I am rather annoyed about the silly games Apple is playing trying to
“encourage” users to transfer to the IMAP protocol by making it hard to
continue with existing pop accounts.

Apple makes it hard to set up any account that isn’t on their primary list of commercial services, including imap. I’ve been using the ‘lie to Mail until it throws up it’s hands in despair and finally lets you see all of the settings’ trick for many many years, at least since lion and probably earlier. It’s even worse now that you can’t turn off the “automatically manage accounts” settings (aka “automatically destroy the account settings periodically”).

I understand that pop has some advantages for some users, but behind the scenes it’s a pita. It’s is inherently far more fragile than imap especially if too much mail accumulates. All of that mail lives in a single text file… When we were still running our own department mail server, the only thing that took more time than keeping up with spam rules was keeping pop running, kind of, mostly, even after most users had been switched to imap. pop has been dying over the decades to the point where it’s current primary useful use is to let a mail service such as fastmail fetch mail from other mail services such as gmail which still support pop since imap doesn’t have a standard way to do that. My personal mail server does have pop turned on, but only for the convenience of getting the mail from one domain into my fastmail account where it gets sorted and turned into imap. Even with that light use, I occasionally have to go onto my server and kick pop back into submission.

I understand your frustration, but this falls squarely into the overall advice to stay current with technology. The longer you wait to upgrade to what’s current and the more you swim against the tide of general usage, the more difficult and error-prone things become.

POP is an ancient protocol that doesn’t fit with how the vast majority of people use email these days, so it’s not surprising that Apple wouldn’t be testing hard against it. Nor should Joe be castigated for failing to notice a bug related to a protocol that he undoubtedly doesn’t use for anything beyond testing, if that. I’m not positive about this yet, but it appears that the Mail formatting problems some people have been having with TidBITS are related to using POP accounts as well.

And if your email service provider can’t provide sufficient space for a 30+ GB email archive, there’s nothing wrong with storing the bulk of it locally, either via On My Mac mailboxes or an independent email archiving program. Frankly, I wouldn’t trust Mail with that amount of mail anyway since it’s not designed for long-term archiving.

3 Likes

I may not have been clear enough - the 33GB of email I have is the count Apple gives of email on my internal SSD. It’s an archive that goes back about 20 years, most of it stored in “OnMyMac”. I do get about 60 MB a week, but much of that is deleted immediately after reading.

I don’t understand what you mean by “all of that [POP] mail lives in a single text file.” Eudora did that, and it could lead to serious problems. As far as I can tell from a little poking around in my InBox, Time Machine treats POP and IMAP mail in the same way, grouping them all together in a single file for each backup, so you can’t recover a single email but you can recover a whole Mailbox. Is there some difference that I’m missing?

Fighting bugs in Quicken Deluxe as well as in Big Sur has left me rather grumpy, so I may have come across rather harsh. I’m much more annoyed with Apple’s handling of the Big Sur, which still has bugs that can freeze installation because it doesn’t check properly for available space and overlooks the multiplication of hidden files. I had come to rely on Joe Kissell’s books, so I was disappointed that he didn’t spot the POP problem, but I do realize he couldn’t catch everything. (Apple certainly doesn’t.)

I set up my current hosting arrangement several years back when my previous host shut down, and at the time they did not offer IMAP – or offer enough storage to cope with a large archive – so I set up POP in the current configuration. I’m not one to fix something that isn’t broken, so I stayed with it since then. From what you’re saying, POP is more broken than I had realized, and that’s going onto the list of things to get done when I can have enough time to make changes properly. (I’ve learned not to make major changes in computing equipment when I have pressing deadlines unless it’s an emergency.)

I still use POP for some of my old ATT accounts with no issues so even if the technology is ancient compared to IMAP, it still works if your provider supports it. Since I’ve usually upgraded to new a OS rather than doing a clean install, all of the information has come over with no problem. But I always test a new OS on a separate partition or disk to make sure everything is working before using it on a regular basis. I still keep my old OS volumes just in case for older software etc.

I don’t know if you tried what I said earlier, but I’m wondering if you upgraded from an old OS or did a clean install and then migrated the information from your email database. I’ve had no issues when upgrading as the new mail program will say it’s converting the mail database and it’s all there. So if you did a clean install and migration with poor results, then I would copy the old mail database manually from your backup and let Mail convert/import it in Big Sur.

Recently, I set up Monterey in virtualization since my Mac can’t upgrade past Catalina. When I checked, all of my mail folders were there as well as what I had in my inbox at the time so if if works in Monterey, it should work in Big Sur I would think.

1 Like

I have a 2012 MacBook Pro that can’t upgrade past Catalina. I am curious as to how you managed to “ set up Monterey in virtualization”.

I was/am in the same situation with a 2012 Mac Mini running Catalina. You can get a VMware Fusion license for free for personal use and use that to install Big Sur or Monterey. You will need to setup an account which is the gateway to receive the license and download.

https://customerconnect.vmware.com/web/vmware/evalcenter?p=fusion-player-personal

You can install the MacOS as a clean install by getting the system installer of your choice using this page for example:

If you want to use your existing Catalina volume to keep everything intact, then you need to make a disk image of that partition and VMware Fusion can use that to install what you currently have now, and then you update from there.

It works OK but is not going to be as fast as a regular system volume but it does work better running from an SSD so if you have that, I would use it but you can try it first on a regular drive. It’s not hard to do but there is a bit of a learning curve if you never setup a VM. Also, if you have a Windows installation you can use Fusion for that too.

2 Likes

Amazing – it never occurred to me that I could run Big Sur or Monterey from a virtual partition using this 2012 MacBook Pro when Apple won’t let me upgrade it past Catalina. Thank you! I will give it a shot – a great learning experiment. I have a 2 TB internal SSD, so I suppose that I could create a partition for Fusion on that drive which would presumably be faster than an external SSD.

I would think that the internal SSD is faster than an external SSD 3.0 drive like I’m using but it’s way better than a spinning drive. My 2012 Mini was the server model which had two 1TB internal drives. A few months ago, I was experimenting with the VM and I had my Mojave partition on it to test some software before using with my main partition. When I started getting the OS upgrade messages in System Preferences, I figured I would see if it worked and it did. I then started googling to see if others had done it and found a few references to similar installations so it was interesting to see if it would work up to Monterey even though the VM software is not certified that far.

Now, I can’t say how well certain apps will work as I’m still experimenting with it but I figure if I need a newer system security-wise, then the Monterey VM will do the job until I get a newer computer.