Yeah, I have both me.com and icloud.com (I got the former by signing up for Find iPhone when it became free during the Mobile Me days–remember that?). The situation is all a bit messed up, but on iOS you can go to Settings > (your name) > iCloud > Mail (at bottom) and there you see every possible address you can receive mail at; set the toggles to preference. Go to icloud.com and use the Mail app to edit the non-primary local-part aliases themselves. The toggles you set also control the addresses Apple thinks you are reachable at, which do appear to impact iMessage and FaceTime (you can see those at appleid.apple.com) and the permissible addresses that appear in Mail.app on Mac. No, I really don’t understand it either.
But iCloud Mail is a Hotel California, anyway. Once enabled, your iCloud address becomes an Apple ID in addition to any non-iCloud address you began with; if you should ever make the supreme mistake of changing your Apple ID to match your iCloud address, then you’re properly locked in to using it exclusively, forever after. Because you can’t disable iCloud Mail once enabled, and because Apple sends all important email to your Apple ID, the only way out is to forward your mail to another account (which may or may not actually be desirable). As far as I can see, deleting all your aliases and then changing the local-part of your Apple address (which, fortunately, is still apparently possible) to something completely unique and otherwise unknown is probably the best you can do. It’s also probably the correct solution, since iCloud Mail counts towards your storage space and you would wish to see it cleared out and stop receiving further mail if you wanted to get your storage back. All of which is by way of saying that I too intend to go back to my own domain and self-hosted email, and this is what I’ve learned in the process.