Given that you’re working with multiple betas, it’s hard to make any useful suggestions—the fact that there are betas involved trumps all other likely solutions.
I don’t quite understand why you’re saying an empty library would upload anything—it should just be downloading if there’s nothing in it. I did this a while back and it worked fine to get my library from iCloud down to the Mac, and new photos added to the Mac synced up fine after that.
Photos does often claim to be “uploading” photos when it’s just checking sync state. It’s faster than initially (when it could take weeks), but it can still take a day or two even when no data has to be moved.
The best I can suggest at this point would be to make sure you have a backup and clear both the iPhone and the Mac of all photos, then sync again from iCloud to both.