I’ve made headway on my Monterey challenge with file sharing, and I think it’s now fixed.
The problem was a DNS issue, combined with seems to be a bug in Monterey’s file sharing code.
My gateway router is an ASUS RT-AC86, running the well-known “Merlin” variant of the stock ASUS firmware. It uses dnsmasq for its DHCP server, as well as for local DNS. (I’ve used dnsmasq for at least a decade, as part of Tomato firmware builds). I configure the dnsmasq DHCP server with a number of reserved IP addresses for servers, printers, and most other known devices on my LAN. In addition, I’ve generally configured dnsmasq with “local” specified as the local domain. That simplifies making connections to various LAN devices, and saves me having to remember and type IP addresses when I want to connect to local printers, file servers, routers, IP cameras, etc.
The domain .local is also used by mDNS for discovery and address resolution of devices that support mDNS. Most Apple devices do, and we have a lot of Apple devices, all advertising themselves with mDNS (Bonjour). The .local addresses used by dnsmasq and the .local addresses used by mDNS have happily coexisted in the past. But there’s something in the Monterey implementation that seems unhappy with the use of .local by dnsmasq.
I changed the configuration of the ASUS router to use “home” as the local domain instead of “local”, and file sharing happiness on my M1-MacBookAir was restored. I certainly don’t understand why. The file server with which the MBA has been struggling to access is a Mac mini running Big Sur. It uses mDNS to advertise itself as an SMB server at miniserver.local. Using mDNS, the MBAir can obtain the mini’s LAN address (both IPv4 and IPv6 addresses). Since it speaks mDNS, there’s no reason for it to issue a unicast DNS query to my router. If it did, dnsmasq would return the same IP that an mDNS query obtains direct from the mini. For that matter, the MBA could obtain the IPv6 address advertised by the mini, and use IPv6 to make the connection.
I mentioned above that I suspect Monterey being partly culpable. A MacBook Air (with Mojave) on the same LAN has not encountered the file sharing issues that I see with Monterey. For clarity, the issues occur when initiating file sharing by clicking on a device icon in a finder sidebar, not when using cmd-K.
In any event, having changed the local domain specified in dnsmasq from “local” to “home”, file sharing across my network seems to be working fine. I have to remember to enter “router.home” or “printer.home” etc to connect to LAN devices, but I can probably deal with that.
For completeness, while researching this problem, I learned that the official TLD now sanctioned by the IETF for home networks is “home.arpa”, not “home”. Details are found in the relevant RFC (RFC8375).
I’m indebted to @trilo for suggesting that DNS might be implicated.
And to @Shamino for the primer on mDNS.
And of course to Adam for providing this community forum…