Definitely use the macOS built-in firewall. Allow what you’re using and block everything else.
Port scanning is a real thing. Criminals routinely scan address blocks looking for vulnerabilities.
That having been said, you’re right that the risk is low if you’re not running an Internet-facing server, primarily because of the way consumer routers work.
If you’re a residential customer, you are only going to get one IPv4 address from your ISP (and it might even be one private to the ISP). All the computers on your LAN are able to share that address because consumer routers implement Network Address Translation (NAT).
With NAT, your router rewrites the source address of all your outbound traffic so it all comes from your single external address. And it tracks outbound connections so when the reply packets arrive, they are forwarded to the computer that originated the connection.
As a side effect of this, unsolicited packets arriving from the Internet are discarded. Not because of any explicit security protocol but because your router has no way of knowing where they should be delivered.
If you run an Internet-reachable server on your LAN, then you must configure port forwarding in order to tell your router where to send unsolicited packets intended for the server. But this can open your LAN to attack, because something that compromises the server can jump from there to everything else.
Which is why I strongly recommend against anybody configuring a DMZ node. This is a router configuration that sends all unsolicited traffic to a specific node on your LAN. Which means that any vulnerability on that computer can be used as a way to attack the rest of the LAN.
Now, if you run IPv6, you generally don’t have NAT. Your ISP typically gives you a large (typically 48- or 64-bit) address block and your computers use SLAAC to auto-generate local addresses within the block.
In theory, anyone on the Internet can port-scan your IPv6 address, find vulnerabilities and attack via what is discovered. For this reason, it’s a good idea to run a basic firewall on every device (most modern operating systems have one built-in).
The main reason why this isn’t a complete disaster is that the address space is so huge. A 48-bit address space is about 280 trillion addresses. The odds of an attacker randomly finding the dozen or so your LAN might be using is really low. And the SLAAC protocol typically causes your addresses to change frequently.
Of course, someone snooping your outbound IPv6 packets could learn your address, but you would have to be connecting to the attacker’s server in order for that kind of attack to work. But it’s not impossible, since an attacker may have compromised a major server or router on the Internet. I’d like to think that such compromises would be quickly detected and blocked, but I don’t believe in depending solely on wishful thinking. And even if the compromised server is isolated, it can still cause quite a bit of damage before that happens. (Think for instance how disastrous it would be if a sever run by Akamai would be compromised.)