I have experienced the same symptoms. I was using VLC to play video located on a drive on a Mac on my LAN, playing and being viewed on a different Mac on the LAN. The video was accessed via Home Sharing. I think iCloud/Home Sharing controls file access between the two Macs, with the video transferred only over the LAN (not to iCloud). Delays and stuttering occur, then go away, at random times. Home Sharing frequently turns itself off, sometimes even while playing a video. I have an Apple TV 4K on the LAN, but it is not used in this video viewing.
The symptoms FishTickler and I see seem the same, but our systems are very different. What our systems have in common is intermittent breaks in video transfer via a LAN.
About a year ago I moved to a larger, recently renovated, three story house. During renovation CAT6 ethernet had been installed. I had never done much with networking, including ethernet, but figured now was time to learn. I bought a Ubiquiti Dream 7 console and began. I am still working at learning.
During this time two things have become clear. First, the state of the art is a mess. Multiple vendors, multiple protocols and conventions, multiple security features in many versions, trying to connect equipment of various ages, … Second, the top priority while muddling through the mess, should be system security. The LAN is connected to the WAN (Wide Area Network, aka The Internet), where The Bad Guys lurk, and The Bad Guys are more and more skilled and aggressive.
This is the background within which playing video on an Apple TV, or a Mac, works, to the extent that it does. I suspect the problems we are seeing are due to network glitches in our LANs. As I understand it, Apple’s tvOS is derived from iOS, and iOS originated by stripping as much as possible from macOS (OS X), to leave a relatively simple, low energy requiring, and secure single user System.
In the world of video viewing on a LAN, problems such as lags and stuttering are solved by creating a dedicated Virtual LAN (VLAN), routing video and only video over it, and prioritizing its packets at switches. Other VLANs are created, and other network traffic routed to them. The ability to manage such complexities is the sort of thing that was stripped out to create iOS, so it is still missing in tvOS. In contrast, the UniFi Network application works well for VLAN creation and management. In my experience, macOS (or iOS) System Settings / Network then has problems with the LAN with multiple VLANs created by UniFi Network.
I suspect the random dropouts and pauses we have seen in video originate in timing issues with causes somewhere in this hodge-podge of LAN interactions. More RAM or faster processing somewhere might suffice as a work around, but I suspect the essence of the issue is network anomalies.