Browsers, Cache, and Streaming Glitches

A question I’ve wanted clarification on for a long time.

I find that streaming content can be hampered by browsers having hefty cache data stored.
I generally find that Safari suffers most from this.
Question: is this indeed an issue with browsers, and how they store, purge, or handle stored data … or issues with streaming sites themselves?
I think some sites so have problematic aspects to this as well, but mostly related to buffering …