As far as the application is concerned, these are two completely different mechanisms. And if you try to open the file via the app’s “File → Open” menu, that’s a third.
Each of these is going to deliver a different sequence of messages/events to the app. If the app doesn’t handle one or more of the events properly, that could easily cause the problem. Or the app might only handle some of them, relying on OS-default behavior for the others, and either one could have a bug.
Since you are seeing the problem with TextEdit and the bad behavior seems (by your description) to be more common than the good behavior, I’m inclined to assume that there’s a bug in the OS-default behavior. But without having access to the app’s code and a way to debug it, there’s really no way to know for sure.
Hence my comments about sending bug reports, along with as much information as you can. Since you now know that filenames with composed diacritics triggers the bug, that may be enough information for Apple (or an app developer) to reproduce/debug/fix/test the problem.