You’re not completely correct there…and I spent the last 20 years of my working life in the computer security biz. You’re not far wrong though in some respects.
Yes…a lot of leaked passwords, standard phrases, speeches, quotations, common substitutions, and what have you are already in the hackers dictionary attack and rainbow attack tables…but at some point based solely on length of password those tables are too big to be feasible and the only method that will work at that point is brute force try every password until it works. That’s just the way things are.
Back when I retired in 2011…the generally accepted in DoD length needed to force a brute force attack was 17 or 18 characters long…it’s surely longer today with faster GPUs and such…I don’t know what it would be today but probably 24 or 25 isn’t a bad guess…especially for a random Joe User account that isn’t valuable or has a lot of money or is a celebrity or NSA spy or whatever.
When you force the bad guy into a brute force attack…he has to input an entire guess…the password isn’t broken a character at a time like you see on TV…and either the entire guess is right…or the entire guess is wrong.
So…one pretty foolproof way is to choose 3 or 4 common words…and it doesn’t matter that each word is in the dictionary…unless the entire sequence of words with any uppercase, symbols, and numbers is in the dictionary/rainbow table as a single entry…it fails.
So…Eagle$$Apple$$Heron1234 is 23 characters long and a quick look at Steve Gibson’s Haystack page says that a brute force attack against that password will take 9.88 billion trillion centuries at one hundred trillion guesses per second. That’s plenty long enough.
You can argue that a completely randomly generated 23 character password has more entropy and will be longer to crack…and you might be correct and you might not be correct. Even through Apple, Eagle, and Heron are in the dictionary attack…they’re useless as a guess since they’re not the password.
And in any event…even if the completely random one is ever so slightly ‘better’…who cares. The ease with which that one can be remembered if it’s your master password and typed without error in the field of bullet characters makes the perhaps ever so slightly less good argument irrelevant.
You say users should not create passwords. I say that users should not create bad passwords and that choosing 3 random 5 letter words that aren’t associated with you (no dogs names, kids names, etc) along with a symbol that you never tell anybody and then 4 digits (again nothing that directly associates with you)…is creating a good password. It might not be the best possible password of a given length depending on how you define the term best…but once you go over a few centuries to crack it…well, it doesn’t really matter at that point.
I’ve been using the several words scheme…and no, it’s not the exact number of words or pattern above, that’s just an example…with numbers and symbols…and to my knowledge I’ve never used the same word twice since I always choose 3 randomly selected words that don’t go together…for instance Hawk, Duck, and Egret would never make the cut. An even if I did manage to pick one of those random words twice the other two that go along with it aren’t the same…so it’s not the same guess and Apple being in the dictionary doesn’t matter a whit. Diceware is based on this very same scheme…although to my mind it doesn’t go far enough since it’s only got a limited number of words…but again, it’[s probably got plenty of words given that you’ve forced the cracker into a brute force scenario.
What you’re talking about with password complexity is entropy…and like in many, many other things…better is the enemy of good enough.