I would rather not have randomness defeat my players at unimportant moments of play but rather have that stuff not even interrupt flow. I want my dice to convey tension every time I roll them. Be it potential to do something great or something that screws the party.
Over time I want lackeys and goons to not matter to the players. I want my bosses and sub bosses to always be a threat. Here you get into something lame where only a few of the enemies really threaten the party,this is not what I want either.
I would like lackeys defending a castle to be easily dealt with but If the game really deals with those people in the castle then I want to have all the forces in the castle be meaningful. I am not sure if this would be level based or defense vs combat attack successes.