Fit is king. Clothes that fit poorly never look good, and simple clothes that fit well often do. We know that.
We also know that silhouette is one of the most important considerations when building an outfit, and sometimes, pants that are just a little too slim or shaped not-quite-right can ruin the intent behind an outfit.
But does an otherwise well-crafted, complex outfit instantly fail because the hem is a little too long, or because the sleeves are a little too short? If the outfit is meaningful, is that meaning destroyed when fit is a tiny bit off?
I don't think so. This might be controversial, but I often think that getting the hem length exactly right is not that important. It doesn't hurt, of course, but it's not the point of this thing we do, and focusing too much on precise fit can distract from building a good outfit overall.
What is important is the overall effect and meaning of an outfit. Details matter, but they only matter in service of the overall effect and meaning.
Sometimes, the effect and meaning depend on hem length. There is a difference between a slight break, artful pooling, and showing off two inches of ankle. There is a difference between a shivering break and no break, even if it's subtle, even if it's more or less lost in motion. There are reasons why you might want to have your pants create a puddle around your shoes.
But sometimes, a piece fits perfectly while you try it on, but rides up a little when you sit down or bend your arms. People move in their clothing. People are dynamic, and constant perfection isn't realistic.
And sometimes, it doesn't matter. Sometimes, the point of the outfit is the way the upper half is layered, and getting caught up in how the pants break on the shoes is missing the forest for the trees. And sometimes, intentionally getting the fit a little bit wrong is the right thing. The canon of sprezzatura is full of sleeves a half inch too long, worn open, dangling past the jacket and past the knuckle.
I find it's extremely practical to keep this in mind when building unusual outfits. If your pants are hemmed to wear with loafers, they might hit your boots a little differently, and they'll probably hit your sneakers in a completely different way from that. And that's okay, as long as the complete outfit can portray the meaning you intended for it to portray.
I'll also say: details may matter more in a simple outfit; the execution is all you have going for you. In a complex outfit, the intent and the expression can come out more clearly, and the details only matter in service of the expression. Sometimes, bad fit feels sloppy. Sometimes, getting the shoulders too big by accident is like painting a happy tree. And often, if you love something that doesn't fit right, you should just get it tailored.
But some alterations are not worth bothering with. Sometimes you find a perfect jacket at the thrift, except the shoulders are a little bigger than you like. Sometimes, the right move is just to suck it up and play with the jacket you found. Maybe you'll learn to layer it better, maybe you'll just learn to wear it in a slouchier way.
The same holds true for casual pieces. Maybe you want a band tee for a band you love, but they just run long for you. Maybe you are in love with a one-of-one shirt that just can't be had in the "perfect size." Maybe your grandpa's cardigan makes you feel warm and fuzzy inside, despite being a little shorter than you're used to.
I've ruined shirts I liked by altering sleeve pitch, or shortening the sleeves so far that I couldn't roll them up anymore. Was it worth it? Of course not. If a shirt is unique and special and can't be tailored to fit perfectly, just let it be unique and special.
Details matter. Fit matters. Getting it right is better than getting it wrong. And by all means, try to match the ideal you have in your head about how clothes should fit. But reality is messy. Embrace it, and build an outfit meaningful enough that minor imperfections don't detract from it at all.