Balance in software design

Firstly... please bare with me, I'm an engineer at heart, and sometimes have difficulty composing my ideas and opinions. I'll get better as I go I promise =)

One thing I've come to realize in at least the engineering positions I've held so far, is that it seems like most employers suffered from one of the following two conditions:

1. Under engineering and under planning - solutions tend to start out as ideas that were never fully fleshed out, proper planning did not take place, and likewise the implementation of the idea never appears to end up more than a glorified proof of concept.
2. Over engineering and over planning - solutions undergo huge discovery, planning efforts, use case modeling, weeks of meetings etc. Typically the project consists of a good plan, and good intentions. Ultimate results depend, however frequently are over due, over budget, and tend to put robustness over performance.

Of course these are fuzzy generalizations, but the point is that a project doesn't have to go one way or another, and in fact I think it should be goal to be at a happy medium. If no planning occurs, a project has slim chances of success, and likewise if things are overdone.. the projects tend to seem to grow in weight quickly and potentially have the manageability problems like the no planning route.

 Is it going to be ok to "fly by the seat of your pants" for x feature? Do we need to implement the provider model for every chunk of functionality?

I don't mean to make light of the topic of data abstraction or data providers, but I've seen all too many companies commit to building out a suite of providers for the data layers and then in the entire 10 yr lifespan of the application... it never changes. In addition, a lot of times the implementations of one servers client functionality is so different that implementing truly pluggable providers is something that will require a significant effort.

My point isn't any one specific aspect of software design, or that you should choose to lean one direction or another, but rather to suggest that software design is an art, a learned skill... and should never be a hard, predefined path.