I believe the problem is a combination of the "essential complexity" of the
business architecture (and the fact that it drives all technical
requirements), but what I believe is the main reason for unpredictability is
the "accidental complexity" that is introduced through new technologies -
much more so than most people recognize.

If you compare large IT projects to construction or Hollywood blockbuster
projects you begin to see the reasons why. With a construction project, I
have reliable specialists that I can count on. Each specialist is expected
to keep up with the changing technology, but the technology doesn't change
so quickly that it can't be assimilated predictably in new projects (think
bricklayer, concrete mixer, electrician, plumber, etc.). Hollywood is
similar, and most budget overruns tend to occur in the special effects
domain (think Waterworld).

IT is different in that specialization is dangerous from both an
organizational perspective and career perspective. To whit, if you look at
mainframe development it is highly predictable. Basic website development
is now the same - and you can get a pretty sophisticated web site developed
for you at a fixed price.

Put another way, if I can rely on specialists to do my work I can more
easily size effort and determine requirements. But new and changing
technology demands that generalists with cross-domain knowledge will
continue to thrive, and drive projects, but create unpredictability. When
dealing with new technology you're always taking risks (which the
generalists enjoy). Being a specialist is boring (but it gets the job
done).

So, the trade-off becomes: Move forward now into uncharted waters with the
latest technologies and gamble on achieving huge ROI, or move forward with
old technologies and achieve predictability with a smaller ROI.

That notwithstanding, I believe there is also an attitudanal issue among
generalists to eschew management controls, but rather indulge in their own
interests. Kind of makes the prima donnas of Hollywood seem pretty tame in
a way.

Cheers,
Neil.