Why Mastery of Technology is
not Mastery of Development
"In the South Seas there is a cargo cult of people. During the war they saw airplanes land with lots of good materials, and they want the same thing to happen now. So they've arranged to imitate things like runways, to put fires along the sides of the runways, to make a wooden hut for a man to sit in, with two wooden pieces on his head like headphones and bars of bamboo sticking out like antennas—he's the controller—and they wait for the airplanes to land. They're doing everything right. The form is perfect. It looks exactly the way it looked before. But it doesn't work. No airplanes land. So I call these things cargo cult science, because they follow all the apparent precepts and forms of scientific investigation, but they're missing something essential, because the planes don't land." - Richard Feynman
The same way cargo cultists create fake airports that fail to produce airplanes, cargo cult developers follow processes that superficially resemble sound engineering practice, but fail to produce working products.
Software development is like rock climbing.
New rock climbers experience something amazing: they'll attempt problems (that's what climbing routes are called - problems) that look clearly impossible to them, yet with a lot of struggle they still manage to get up the wall.
Practice eventually makes these first routes easy. And to the new climber's surprise, this ease doesn't come from greater strength or endurance, but experience that tells them that they'd been attempting to ascend in unnecessarily difficult ways.
There is a strong parallel to this in software development. There used to be an old joke about starting with a gun to your head. Tools and languages have improved, but the gun is still there, and that gun is complexity. Programming isn't hard at all to those with sufficient skill - but without care, it's easy to make it far more difficult than it needs to be.
This is a widespread problem in business system development. And unlike climbing, it's not one that plagues the newly-initiated, but the experienced.
The Psychology of the Developer
Unlike the gifts of the athlete, which can be easily seen, the gifts of the mind aren't always obvious. For developers, this lessened visibility can combine with a desire to demonstrate proficiency to produce to two professional weaknesses.
Estimation is an inherently inaccurate task in software development because everything that can be automated has been - what's left doesn't lend itself well to simple analysis. But programmers are also poor estimators because they fear that if their estimates are too large, others will look askance at them.
Paradoxically, the prospect of being labeled slow doesn't seem to bother those who don't belong in the profession - it's an accusation that only really gets under the skin of the intelligent.
Human factors are just as important in software engineering as they are in other fields; at the end of the day it's humans that must interact with and make sense of these systems. And yet there's surprisingly little serious research devoted to this - to human factors in software, for developers - and it is their natural intelligence that strangely predisposes the industry to adopt architectures and practices that seriously disregard human factors, making application development take much longer, and require many more people than necessary.
The least controversial example of this is the early and disastrously unwieldy incarnations of ORM. Tech leads would hear developers complain "it took me a day to populate a dropdown" and somehow not wonder if there was something wrong with ORM itself, and not the people trying to use it. Always the solution was, get better, more elite developers.
Two decades of failed, explosively complex frameworks have come and gone, and they've always been defended with the assertion "this is how we achieve maintainability", when it's plainly untrue. Also a common refrain: "we develop this way because we have all these developers". Those laboring under this lie rarely speak the obvious and opposite truth: "we need all these developers because of the way we develop". The claim that one can have either a maintainable or a developer-friendly design is a false choice - the two live and die together.
Good developers know that the worst mistake that can be made in engineering is unnecessary complexity; but they can fear that by openly criticizing it, they'll be judged as simply lacking the skill to wield the tools properly. Unnecessary difficulty becomes a perverse test, and the profession turns into a cargo cult.
This is why mastery of technology is only the beginning, and not the end, of mastering development: the mark of a master developer is to regard complicated and difficult designs with a deep and abiding skepticism, and to use their talent with efficiency and restraint; the greatest value-add of engineering is the elimination of unnecessary complexity.