Gurus Respond — Are We Artists?

I have always had the feeling that developing software is an art. It requires creativity, vision, devotion, and, especially Hard Work. The first time I found material supporting my theory was when I read Frederick P. Brooks’ Mythical Man Month, that contains his excellent article “No Silver Bullet”. The article argues that developing software is inherently complicated; like an art, you must be talented, or at least skilled to do it.

I was triggered to write this post, because recently I received an email via the Espresso mailing list (Espresso is a software engineering research group I’m an associated member of), from Addey De Roubaix, who is completing his masters degree (at the university of Pretoria) related to Brooks’ “No Silver Bullet”. Addey took the initiative to write to a few gurus to get their views and — surprisingly — he received pleasant and courteous answers from all of them. They are, namely, Robert Glass, Dan Berry, Scott Ambler, and Fred Brooks himself. His supervising professor, Derrick Kourie, forwarded Addey’s communiques to the Espresso mailing list under the title “Brooks and other gurus respond”.

The question was:

Do you have an opinion on whether, over the last two decades, with all of the trends, tools and innovations —- such as UML, Process Maturity, Agile Methods, Object Orientation, etc. —- we are effectively tackling the essential difficulties asserted by Dr. Brooks in 1986?

One of the responders, Daniel Berry, wrote about the subject in his “The Inevitable Pain of Software Development: Why There Is No Silver Bullet”. He has devoted a separate page on his internet site to this topic. The article sounded quite interesting to me, so I printed it, and read it in my daily train-commute. I found in it a good explanation about essences and accidents of software development. I like to quote it here to put the guru question in context:

[Brooks] classifies software issues in essences and accidents. The essence is what the software does. The accidents are the technology by which the software does the essence or by which the software is developed. “The hardest single part of building a software system is deciding precisely what to build… No other part of the work so cripples the resulting system if it is done wrong. No other part is more difficult to rectify later.” This quotation captures the essential difficulty with software that must be addressed by any method that purports to alter fundamentally the way we program, that purports to make programming an order of magnitude easier, that purports to be the silver programming bullet we have all been looking for. Heretofore, no single method has put a dent into this essential problem, although all the discovered methods have combined to improve programming by at least an order of magnitude since 1968, the year the term “software engineering” was invented.

Moreover, Brooks, based on his experience, predicted that “There is no single development, in either technology nor management technique, which by itself promises even one order-of-magnitude improvement within a decade in productivity, in reliability, in simplicity.” He made this prediction in 1986 when he first published “No Silver Bullet” in Information Processing ‘86. Since we are now well past a decade after 1986, and no such single technology or management technique has appeared, he has been proven correct. He added a slightly stronger, but still conditional, prediction with which I agree. “I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation. We still make syntax errors, to be sure; but they are fuzz compared to conceptual errors in most systems. If this is true, building software will always be hard. There is inherently no silver bullet.” Because we will always be building systems at the new frontiers opened by the latest advances in the accidents, I believe that the antecedent of this conditional statement, that conceptual errors are the hardest kind to find, will always be true. Therefore, I believe that the conclusion will always be true and that there is inherently no silver bullet.

Mister Berry is very clear about it. Developing software is inherently complicated and not only does it require Hard Work once, no, it will always need Hard Work.

I’m reluctant to copy the other responses verbatim into my post, since it was a conversation between De Roubaix and each individual guru, so I probably should ask permission before quoting their responses fully. Summarizing, they all respond that Brooks’ article is still very relevant. Not one of them thinks we have found a silver bullet (neither have I met nor read from anyone who thinks we have).

Robert Glass, for example, points out that most new technologies (like OOP, Agile, UML, Process Maturity) are much less used than academics think they are. The last 5 years of experience I myself had with software development companies confirm his point, and I think many software developers at companies can confirm this. After all, these technologies only help, and not abolish the difficulties with software development.

The following from Frederick P. Brooks himself, I could not resist to cite:

Notice that most of the people claiming higher productivity (1) are tool builders with products to hype, (2) seem to be given to future tense pronouncements, (3) are expressing unsupported opinions, without data.

Yes, as sharp as a chef’s knife. He is positive as well:

I think object-oriented programming in fact lifts design thinking to a higher level and thus addresses the essence of the problem.

An abstraction of a problem into objects and relations between objects, I also think helps tackling the problem.

So it seems, in general we are still fighting accidents only. Maybe software engineering isn’t engineering after all. As Daniel Berry, I believe too there is no secret potion. There is no one way to do it. Now suppose for a second we are artists. Even painting has its basics. Handling a pencil for example. Choosing the right paint. Choosing the right materials. It certainly will affect an artist’s paintings. I could have never expressed it as clear as Daniel Berry does in his “Inevitable Pain of Software Engineering”:

Software engineering is an art, no less than painting. Indeed, the first part of the titles of Donald Knuth’s books in his series on algorithms for computer programming is The Art of Computer Programming. The fact that software engineering is an art does not minimize its technical roots. A programmer must know the languages, the hardware platforms, the domain areas of his or her software, etc. However, what he or she does with them is very much a matter of art.

Even traditional arts, e.g., painting, have technical aspects. No matter how talented a painter is, if he or she does not know how to work with paints, his or her paintings will not be good. Furthermore, as has been demonstrated by mediocre artists, knowledge of the techniques does not insure good quality art.

Actually software engineering is an art just like mathematics. Both creativity and knowledge are required. Would mathematicians get upset if they were told that it was impossible to turn mathematics into a systematic engineering discipline? Would mathematicians even try?

If we know a domain well enough that architecture-changing requirement surprises are significantly tamed, as for compiler production these days, we can go the engineering route for that domain, to make building software for it as systematic as building a bridge or a building. However, for any new problem, where the excitement of major innovation is, there is no hope of avoiding relentless change as we learn about the domain, the need for artistry, and the pain.

Suppose programming needs talent. Take talented sportsmen, they have their favorite gear too. They are certainly far better than average with the most rudimentary gear, but to outperform other talents they will need the best gear money can buy. They fight their accidental difficulties (effectively) using the right gear.

That’s why I think addressing the accidents is important too. You need the right team, the right people. But you can make them perform much better by tackling the right accidents. For me the most important accidents are the process and the set of tools. Amongst others:

I can make the list as long as I want. If you think of it, we all have our personal list of accidents that we think are most important and should be handled. For each team and even for each project, the list probably differs every time. Actually, I’m addressing most of my own list here on this blog, but I too obviously do not have the recipe for the Silver Bullet.

Back to the initial question. Are we artists? Maybe. Either way, it is just a metaphor. But one thing I know for sure after a decade of developing software, either alone, or in a team: for software to be good, it requires Hard Work. That’s the inevitable pain of our job.

Either accept this fact, or find a different job.