Generally, the base for a sturdy construction is a robust foundation. Take, for example, the good old Romans who built a stable Europe (for a short while). As builders, they were very inventive. Their structures have lasted for many years up to this day. Among other interesting architectural highlights, Roman builders commonly used pillars to support their imposing structures. The reason to refer to the Roman pillars is because I will use them as a metaphor to propose my own three pillars of abstraction. Just as the pillars in the Roman time supported their grand constructions, the three pillars of abstraction are my foundation for a well built software system.
I already introduced my five principles of software design, but then I purely focused on architecture. However, the scope of software engineering obviously is broader, and consists of more than only architecture and design. With this post I want to address a broader the scope, instead of focusing on architecture.
Let me start by defining software engineering. For normal human beings, the essence could be summarized as “delivering a software solution for a problem”. With that being the goal and since software engineering is my job, I would like to investigate what I usually do to get from problem to solution, from my experience. I have been developing software for more than 10 years, in different teams, and on different scales. I have used many programming languages, and I have used many tools. I have been confronted with different software engineering processes and I have developed a customized software engineering process. But each and every time, one thing was the same for me: I relied on three levels of reasoning about the original problem and its software solution. Three levels of abstraction.
Before going straight into the details of each of the three levels, first a word about abstractions in general. There’s a definition of abstraction on Wikipedia, that says it all: “a mechanism and practice to reduce and factor out details so that one can focus on a few concepts at a time”. The main goal for abstraction in software engineering is to be better able building and understanding a complex system. Besides focus, abstractions also allow to spread people’s expertise, because a different concept might require different expertise. Furthermore, estimations can be more precise per concept and it might as well be easier to determine the amount of risk per concept. As a result, overall estimations and risk determination can become much more reliable.
Alright, let’s go back to the basics. As I said, I always rely on three levels of abstraction. Namely, the three pillars of abstraction:
- Function
- Architecture
- Implementation
Each of them focuses on a different level of detail. More specific, a higher level of detail leaves out as much as detail as possible from the lower levels.
There is a chance that you got scared looking at the list, because it might look an awful lot like the Waterfall model, or it might look like it promotes BDUF. But no, no, no, it has nothing to do with neither of them. The Waterfall model has similar concepts, but imposes a strict order on them. For example, the Waterfall model says that you cannot start designing before you have put down all requirements. The same holds for BDUF, which tells that the implementation cannot start once the design is not finished. However, the pillars of abstraction do not impose any order; you can be reasoning on the level of architecture —- and even implementation —- while you are busy nailing down the requirements. Likewise, you can be reasoning about architecture while you are already coding (which is not such a bad idea anyway). The other way around, the different levels of abstractions can be perfectly used too in the Waterfall or BDUF way of working (not that you’d want to be working that way). In general, the pillars of abstraction are valid independent from the software engineering process that is used, whether it is an iterative or Agile process, such as XP, or the conservative Waterfall model. They can be best seen that they implement a way of “separation of concerns” as Edsger W. Dijkstra put it so eloquently already in 1974.
Starting at the first level of abstraction, the functional level, the focus is on the problem. The details of the solution are factored out. Though at this level things are also comprehensible for the customer or end-user, it is the toughest and most tiresome pillar to be reasoning about for the average software developer. This is not strange, considering the fact that defining the functionality is, as Brookes explained it, the inherently difficult part. Consequently, this pillar is usually not that strong. My experience is that the capabilities to reason about the use of a system, and why it is being built are generally very meager among project members. That gives us a thin functional pillar.
On the architectural level, the next pillar, the focus is on the solution. How is the desired functionality converted to a software system? Which algorithms are used? What is the blueprint of the system? This is the context wherein the five principles of software design can become of use. Without the exact details of a particular implementation, on this abstraction level the machinery of the system can be communicated. This is the level where programmers start to have a little bit of fun. Still, they find it usually very hard to separate the implementation level from the architectural level of abstraction.
Last, and certainly not least, we have the implementation level. For programmers this is the level at where they can enthusiastically debate with each other for hours. This is their most comfortable domain. This level is about the technical details of the solution.
An example.
Suppose somebody wanted me to make a PDF viewer, by way of an example, with only the limited functionality to browse through the pages in a PDF document. The functional abstraction level leaves out everything that has to do with the architecture and code. That means that on functional level we could be talking about how a user might select a file to view. Which buttons he or she would use to go to the next page, and if the scroll wheel of the mouse will work too. If the program will work on Mac OS X or Windows. All terms that an end-user will more or less understand.
On architectural level, I’ll be thinking of a PDF parser, a PDF renderer, and some building block that will read user input. Whether to separate reading keyboard and mouse input, is a relevant question on architectural level. Maybe support for different types of documents, gives rise to an intermediate data structure in the design. The PDF parser fills that data structure, and the renderer will render the custom data. Making the renderer independent from PDF. For a simple PDF parser that architecture might be overkill, but for a more generic document viewer it might be a good idea.
At the level of the lines of code, we might be discussing whether to use the Qt or the wxWidgets library. Whether this or that function should have a reference as argument or a pointer. What the size of the blocks will be in memory when we are parsing the input PDF. And so on.
Another example: the elevator test.
In the process of writing this post, Jeff Atwood wrote about the elevator test. This test is closely related to the level of abstraction, I think, and gives a nice practical example of use too. The test is about the fact that very few developers can explain why they are coding. Usually when you ask a developer what he is doing, you will get the implementation level answer. If you are lucky, you will immediately get the answer in terms of design. Then you start asking why. Slowly you will move towards the functional level. You pass the elevator test if you can give a functional level answer about what you are doing within 60 seconds.
Whether or not you can go from implementation level to function level in 60 seconds, for me, all the different levels count. I think it is a good thing to know the separation between each level. I already noted that commonly the functional pillar is underdeveloped and that separating the architecture from the implementation is difficult for developers, and, as a result, we have a skewed foundation. A thin wooden pillar for function, a bit thicker architectural pillar, that is partly blended together with a firm pillar for implementation.
For the best result, of course, the pillars should be balanced.