I may have published this reminiscence here before, no matter. If you’ve already read it, feel free to skip it. As many of you may know, I was an engineering/scientific programmer in my prime, working for a major oil company and in Silicon valley. The industry has changed much since then, and so has the technology, and I haven’t kept up much with either. Also, as I re-read these remarks (written 20 years ago) I realize I have as well, and I occasionally wince at some of my quaint opinions and prejudices. No matter, I was there, and this was how I remembered it, and I was as honest as I could be when I wrote it. Hopefully, you may profit from this historical document.

The period covered here was dominated by batch processing on big mainframes. The minicomputer and interactive processing was making inroads during my salad days, and by the time I left programming in the early 90s workstations linked to big servers (with graphic interactive interfaces)were just starting to become common. But hopefully, some of the philosophical issues I address here still have some relevance today.

I remembered quite a bit of my college FORTRAN, I only required a few days of reading as a refresher, something I took care of before I started. I could write clever and intricate code and solve involved and complex problems. My shortcomings were that I had always written my own programs, now I was working in an environment where my programs were simply outlines which called and executed other programs, pre-written routines which had either been purchased outright, or had been written by my colleagues and their predecessors. The skill was not so much in how you devised the algorithm, but in locating and linking this other work to your own; it demanded knowledge of the software libraries and of the software tools used to manipulate and manage them. Programming was no longer an intense individual activity, it became public and shared, it was the beginning of the automation of bureaucracy. The skill demanded by this technology was more organizational than mathematical. The key to writing good code was not to make it compact and efficient as I had been taught, but to make it clear and easy to follow, so that it was easily modifiable by others and by yourself. This demanded certain organizational methods and techniques that, although essential to the long-term usability of the code, imposed annoying obstacles to getting quick results. It also meant paperwork, to help keep track of everything. The reason for this was that programs were being constantly adapted and modified to slightly different uses, which in turn introduced subtle errors which might not manifest themselves until much later. Most of one’s time was not spent solving problems with the computer, it was mostly consumed in solving computer problems. As the software structures we created became ever more complex, they became increasingly more prone to failure, and the corrections we made to errors often introduced errors themselves. No one understood everything, so we became specialists with all the friction at the boundaries that implies. The tools and procedures we were given to help manage this complexity only added to the complexity itself. I began to become aware of a characteristic of software that I believe imposes natural limits to its utility. All of man’s artifacts, be they mechanical, or social, or procedural, are bedeviled by two contradictory requirements. We like to make them as capable as possible, but we also want them to be as reliable as possible. Of course, these are antithetical. The more useful something is the more complex it has to be, the more likely to break down as a result, and the more troublesome when it does.

Think in terms of a mechanical object, say a fighter plane. The designers try to come up with a machine that is a compromise of all the properties which make a fighter effective: maneuverability, armament, speed, range and so forth. In addition they have other capabilities which have to be factored in, ease of maintenance, of manufacture, of training the personnel to fly and fix it. Then there are the constraints of cost, if you make it too good you won’t be able to make enough of them. All of these properties can be enhanced at the expense of the others, but what makes an effective fighter is the right mix of all of these combinations in perfect balance, as it is tested by actual operations and eventually combat. Engineers are supposed to be able to do things like this, to anticipate the requirements and to produce that blend of factors that most effectively solves the problem. This combination is reflected in the final design, it is the mental understanding of the engineer translated to metal shapes fitting and working together.

Let’s assume that the design is perfect, it never is, but let us assume for the sake of argument that it is. Once the plane goes into operation, shortcomings are bound to become apparent, such as it needs to operate from unimproved airfields, or it needs to be able to use lower quality fuels, or it needs to carry a new and improved cannon now available. The aircraft will have to be modified to make these adjustments, but these changes will negatively affect the design (remember, we assumed it was perfect, any change compromises the original concept). Every problem which is corrected, and every enhancement to performance degrades the original design and introduces unanticipated problems which propagate through the system with unforeseen results. These provoke still further modifications and adjustments, which in turn cause more problems, and so on. As changes accumulate in more advanced versions of the plane, new problems are introduced as the old ones are corrected. Although the benefits of these changes initially may make up for the degradation of design, eventually the aircraft will become so modified that it’s performance begins to suffer. However, at this point you cannot afford to just build a new plane. You now have a constituency for the older model, pilots and ground crew who are familiar with it, factories which are now fully tooled to produce it efficiently. In short, it isn’t practical to abandon it and build a different plane, you are forced to continue operating a less than effective (and now, obsolete, because it’s been in service a long time) system because you’ve got too much invested in it to tear it all down and start over. These problems have been bedeviling engineers for a long time, and they have developed strategies to deal with them. For example, they have learned to design aircraft so that they can be easily modified down the line. They no longer just design for performance, they design for future evolution of the system. Unfortunately, software engineers have yet to learn to make these adjustments.

The same problems arise in the writing and maintenance of software. But there is one major difference, with a mechanical system, like our fighter plane, you just don’t call the assembly line and tell them to say, modify the plane so it can be used as a bomber, or strengthen it so it can be flown from aircraft carriers. Major changes like this require major forethought and careful planning, and usually a lot of testing. However, with software, all you have to do to make major changes is to sit down at a keyboard and do a little typing. There is no one there to say “Wait, this turkey won’t get off the ground”. The result is that most of our software is loaded with problems, the problems are getting worse, and it is compounded by the fact that very often perfectly good software is replaced by inferior or problematic code simply because the older is considered to be obsolete. Competition and the market encourage us to try to exploit every technological advantage as soon as it’s available, even though existing systems may already be optimal for solving our problem. Worse still, even if the new system is the perfect solution to our problem, the staff may not be trained to use it, or the manuals may be poorly written. Software engineers are only now coming to grips with the problem of designing for future evolution of the system.

Software people often talk in terms of ‘systems’, using the dictionary definition of the term as a complex collection of interacting parts. But they usually understand the term to be limited to the machines themselves, and the programs which execute on those machines. But the system also includes the people who use these facilities, those who operate and maintain it, their training and job conditions, the support staffs, even the manuals and the librarians who take care of ordering new ones. This system is extremely complex, and therefore, prone to failure. It makes no difference that you have a suitable routine to solve a particular problem if the new guy you’ve hired doesn’t know where to look it up. It takes a long time to learn the intricacies of how a system is implemented, even if it’s just the small part you are tasked to work on, and by the time you start getting good at it the technology changes and the system is replaced by something ‘better’. Before long, even the most experienced staff is busy playing catch-up and no one really knows exactly how the damn thing works. This arrangement tends to favor the individual who learns
quickly but superficially, who is satisfied with short-term results, not long term understanding.

Another example: NASA has launched many spacecraft to explore distant points in the solar system, and software plays a key role in how these robot spaceships work. But it takes years to get a space probe operational, political and funding delays as well as long development times usually mean that by the time the spacecraft is launched the computers aboard are obsolete. Add to this the length of the voyage itself, which may take years, and by the time the craft gets to where it’s going the software systems on board are positively archaic. In spite of this, the operators of these spacecraft have proven themselves to be extremely clever at squeezing totally unexpected performance from these systems, working around catastrophic malfunctions, often remotely under very difficult conditions. This has been possible because mission staffs are devoted to the mission, not the technology, they have risked their careers by learning everything there is to know about a now-obsolete system. And they have had the time to learn to get good at it. This is not the mind set encouraged by the working environment of most technologists, where experts on old systems are considered fossils or Luddites. The result is that for the most part, programmers and system engineers do not know what they are doing, everyone is working in the dark, by intuition. We use only a tiny portion of the capability of our equipment, and we cover up our failure to utilize it fully by constantly demanding even more capability. The situation is not quite as bad in hardware, where it takes time and effort to move a concept from the engineer’s mind to the marketplace, after all, there are all those factories and machine tools which have to be mobilized. But in software a fundamental change can be typed into a keyboard in the morning and out to the users in an afternoon email. This is why computer hardware is so reliable and computer software is so prone to failure. Putting it another way, we couldn’t afford to build the Panama Canal today, the software costs would be too high.

Managing these issues is what programmers really do for a living, not just writing code. The job is procedural and administrative, and the key issues are in devising systems of manageable complexity. This means developing methodology, documentation, devising structures; that is, designing software systems the way an architect designs a building. Of course, there is a hierarchy of design, deciding how to go about writing a sorting routine is a little different than writing a major computer operating system, but the key is that the actual committing of the individual instructions to paper is the last thing to be done, not the first. My task at the lab was to take responsibility for several mapping programs; in the trade jargon this is called ‘software maintenance’. My job was to locate and correct system bugs as they were reported by users out in the field, and to install changes and enhancements requested by them. I was responsible for deciding how to implement these changes, coding the changes, testing the modified software to make sure that existing capabilities were not degraded by my improvements, and then installing it in the production version of the program. It also meant maintaining slightly different versions of the same software at different data centers throughout the country. All of this had to be done while keeping in mind that every correction and addition I made to the software was unanticipated by it’s author and had to be seamlessly integrated into a design that was not meant to accommodate it. It also meant utilizing the company’s software libraries which had preprogrammed and thoroughly tested solutions to many problems we were likely to encounter, if we did not take advantage of previously written code every job would take forever. Of course, these libraries needed software maintenance too, and so on, and so on.

What this all means is that whatever software writing skills I brought to my job, they were only a fraction of what I was expected to know. I learned that the industry average is twelve lines of new code per programmer per day. Since it is easy to write several hundred lines of good code in an afternoon, it should be obvious why programming has very little to do with what programmers do for a living. I loved to program, I did not enjoy being a programmer. As a scientist/mathematician, an outsider, I was not in awe of the mystique of the programmer. I recognized that computers were great for navigating spacecraft or solving systems of differential equations, but that the overhead was so severe that it was often easier to do a calculation by hand than to go to the trouble of programming it. I remember that the USS Dean had three clerks working in it’s disbursing office, and one didn’t fully count, he was an officer. These men handled all the ship’s financial affairs, salaries (in cash, American or foreign currency), savings plans, payroll deductions and tax records, witholdings for families at home, cash advances, and so on. They managed the books and records and also purchased food and fuel in foreign ports. The only mechanical aid they used to take care of the finances of the crew was a ten-key adding machine. Can you imagine a modern company of four hundred employees, supported by less than twenty or thirty people in Data Processing? Not to mention that almost every desk will have a computer on it. With the increased capability comes increased complexity, managing that complexity requires overhead. I have always valued computers, and enjoy working with them, but they are highly overrated as a solution to our problems. It seems that their major attractions to business are the ability to replace skilled technicians and craftsmen with interchangeable low-paid illiterates, and to replace expensive design and manufacturing processes with inferior but cheaper plug-in software solutions. We have done the same thing to the computer that we did to the automobile, and to the television. The machines are not at fault, they are potentially useful devices which could make our existence better, but we are using them for strictly commercial reasons and they are devastating the quality of our lives. This is the technology that could have taken us to the stars, we use it to peddle pornography on the Internet.