Technical debt in semiconductor equipment: it's time to pay it down

07/01/2010

Executive Overview As the semiconductor market begins to recover and customers begin to order new products in higher volumes, product competitiveness, and time-to-market will be absolutely critical. This is a good time to take stock of your development readiness and to understanding how well legacy software assets will support the product roadmap.

Dan O'Connor, Foliage, Burlington, MA USA

The software development community is known for inventing useful metaphors as a means to relate technical topics to "real world" concepts. The technical debt metaphor was first introduced by Ward Cunningham [1] to compare deficiencies in software with financial debt, a topic that is now all too familiar for most of us. Just as the U.S. will pay the penalty for excessive financial debt with slower growth and less economic competitiveness for the next few years, excessive technical debt has the impact of slower development velocity and reduced product competitiveness.

There are many situations where a development team will decide to ship a product with known deficiencies in the software design and/or implementation. The code freeze date was announced months ago and there is a scramble to get the feature enhancements completed. Consequently, additional shortcuts are taken. Finally, the decision is made to move forward. Ship it! Any additional improvements will have to wait until later. But later too often translates to never; it is all too easy to forget about those shortcuts and deficiencies after the product is released and the organization moves on to product support and enhancement activities. This is an example of technical debt; it must be recognized to be serviced one way or another.

The technical debt metaphor turns out to be a useful vehicle for communication between product development teams and executive management, who otherwise may not be eager to invest resources in cleanup tasks, redesign, or refactoring. Servicing debt, however, is a concept that management does understand, and its importance tends to sink in when development can identify the specific costs associated with long term, unattended debt.

Deliberate debt and the downturn

Martin Fowler adds an interesting perspective with his classification of technical debt into quadrants [2]. His suggestion is to distinguish between deliberate and inadvertent technical debt as well between prudent and reckless debt.

It's likely that most equipment control systems have accumulated deliberate technical debt during the last several years. Many capital equipment firms have been running on a skeletal R&D staff for several years as software development organizations have been impacted more than in previous down cycles. General managers have been faced with difficult cost-cutting decisions and in many cases have had to put R&D on ice to wait out the downturn.

In this environment, it is prudent to take on some technical debt. In many cases there are very few feature enhancements being made; only critical bug fixes will be released. There are simply not enough development resources and budget available to make significant improvements in the software.

Assessing technical debt

It is clear that, with the anticipated market upturn, equipment suppliers will again be challenged to support the next semiconductor design node and the fabs will begin to make demands for new features and new tools. Process and metrology tools will require finer accuracy to meet ever-decreasing critical dimensions.

Technical debt can manifest itself in many different ways. There are several familiar clues to look for however, and tools can also help identify potential problem areas.

Complexity. Complex code is a fundamental sign of technical debt. Many legacy control systems have a few infamous code modules that are difficult to maintain. You may have heard statements such as, "No one understands how the recipe manager works" or, "The scheduler code is too fragile, only Steve can modify the scheduler." If these comments resonate, then you have experience with technical debt and the consequences of reduced ability to modify or enhance modules or poor reliability due to the inability to appropriately test the application. McCabe cyclomatic complexity is the most common complexity metric and many static analysis tools can easily calculate the measurements for your project. The Software Engineering Institute [3] publishes guidelines for interpreting complexity metrics (Table 1).

Code duplication. When a code freeze is approaching, "cut and paste" can be a favorite approach in the developer's bag of tricks. This is classic technical debt: shortcut, now, pay for your sins later.

Documentation debt. The next item that tends to get jettisoned at the release deadline, or during periods of tight development resources, is documentation. Developers are not known for being enamored with documentation in the first place. Lack of documentation does not directly affect the runtime characteristics of the software system, but missing design or test documentation needs to be counted as technical debt because there is interest to be paid.

Testing debt. Unit testing is another item that gets squeezed due to schedule pressure. The interest on this type of debt is the lack of a safety net to catch regressions. Current best practices in software development call for unit tests for all modules and for running all tests during continuous integration. Michael Feather calls this "code that bites back," [4] meaning that the system informs you (usually by email) if your submission just broke a unit test. Again, good code coverage tools exist to help you understand your testing debt.

Architectural debt. Architectural debt exists when the software architecture is no longer well-aligned with the key product drivers and this can happen in two ways. First, developers can cause architectural decay if they do not understand and follow the architectural rules and maintain the intended conceptual integrity. A good example is not following the architectural layering rules and perhaps introducing a circular dependency. The second path to architectural debt is when the product itself evolves well past the capability covered by the original architectural design. This often occurs with very successful products or product lines; if the market loves the initial product capability, it tends to want to leverage it in new applications and demands additional features, modifications and configurations.

Tracking technical debt

It is important to make these different types of debt visible. Some agile development teams use informal debt statements, written on index cards and posted on a bulletin board. Other organizations use a more formal tracking method which calls for developers to enter debt issues into the tracking system. This allows the organization to prioritize the issue and estimate the time for the required refactoring. It also allows the debt issues to be added to the schedule and to be managed like any other development task.

Another established, but simple, effective method is the use of a "pain dashboard" to track technical debt. This can be manifested as a wiki page where developers could easily add entries describing the parts of the system that are difficult to understand and debilitating to their productivity. The system can allow for a voting scheme so the team has a self-balancing mechanism to continually prioritize their technical debt.

Bankruptcy as the last resort

When faced with crippling financial debt, bankruptcy is always the last resort. We can extend the technical debt metaphor nicely here. For equipment control systems, bankruptcy occurs when you conclude that the legacy code base is a dead end and is no longer viable to support the forward-looking product roadmap. A complete rewrite and a new platform are needed for the organization to be competitive going forward. In this case, the technical debt can be retired along with the legacy system, and like filing Chapter 11, you are no longer responsible to address all the sins of the past.

Initially, this may sound like an attractive option, but the decision to undertake a rewrite should never be taken lightly. Developers rarely get the opportunity for a clean sheet redesign, and for good reason; it is expensive. Developing a new equipment software platform can represent an investment of perhaps millions of dollars.

Still, it is true that control system software does have a finite shelf life. If the existing equipment software is ten or more years old, there is a good chance that it does not leverage advanced off-the-shelf software technologies or modern design and architecture concepts. A redesign represents an opportunity to significantly increase the productivity of your software staff (with a resultant cost reduction) as well as a chance to retire a decade of technical debt en masse.

Benefits of paying down the debt

Your software control systems have, more likely than not, incurred significant technical debt during the economic downturn. It's time to pay down the technical debt and your organization will benefit:

Increased R&D efficiency and improved time-to-market. Once the code base has been cleared of crippling technical debt, the development velocity will increase. The design refactoring is effectively cleaning the molasses out of your software development machinery. You can expect subsequent software modifications to be implemented more efficiently, and therefore, time-to-market for new features and new products will improve.

Hitting commitment dates. Paying down the technical debt will increase the team's overall understanding of the code base. This will have the effect of getting better estimates for work to be completed and reducing risk during subsequent modifications.

Performance and technology upgrades. Another positive side effect of attacking the technical debt is the opportunity to upgrade to the latest technologies. This can mean moving to the latest versions of compilers and development tools. It can also mean upgrading to the latest versions of the operating system and third-party libraries and possibly even to a faster processor.

References

1.  http://c2.com/cgi/wiki?TechnicalDebt

2. M. Fowler,  http://martinfowler.com/bliki/TechnicalDebtQuadrant.html

3. Carnegie Mellon, Software Engineering Institute.  http://www.sei.cmu.edu/

4. M. Feathers, Working Effectively with Legacy Code. Prentice Hall, Englewood Cliffs, NJ, 2005.

Biography

Dan O'Connor received his MS in computer science from Boston U. and is a software architect at Foliage, 168 Middlesex Turnpike Burlington, MA 01803; ph.: 781-993-5500; email  [email protected].

More Solid State Technology Current Issue Articles
More Solid State Technology Archives Issue Articles