Technical Debt is a useful analogy which can help explain to project managers, clients, stakeholders or any non-technical people how it’s possible to unwittingly accrue a burden of hidden costs as a side-effect of change, growth, delay or just taking the ‘easy’ choices during software development. This debt can in turn become a time-bomb of unexpected expense and effort when trying to finish a project or build upon an existing software codebase.

Debt is not always a bad thing. People, companies and even countries actively decide to get into financial debt to enable and accelerate progress – such as buying a house, investing in more staff and raw materials, or building infrastructure and education. Of course once you’re in debt you’ve got to keep paying off the interest just to stay afloat and you need to get back in the black before the burden outweighs the reward of the investment. Debt is a temporary tool.

We make similar investment choices when building a piece of software – anything from a simple web or mobile app up to a major business platform. This sounds fine, but the big problem with technical debt is its mysterious invisibility: how do you know a) whether you’re in debt, b) by how much and c) whether your ongoing decisions are making things better or worse?

I think an understanding of how to identify, estimate and control technical debt will help manage some of the bigger, longer technical projects in digital media and innovative, custom developments – especially Agile projects.

Agile methodologies embrace change. The philosophy is proud to differentiate itself from Waterfall in its acceptance and even encouragement of change and in its ability to build change into the core process rather than make futile attempts to blockade ourselves against change. Sources of change in a project range from positive effects such as client education, R&D, focus groups, market changes as well as the more obvious responses to difficulties or budgetary constraints, so facilitating variations to the original design is critical in enabling innovation and evolving the ideal product.

But just because Agile allows change doesn’t mean it comes for free! Twisting and turning a piece of code from one purpose to another may one day require it to be completely rebuilt or at least “refactored” to give it any longer term future. A software component originally designed to handle only light throughput might be squeezed to handle higher loads, but may contain specific architectural choices which will limit its growth.

It’s easy to see how specific an architecture can turn out, when considering the compromises made between the typical Agile “value sliders” of high-performance, reliability, scalability, efficiency, security, usability and so on. There are polar conflicts of interest between some of these: high-performance vs. efficiency for example. Imagine deciding your sports car is too expensive to run and you want a mechanic to turn it into a family station-wagon? This would require such radical changes that it would prove impracticable – a complete rebuild would be needed. This is a silly example of course, but you can imagine there are shades of this in any change to software’s purpose.

There are many other sources of technical debt. Time pressures to get to market are a typical source, as is the obvious focus of clients’ expectations on visible features. Combine this with the difficulty developers have in convincing people why this stuff matters beyond geeky purism, and it’s easy to see how we come to make short-term choices.

We’ve all been there in a busy project where the pressure to “just get something released” means we don’t spend the extra time to setup an automated testing and deployment system, we squeeze one more feature in and just FTP the files up to the server. We felt like we got more bang-for-buck on this release, but there now lurks the inconvenient truth that we skipped a few best-practises in the rush. We’ve deployed a platform which is unversioned, undocumented, unreproducible. We’ve started down a road to an expensive series of error-prone, manual release cycles – unless we go back and address the situation by shoe-horning those best-practises in now. Of course it will probably be harder to retro-fit these workflow mechanisms than it would have been to have embedded them earlier in the project, but that’s the interest we chose to pay!

So how can we tell if we are in technical debt – and by how much?

This is the 89 point question! It’s a question which will require the help of your developers to answer fully, primarily those that know the project.

Keeping track of your incurred debt in a green-field project is all about communication with the developers – both ways. You need to ensure they share the same values as those driving the project. They need to keep you informed of additional work required to get stories to be truly done, including any future integrations with other story features or polishing to particular quality standards. Pending integration and refactoring work needs to be discussed regularly both to highlight to clients the need for this ‘invisible work’ side-effect of modularity, but also to keep the developers from forgetting it!

For a highly fragmented system which hasn’t been fully planned in advance but built in a more fluid, agile way, integration work will take longer and may lead to more surprise refactoring. So don’t leave system integration until the end of the project – unless you have an entire team of API geniuses.

For an existing platform being inherited, extended or upgraded the original dev team will almost certainly have strong opinions on the quality and professionalism of the codebase – however they may be skewed by pride or fears of exposure. Some developers (more often seniors) are quite forthcoming about the quality of their productions: partly because they understand a broader range of qualities but also because they have learnt that quality is strongly impacted by the overall teamwork and thus they were not wholly responsible or in control of the compromises. Senior developers usually welcome a chance to discuss how they could improve the code given more time or hindsight. There are some great structured debrief techniques which both facilitate the blame-free sharing of honest information with a sense of closure on a difficult project as a bonus.

Throwing a fresh developer at a codebase could also be a good way to briefly test whether it’s well organised, documented, easy to set up and build. If they are senior enough, they could peer review the architecture or APIs and any key areas in the code, performing a kind of risk assessment or due-diligence. Doing one small end-to-end change could provide a benchmark for future work estimation.

Talking to the project manager or client, will give you insights into the values which drove the initial development. Was it a tight budget or deadline? Was it well planned? Were there a lot of changes? Did it go massively over budget? Did everyone work late, stressed or in conflict? Were there a lot of problems, bugs, delays? Were the time projections for technical task accurate? Quite simply: code which is written in difficult situations will have more technical debt.

From these discussions you should be able to gauge the general feeling of whether the application was slapped together with string, or if it’s a robust, scalable beast in the making. So it would be useful to start the conversation with the kind of direction in which you are intending to expand the software such as more traffic, different user or device demographics or new functionality.

There are some litmus tests which could give you an idea of whether the project has been invested in or run on fumes.

Automation of build, test and release are relatively expensive to set up but very quickly reduce the time and cost of manual labour for an ongoing product development cycle. Software which is being manually built, tested and deployed clearly has had little investment made beyond building features and is living in debt.

Software design patterns are also a good sign. If the architecture exhibits clearly identifiable, standard patterns such as Observer, IoC, Facades or Delegates and architectural patterns such as N-Tier, MVC or MVVM, then a new developer will have a big head-start on understanding the codebase because the overall structure will be like a good workshop – “everything has a place and everything in its place”. They will only have to focus on the functional code modules – not the whole system. However if the system does not have a typical overall structure, there will undoubtedly be a lot of custom infrastructure code which will take time to understand. Sometimes this service skeleton will be horribly mingled in with the functional code making comprehension, debugging and changes very difficult.

Do the developers have a good environment? “The Joel Test” outlines common environmental concerns which impact the quality of code (and technical debt of some types). Source control, bug tracking and regular builds are the 101’s on this list – without them the team has little chance of staying organised.

So how can you make decisions which will reduce or manage unexpected technical debt?

The broadest guide is simply to compare it with a real debt situation. Are you sinking investment into this application, caring for its future and letting the production team exercise best-practises? Are you letting the developers spend time on non-functional aspects and housekeeping which may have no immediately tangible output? If so, then you have the right mindset to begin avoiding future debt. Ignoring developers’ housekeeping pleas is like ignoring your credit card statement – you can do it, for a while.

Revisiting the team’s agile value sliders might be a good way to reaffirm your commitment to what should underpin future developments, such as staying ahead of technical evolution or hackers, globalisation or localisation, new integrations or migrations. Establishing a trusting relationship with one or more key developers will help you tell the difference between well-considered technical infrastructure investment and personal geeky elitism.

Post-analysing task blow-outs should give you insights into whether it’s caused by the hidden effects of earlier cost-cutting. Draw up a pie chart of the extra time needed on a task or project from the time-tracking system and look for common themes which smell of interest payments (i.e. ignore changes, basic underestimations, task-switching, hangovers etc). Look for chunks of time unexpectedly spent before the ‘real’ task could really begin, as these may indicate housekeeping work which was left pending from the last iteration – classic work debt.

At the end of the day the problem with having this debt is the unexpected cost it adds to your tasks. So the only truly scientific way of assessing the success of your “repayments” is to monitor the efficiency of the work and the accuracy of time quoting for technical tasks. If the error margin of predicting work is coming down and there’s less talk of “Oh I had to refactor a bunch of stuff first, which wasn’t in my quote”, then you may have already accommodated for this extra work. Of course trends like these only become viable to recognise and utilise for feedback in longer-term projects with more data history. But then again for short, throw-away projects like campaigns technical debt isn’t a problem!

For further reading, have a look at these great descriptions:

http://martinfowler.com/bliki/TechnicalDebt.html – a short, very readable description by a the leading software engineering writer.

http://www.scrumalliance.org/articles/451-technical-debt-for-pms – some descriptions of how a developer can measure/identify debt.

http://en.wikipedia.org/wiki/Technical_debt – a good plain definition of it, and where it comes from with links to videos of talks and seminars.

http://www.c2.com/cgi/wiki?TechnicalDebt – Ward Cunningham’s original invention of the analogy.

http://blogs.construx.com/blogs/stevemcc/archive/2007/11/01/technical-debt-2.aspx – Steve McConnell discusses intentional vs. accidental and short vs. long term debt.