.png){kind=small}
Use ROI and risk-based decision making to drive results
Executive summary
Across a large portfolio of software codebases and projects, you will find many that are troubled or failing. They experience a lack of agility, waste, schedule delays, unpredictability, quality problems, safety & cyber risk, difficulty moving to the cloud, unacceptable TCO, and short lifespan. The root-cause in many cases is that the technical health of the system (architectural health, code quality, and test coverage) is impaired due to initial design issues or degradation over time. Over the past 15 years our team, spawned out of a collaborative MIT and Harvard Business School research effort, has developed new methods for measuring and benchmarking the health of codebases. We have statistically modeled relationships between technical health and economic- and risk- outcomes felt by those who rely on a fielded system and those responsible for its continued development.
Based on this work, Silverthread offers CodeMRI® Discovery and analytics to help executives responsible for software portfolios gain control of their software. Silverthread can help you decide which systems are in good shape, and which should be refactored, modernized, or retired using financial ROI modeling. With the ability to objectively measure and visualize high-level technical health challenges and gain perceptiveness into risk and cost of ownership consequences via predictive analytics, it becomes easy to know where to invest short-term dollars for large long-term gains. Managing portfolios in this way could significantly reduce risk and ultimately save a complex portfolio owner a significant portion of the 50% of the annual software spend they are wasting today. These capabilities can also be used to better manage 3rd party vendors and make acquisition decisions. Silverthread also offers tools to help developers fix codebases, ensure that modernized systems are better than those they replace, and prevent problems in new systems. Executives can model economic improvement, fund strategic initiatives, and then oversee improvement in both technical health and financial / risk results.
Silverthread’s solutions and CodeMRI® technologies have been used in commercial and government settings on hundreds of systems, with one major US DoD customer reporting a >15X ROI. CodeMRI® is fully automated and supports most common modern and legacy languages. It has been used in classified settings and can allow stationary code to be scanned in a secure location while portfolio-level analysis is done centrally.
Half of the money spent on software is wasted
As an executive, you might find yourself responsible for a portfolio of software systems. There may be hundreds, or perhaps thousands, of operational systems, development projects, or sustainment efforts going on at the same time, all under your charge. You may also be responsible for acquiring and adopting software capabilities. When in that seat, with your resources spread thin, the ability to identify which systems should be pushed to the top of the triage list can be business critical.
Sub-optimal strategic choices and difficulty planning at the portfolio level cause organizations to waste nearly half of their software spend over the long-run. A majority of software systems and development projects in a portfolio are troubled or failing. They experience a lack of agility, waste, schedule delays, unpredictability, quality problems, safety & cyber risk, difficulty moving to the cloud, unacceptable TCO, and abbreviated lifespan. Many fail to ship at all. Unfortunately, managers often stick with a failing status-quo because they lack visibility, cannot identify the root cause of problems, cannot make an objective & financially sound business case for investing in change, and lack controls necessary to ensure that ROI is achieved.
Executive-level choices & tradeoffs to optimize portfolio ROI
Across a portfolio of development and sustainment programs, the executive has a variety of choices impacting cost of ownership and risk for each program individually. Managers can allow development to continue ‘status quo’, but also have the ability to make investments to improve & control technical health, redesign, replace, or sunset a system. Making the optimal choice for each system in a portfolio would lead to long term schedule predictability, agility improvement, risk reduction, and waste reduction across the enterprise.
At any point in time, the vast majority of projects in a portfolio are being developed in a ‘status quo’ manner. Teams are focused on developing features. They also try to fix bugs that present themselves as quickly as possible. Teams generally fail to attack underlying issues causing delays and defects, however. When the technical health of systems is impaired to some degree (as it often is) productivity and quality suffer. This places a heavy drag on the performance on the organization that may not be fully perceived. There is a systematic underinvestment in short-term efforts that would dramatically improve the long-term software economics and risk profiles across portfolios.
Executives typically wait much too long to fund the incremental improvement, redesign, replacement, or sunsetting of individual systems, even though strategically placed investments would have an enormous ROI in the aggregate. The risk and cost associated with the redesign or replacement of each individual systems (vs staying with a suboptimal status quo) is sometimes seen as prohibitive, even though a portfolio perspective pools, and therefore reduces these risks in the aggregate.
A Standish Group survey of thousands of large development projects found that 52% were challenged (significant budget overrun, schedule overrun, or scope reduction) while 41.1% were written off entirely (project abandoned or started again). Only 6.4% were successful.
Technical Health: Strongest driver of agility, risk, and waste
Over the past 15 years our research group, led by Professors at Harvard Business School and MIT, devised new methods for measuring software health. We scanned thousands of codebases and found architecture flaws in many of them. We published studies connecting architecture degradation to business impact. We found strong connections between technical health and agility, defects, system safety, developer productivity, staff turnover, vulnerability, and the ability to drive new revenue. (See publications below.) Our group also recently founded Silverthread, Inc, a company that has helped 100+ commercial and DoD customers gain visibility, quantify risk, ‘cost of ownership’ and risk consequences, and regain control.
This experience convinced us that that long-term agility is only possible if you are working inside a healthy and agile product architecture. If human beings cannot easily understand or modify their code, then teams may be working with all the best Agile practices, but their ability to respond to market demands will be far from agile. The performance of the value stream will be weighed down by technical health problems.
Predictive analytics on risk and economic impact
Silverthread’s CodeMRI® Discovery combines software architecture and financial/risk insight to enable productive evidence-based conversations between executives, PMs, and technologists.
Figure 2 shows how modularity (an important architectural health metric) in SystemM compares against thousands of others. (SystemM contained an interconnected core of 600 files where modularity had completely degraded.) Modularity (and many other codebase properties we visualize and measure) has been statistically connected to business outcomes in multiple studies. CodeMRI® Discovery allows developers to compare a system against thousands of others and to drill in on specific problem areas within a codebase to get guidance on what and how to fix.
CodeMRI® uses a battery of technical health metrics as inputs to predictive models on business outcomes. Figure 3 shows the risk and economics consequences of technical problems on SystemM’s development project. Models predict that over $800K will be wasted per $1M during development and maintenance because developers will be unproductive and spend too much time fixing bugs. The time to develop a new 1 KLOC feature will be 32 days (much longer than an Agile sprint) vs 13 in a fully healthy system. 46% of developer time will be spent fixing bugs. Development in SystemM is dangerous – on average, any change to one file will have the potential to trigger defects or unintended side-effects in up to 597 other files.
At the portfolio level, CodeMRI® allows executives to see the relative health of systems and projects so they can easily target problem areas and gain an objective lens into the cause of business challenges.
CodeMRI® helps technologists and projects leaders drill in and build a consensus and business case for change.
The Case of SystemS: Why Architectural Agility Matters
Silverthread has helped the US DoD explore several software portfolios to diagnose the root cause of project difficulties. One of the systems diagnosed was SystemS - an operational and mission critical legacy codebase with a 50 year history. In recent history, SystemS had been troubled. In 2009, executives determined that a maintenance contractor’s team had failed and awarded responsibility for SystemS to a new organization. Unfortunately, the new team also had similar challenges. They were unable to produce new features or fix critical bugs in a timely or predictable manner. This experience demonstrated that team or program leadership may not have been the root-cause. In 2015, executives again tried to fix the situation. They decided to train developers in Agile Software Development practices and modernize the development process. This initiative again failed to produce results.
Silverthread scanned SystemS using CodeMRI® and found that while it was in reasonable shape on one technical health dimension - code quality, it was had dramatically impaired architectural health. Predictive analytics showed that while a new 1KLOC feature might take 3 weeks to complete and debug in an ‘optimal’ codebase, that same feature would take over 80 days to complete in SystemS. Developing the feature was predicted to take more than 30 days due to impaired understanding and productivity. Development was also predicted to introduce or expose so many bugs that more than 50 days would be spent on debugging and fixing. Unfortunately for the team, an Agile ‘sprint’ is much shorter than 80 days. In the end, the root cause of project problems was not team quality, the project leadership or process agility. The issue was caused by failed Architectural Agility.
Modeling the ROI of courses of action
Many systems are challenged, but it is often difficult to decide what should be done. Decisions to refactor, redesign, modernize, replace, or sunset a system are made in a subjective, political, and wasteful manner. Well-meaning stakeholders lack the tools needed to target systems in a portfolio and objectively analyze the value of different courses of action for each.
Silverthread has helped several commercial firms overcome these barriers by mining data from software management systems and helping them do ROI analysis using a combination of data and project-situational factors. Executives need to consider several possibilities. For example, in many systems, technical health is troubled, but can be recovered with incremental refactoring done in parallel with normal development. The benefits of a healthier system outweigh the estimated costs of getting there. (Silverthread’s CodeMRI® Modernize developer-facing tools can help technology leaders build technical insight, fix, and stabilize a system in this case.) A second possibility is that a troubled codebase’s architectural health has degraded too far, and that the cost of a total rewrite may be substantially lower. A third possibility is that the future benefit of a healthy system won’t outweigh the cost of getting there via any means. In this instance, the highest ROI solution will be to triage until end-of-life. A fourth possibility is that a codebase is in good shape, and that it should simply be controlled and monitored to make sure that it stays that way.
If incremental improvement seems warranted, managers should model the ROI of different types of improvement. Figure 5 shows a diagnostic tool for modeling the ROI associated with three candidate efforts: improving code quality by reducing McCabe Cyclomatic Complexity, improving architecture/design quality by improving modularity, and improved automated testing. This diagnostic tool has helped leaders do scenario analysis to reason about payoff and time-to-breakeven point before asking developers to prioritize an improvement initiative above feature development.
In many cases, improving the technical health of a codebase will have a substantially higher long-term ROI than the incremental addition of new features and firefighting in an impaired codebase. Temporarily suspending development to address these issues often pays for itself several times over. Without ROI modeling and related tools to help developers fix and demonstrate progress and economic benefit, however, it is very hard for managers to prioritize complexity reduction and improved testing. With ROI modeling, managers can strategically act in the interest of long-term system health, reduced risk, and better outcomes.
What is technical health? (And tools to improve it)
Our team, during 15 years of R&D at MIT and Harvard Business School, invented algorithms for measuring different facets of technical health in a software codebase. These automated methods use graph theory to measure the extent to which a system is structured as a hierarchy of modules with strong high internal cohesion, low external coupling, well understood APIs/interfaces, healthy reuse, and a clear sense of what is top, middle, and bottom.
A healthy system has several benefits. Despite the size and complexity of a large modern codebase, separate components can be understood and managed by small teams. Changes can be made to one component without triggering bugs or side-effects elsewhere. Parts can be substituted with minimal disruption to the system as a whole, allowing evolution over long periods of time. Porting to open architectures or to the cloud can be relatively straight-forward. Replacing user interfaces is also relatively simple. The structure of the system can be mapped to the structure of the organization doing development and maintenance, allowing reasonable management and oversight. This overall structure creates an Agile Architecture – one that allows the system to be changed in unanticipated ways at relatively high speed and low cost. Finally, a healthy architecture is secure. A modular system with strong APIs has a small attack surface area and prevents would-be attackers from exploiting interactions and side-effects in efforts to defeat it.
On the other hand, a system where APIs have eroded, cyclicality has been introduced, modules converge and grow too big, and code quality problems are introduced suffers in a variety of ways. Developers can no longer understand their code and anticipate side-effects. Team boundaries erode. The ability to evolve, port to new platforms, or move to the cloud easily is lost. People become unproductive and time wasted on defects increases. Schedules become unpredictable, leading to difficulty estimating and overruns. Quality problems grow. Many defects are introduced into the field, causing safety and usability problems. Finally, an unhealthy architecture makes a system easy to exploit. Sophisticated attackers try to find an entry point and then to use interactions and available vectors to navigate to sensitive data (such as credit card numbers) or sensitive functionality (such as root-level access). Unhealthy architecture often leads to early and painful end of system life.
Silverthread has created developer-focused tools to help them build understanding, fix challenged systems and prevent new problems. When used in combination with CodeMRI® Discovery for strategic decision-making, technologists and business leaders can work together using objective analyses to make good decisions and drive results.
Conclusion: Driving change and realizing the benefit
Executives steering software portfolios feel out of control because they regularly cope with schedules that slip, unacceptable failures, and enormous waste. It is now possible to regain control by proactively targeting systems for improvement, uncovering the reason for challenges, modeling the ROI of strategic alternatives, improving technical health, and driving results.
References
[1] A. MacCormack and D.J. Sturtevant, “Technical Debt and System Architecture: The Impact of Coupling on Defect-Related Activity,” J. Systems and Software, Oct. 2016, pp. 170–182.
[2]. D.J. Sturtevant, “System Design and the Cost of Architectural Complexity,” PhD dissertation, MIT, 2013.
[3] A. Akaikine, “The Impact of Software Design Structure on Product Maintenance Costs and Measurement of Economic Benefits of Product Redesign,” master’s thesis, MIT, 2010.
[4] S.M. Gilliland, “Empirical Analysis of Software Refactoring Motivation
and Effects,” master’s thesis, MIT, 2015.
[5] C.W. Berardi, “Intellectual Property and Architecture: How Architecture Influences Intellectual Property Lock-In,” PhD dissertation, MIT, 2017.
[6] C. Baldwin, A. MacCormack, and J. Rusnak, “Hidden Structure: Using
Network Methods to Map System Architecture,” Research Policy, vol. 43, no. 8, 2014, pp. 1381–1397.
[7]. A. MacCormack, C. Baldwin, and J. Rusnak, “Exploring the Duality between Product and Organizational Architectures: A Test of the ‘Mirroring’ Hypothesis,” Research Policy, vol. 41, no. 8, 2012, pp. 1309–1324.
Contact Us
Silverthreads’s mission is to help organizations gain control of software economics, risk, and technical health. Our team includes experienced technical & operations executives, architects, system engineers, project managers, and developers. By focusing equally on the needs of both business leaders and technologists, we aim to help them build trust, make better decisions across a portfolio, fix systems, and drive results.
Comments