It doesn’t matter what kind of technical debt it is. It could be defects/bugs, it could be delivery, it could be scale and performance, it could even be the UX. It could even be a lack of knowledge exchange and communications among business units and teams, which I refer to as organizational debt. All of it can and will eat a company alive if it’s not dealt with in a sustainable manner.
I’ve seen this too many times now not to talk about it or write something specific about how companies ignore their technical debt to their own detriment.
Technical debt can cost a company twenty to forty percent of their annual budget. If a company has fifty developers with an annual pay of $100k, spending roughly thirty-three percent of their time on technical debt, then that debt roughly costs around $1.65 Million dollars annually. Minimum. (*source)
The Red Flags of Tech Debt
Here are a few things I’ve always noted as red flags for companies that reach the five year or even ten year mark and think things are going fine, only to have someone raise the alarm, have it delayed or dismissed, until a near catastrophic or catastrophic failure of some part of the business occurs. These are a few things that could be happening on teams and in business units:
Not having any kind of Software Development Life Cycle plan, at all. And if there was one, it’s not reviewed on a regular basis. This would include not only supported platforms/devices, but also libraries and tools that are used to create the product.
Unprepared to scale up or scale down operations in a sustainable way, which maintains the budget and people’s sanity.
Unable to easily promote newer processes or technical platforms for delivery because of several limiting factors like money, cults of personalities, tight coupling of pipeline with product, unstable code, etc.
Feedback loops informing folks of quality issues are lagging, not useful (aka feeding back on the wrong information), ignored, broken, or all together non-existent.
Lack of clear communication with a customer. If someone creates a ticket, whether internal or external, and general response time is more than 72 hours. Regardless of whether the issue will be fixed or not. A customer deserves an answer within a reasonable amount of time.
Lack of onboarding framework which can also scale with the company.
Lack of accountability and planning around infra and tools.
Too many orphaned tools, processes, code which no one wants to own and are generally too afraid to clean up.
Any one of these can have a serious impact to a company’s bottom line if allowed to continue.
I’m also assuming here that the plan for most companies is to actually continue, not be bought out by a larger company. If a buyout is the goal, then helping employees frame risk around maintaining a smaller set of critical systems and processes toward that end should achievable.
Beyond Initialization
If you are an exec with a business that either creates tech or heavily utilizes it and have a desire for your business to survive, you need to do more than plan for your horizon models for revenue streams. Build a plan for sustainable obsolescence, scaling, and knowledge transfers, especially if you’ve passed the five-year mark and substantial growth is in the business forecast.
Not having someone, or several someone’s talking about sustainable growth and how to achieve it at the manager, director, and executive levels means the company is already floating in waters with a large kraken waiting to drag it down.
In my opinion, any senior manager and director should have an assessment of their business areas and understand not only what value they can deliver, but what technical debt could actually threaten that value. AND those senior management folks should require their managers to report to them not only about the progress they have made on features, but the progress they have made toward addressing any technical debt.
If you’re a CIO/CEO/CFO and want indicators of whether your company is struggling with technical debt, here are a few to watch out for:
Hidden Service Costs - the cost of any services whether created internally or third party should not out pace users or income generated from the product using those services. There might be a reason, such as start-up costs, but there should be an expectation that the cost of providing a product should be LESS over time.
Cloud services and storage are a great example. You could start with a certain bandwidth and number of cores, but if your team isn’t observing usage or peak times, then it’s eventually burning money like a leaky pipe.
Building internally is great if there is a wealth of experience, it’s a core competency of the business, or experience is specifically hired to build the tool(s) and maintain them. However, there is a misconception that building is cheaper than buying a tool first and learning from those tool(s). Once you figure in cost of building internally, cost of training, cost of maintenance and upgrades, you have to decide if this is a competency you want your company to have or if it would be more cost effective to purchase supported, widely used subscriptions.
Increase in Support Staff without a clear reason - PMs, QA, TPMs, Leads, SDETs, etc are all important to keep high functioning teams communicating and coordinated. If there is a significant increase, especially for a product which doesn’t have a lot of new development, it’s metric that should be investigated.
For example, if you have five people focused on testing a product one year, and the following year management requests more headcount without any significant changes to the software or the creation of a new product, then there’s a pretty decent chance you are hiring people to manage technical debt.
Tooling Changes without a significant cost reduction - if your teams are introducing tools and services without clear reasons or benefits, it should be investigated.
Granted, large businesses do need a lot of tools. However, if you can identify several products doing similar things but no one is actually “using” them, that’s money being wasted. Tooling bloat is very common as a company grows, but it can be managed through a technical advisory board or architecture group, along with onboarding so everyone is familiar with the tools available to help them solve problems.
Increase in Customer or Product support needs without clear ROI - customers call for all kinds of reasons. If the majority of phone calls increase because of down time, serious release issues, critical functionality issues, and these continue without being addressed quickly, or increase in frequency (outages ramp up from one or two critical cases a year to five or more) the business is likely spending a lot of money and contact time with customers to keep the perception of the company or product from being impacted by technical debt.
As a temporary measure, it’s not a bad thing to have dedicated customer service folks for accounts. However, if that becomes the norm because so many issues come up and important clients need to be reassured a lot, you definitely have technical debt that needs addressed immediately. It won’t just overwhelm customers; it will be the reason good employees leave.
Business Solutions Exist
There’s a lot of talk about 60/20/20 which is: Sixty percent new or upgrades to products, twenty percent to admin work like meetings and emails, and twenty percent to addressing identified technical debt.
Problem is that last twenty percent rarely happens (even less than the admin twenty) and it’s rarely planned or documented. So, when Janet leaves to go to a new position, and her work upgrading the libraries in an important tech stack is treated as a side project, guess what happens?
Or let’s take an even more realistic scenario. Layoffs. Systems, people, and equipment are being “carefully” decommissioned for the sake of profit and efficiency. Many times, this is when companies find out that something critical was missed.
Communicating the technical debt up, also includes communicating down about business decisions in regards to how products and tools are being used. Or how people that are no longer there used tools to create the products.
“Small” Organizational Tech Debt Problem/Example
I was working with a business group that wanted to decommission a forum with no plan to replace it. They announced the decommission, giving folks less than thirty days to find alternatives, and pointing to the official customer intake form for issues instead of the forum. The announcement triggered panic as several other business units were using the forum to literally track customer defects and communicating directly with the customer, working around the internal tracking tool and intake forms. There were also a lot of knowledge files attached to the forum which would have been lost when decommissioned because management had no idea that critical tool information was being shared and stored on the forum.
So many teams and businesses give lip service to technical and organizational debt until it drags them down and takes a bite out of their ass. If your initial business plan does not address growth, business focus changes, and possible obsolescence, it’s already creating technical/organizational debt.
Change can get out of control really fast. We see that everywhere today. If execs want to shepherd their business through tough times and harsh economic waters, make sure business units aren’t just “keeping an eye” on tech debt. Tech debt should have a plan, like business growth should have a plan - not only for the product, but the tools and people that make the product, beyond just a headcount.
Addressing Risks Early
Don’t use risk as an excuse to ignore something until it’s too late. Address it while the risk is relatively LOW. Don’t kick the can down the road because it’s low risk, unless you are absolutely sure that the risk will stay low or never change.
Assess cost of change while it’s still relatively easy verses how complicated it could be when you company grows from a hundred people to several thousand.
Identify critical & core business needs and keep an eye on those specifically and have plans to update and change those things as a matter of business practice rather than catastrophic failure.
Building a habit of addressing tech debt while a company is small, leveraging incremental change management, will reduce what could be a catastrophic failure into a less stressful and more measured response.
Acknowledge The Monster Exists
If a company is already struggling with these issues, it’s never too late to shift culture and processes to address the technical debt monster.
Businesses can begin to tackle it by prioritizing the debt which is absolutely necessary to maintain the business AND keep customers happy or at minimum still wanting to use your product and pay for it.
Catastrophic failures from technical debt can be avoided. Having a clear business plan and practices in place before things become mission critical is the key. It doesn’t have to be complicated; it just needs to exist.
Don’t wait until the tech debt monster surfaces, shift to go after it and minimize the risk to business as much as possible.
If you’re a business currently dealing with these issues and would like to discuss how to manage your operational and technical growth in a free thirty-minute chat, feel free to contact me (melthetester@gmail.com or drop me a DM on LinkedIn.)