Thought Leadership

Ensure a sustainable, cost-effective data stack by avoiding technical debt

Nick Amabile


February 1, 2021
Featured image for “Ensure a sustainable, cost-effective data stack by avoiding technical debt”

From breaking down data silos to enabling self-service access to new technologies, launching a modernized analytics program isn’t easy.

But once the pieces are in order, you can take comfort knowing your path toward long-term, data-driven success is secure.

Not so fast. Unfortunately, the work of ensuring your data stack stays effective is far from finished after implementation. Without a solid foundation of best practices, your organization can regress into the same data chaos you worked to eliminate.

If you manage data for a fast-moving organization, you’re at especially high risk. As deadline pressure mounts for product launches, analytics requirements change and your users create duplicate tables and reports. If left unchecked, this technical debt diminishes the user experience for your teams. Instead of finding accurate, easy-to-understand information tailored to their needs, users see a mess of uncertainty.

While technical debt is a natural by-product of teams working quickly to meet business goals, it’s far from unavoidable. With the right tools in place, you can set your data stack on a path toward long-term clarity and cohesion.

Otherwise, your users will grow increasingly frustrated with your analytics platform and adoption rates plummet. Not only will this diminish any gains from upgrading your data program – it will gravely impact your return on investment from a modernization initiative.

How technical debt accumulates and saps your analytics efforts

Difficult deadlines are a fact of life for nearly every business. But without the proper guide rails in place, a rush to finish a project is a contributor to technical debt.

Factor in the pressure of a highly competitive business landscape and breakdowns in data governance grow all the more likely. For example, imagine a cable network racing to launch a new streaming service. Facing an already crowded field, they needed to hit a target release date or risk falling further behind. Consequently, when the company looked to extend its data to the new product – creating new dashboards, reports, and analytics functions – their teams inevitably cut corners.

In some cases, users created multiple reports that compiled the same data in the company’s business intelligence software. Or they added new tables to the data warehouse that served a single purpose instead of taking a more considered approach. As time grows short, users distributed across the organization followed the paths of least resistance to create what they needed to meet their deadlines.

This kind of rush may allow your business to meet deadlines, but your data loses cohesiveness with every poorly considered addition. Consequently, when users go into a platform like Looker to evaluate key details such as subscription rates or sales numbers, they see multiple reports that seemingly do the same thing.

As uncertainty mounts, your teams lose trust in your data. Without the ability to determine which reports and calculations are correct or even applicable to their position, your users may give up on your data system. Soon, your organization descends into data chaos as users pull the wrong numbers to generate their insights. Or, worse yet, return to spreadsheets to generate the insights they need.

Fortunately, there are ways to avoid losing this battle – and ensure investment from your data’s stakeholders.

Prevent technical debt by retaining organizational best practices

Technical debt may be the natural result of teams working in a hurry, but it’s not the only contributing factor. When analytics teams are deployed across different areas of your business, they may not always communicate with one another about their work. By implementing a set of organizational processes, you can allow for more coordination between teams to limit technical debt before it starts.

On one hand, preventing technical debt requires balancing your organization’s need for speed and the critical importance of a cohesive data model. When firms recognize how a modern data stack allows teams to generate insights fast, they’ll often favor speed over establishing the necessary controls to prevent technical debt.

In some cases, adding much-needed safeguards to protect the cohesion of your data will slow your teams down. But by applying modern software development practices to your analytics efforts, the gains in reducing technical debt offset the cost.

Forming a data quality group that’s composed of stakeholders from across your organization establishes your business information as a shared responsibility. Anyone making changes to its definitions must document their request for internal review (or a “pull request” in software development terms). Changes should also be subject to a peer review by the data quality group, which will minimize errors or duplicates.

A centralized group of power users is part of an effective Looker rollout and provides your organization with a center of excellence that ensures your teams coordinate their efforts. But organizational changes are not the only resources at your disposal to ensure your data remains trustworthy and governed.

Data tools provide additional safeguards and governance

Ensuring data governance is critical to a sustainable modernization effort. A data catalog allows you to ensure your data sources are accurate and provides documentation of how they work together.

A tool like Alation allows you to access and evaluate information from your data warehouse and analytics platform. Alation provides a resource to validate each report is using the latest, agreed-upon business definitions. Plus, the software illustrates which parts of your organization are using each report. If some haven’t been accessed for a long period, you can deprecate them to prevent accruing technical debt. Instead of needing to consult multiple teams about how reports are calculated, a data catalog provides an accessible directory your teams need.

Your data stack also provides a means to limit the growth of technical debt. The metadata in your data warehouse and business intelligence software illuminates how often datasets have been accessed. Once you understand the areas of your data that are not being used or maintained, you can delete unnecessary clutter.

Along with allowing you to create dashboards dedicated to specific business needs, Looker lets you monitor usage rates as well. With an admin dashboard, you can generate a report from Looker to show you which reports haven’t been used over a certain number of days. With a data quality group, you have a centralized team that can evaluate these reports and work with their creators to determine if it’s still viable.

Technical debt is natural but not inevitable

Today’s competitive environment requires organizations to move fast. Unfortunately for your data, a need for speed inevitably leads to shortcuts. Your data stack must be built on a foundation that allows proper data governance through organizational processes and technical tools. Otherwise, it will grow unstable.

For all the capabilities a modern business intelligence program can deliver, duplicate, confusing information diminishes its user experience. As a result, the teams that can most benefit from its powers will lose faith in its effectiveness and opt out from its use.

Given the investment in time and energy implementing a data program requires, you need to ensure its long-term health. Fortunately, as easy as duplicate or unused components of your data can be to accrue, they aren’t inevitable. You may not be able to eliminate difficult deadlines, but you can establish systems to keep your data clean and cohesive. With the right internal processes, you can stay out of technical debt and ensure your data delivers a consistent return on investment.