In 15 words

Incentives can be a quiet corroder of data quality in your company.

In a few more words

We all know the conventional "you get what you pay for" wisdom. But this common trope is particularly insidious for organizations when it comes to data quality because of how quietly and honestly it manifests. 

Modern management demands measurement: quarterly, biannual and annual reviews, rating systems, and personality tests are all tools to assess, track, and communicate about performance. They create data to support evaluations. The booby trap, though, arises from the fact that the data used to measure individual performance is oftentimes the same data used to assess the business's performance. As a result, leaders risk finding themselves in a "How to Lie with Statistics" situation.

Three real-life examples

Organization A: "We push code twice per day"

This is one of the stats cited during new-hire orientation, and emblazoned on the company website as a core value. As a shorthand for "we move fast and break things," releasing new features, fixing bugs, and deploying code to the production environment twice per day is meant to signal a highly productive organization.

The reality: Software development teams deploy code at such a rapid pace that they don't have time to coordinate with other teams, assess potential downstream impacts, or communicate what changes are coming.

The consequence: Disjointed user experiences, stressed software developers, and extremely frustrated data teams. As downstream consumers of data, engineers and analysts are the first-line of defense against changed data. Sudden and significant shifts are quickly spotted, but subtle changes often pass baseline checks. As a result, many changes aren't noticed until end-users raise questions. It is not uncommon for data analysts to field multiple questions per day from business users trying to make sense of unexpected or recently-changed data. This strains the relationships between the business and the data team.

The lesson in incentives: When developers are incentivized to push feature changes rapidly, they don't have time to communicate potential consequences to downstream team members. Ultimately, this leads to trust in the data declining because the meaning of the data is changing.

Organization B: “We value experimentation”

Experimentation is often considered one of the hallmarks of a data driven company. At one large e-commerce company, experimentation is considered so important that product managers’ compensation is directly related to the number of successful experiments created during the review period. 

The reality: When volume and positive outcome are rewarded, quality can decline. For the data scientists supporting the product managers, the experiment design negotiations are fraught with tense exchanges,

“You can’t test eight things at once,”

“Not enough time has passed to be confident in the effect,”

“Correlation doesn’t mean causation,”

“There’s a confounding variable at play here,”

And perhaps most importantly, if the outcome is a lack of support for the hypothesis, then the experiment is considered a total failure.

The consequence: Rapid, half-baked experiments lead to a lot of data. And given the reasons cited above by the data scientist, it can be difficult to tell which experiments are truly impactful and which ones are just noise. In a counterintuitive outcome, the incentive to experiment allows executives to claim being, “data-driven,” but the on-the-ground reality is that the org was in a state of “data-drowning.” 

The lesson in incentives: When the product team is incentivized to fulfill a quota of experiments, the meaningfulness, design, and conclusions suffer as a tradeoff. The risk is trust in data declining from producing too much noise and not enough signal. 

Organization C: "We can address this with a bonus plan"

At a different company, the Chief Product Officer was worried: decades of rapid releases in agile fashion had produced a mountain of tech debt. Developers were supposed to go fast, and now the infrastructure was straining under the enormous weight of workarounds and band-aid solutions. In a hail mary attempt to get the department back to net-zero on tech debt, the CPO decides to create a bonus plan: everyone will receive a $2,000 bonus for their individual tech-debt-reduction efforts, and an additional payout if the developers achieve a group-level goal. 

The reality: Knowing they’d have to substantiate their bonus worthiness with data, developers scrambled to create data breadcrumbs. Whether it was pushing code, writing new documentation, or creating monitoring dashboards, they all put in new efforts. One data scientist made a Tableau dashboard to monitor the uptime of an application that housed a machine learning model.

The consequence: The allure of the bonus shifted everyone’s priority to work on tech debt. And while this is a worthwhile - and necessary - pursuit, the structure of the bonus plan measured the wrong things. For the data scientist, using Tableau for monitoring uptime contorted the tool into an abnormal use. Whereas all of their other reporting was about model accuracy and business impact, the uptime dashboard focused on infrastructure. When it was added to the folder alongside the other dashboards, business stakeholders were confused because the purpose was so different. The questioning got to be so great, in fact, that the team had to remove the dashboard altogether.

The lesson in incentives: Every company has a story told through data that can inadvertently be put at risk when incentives promote creating new narratives (see "How to Lie with Statistics," originally published in 1954). In this case, trust in data declined because it was being used in a non-standard way that raised questions.

Bottom line

What gets incentivized gets measured, and what gets measured writes the story of a business through data. Therefore, part of data strategy should be examining and aligning the upstream incentive strategies.