Dashboard Metric Traps That Lead to Wrong Decisions
Spot common metric traps such as misleading averages, Simpson's paradox, small bases and Goodhart's law, and use AI to stress-test your dashboard.
An online shop's whole team celebrated because the dashboard showed sales in a new channel up 300%. On closer inspection, last month had 2 orders and this month 8. Every number was correct, but the conclusion was wrong. Mistakes like this do not come from bad data. They come from "metric traps" anyone can fall into, including the AI that helps you analyse.
What is a metric trap?
It is a situation where numbers are technically correct but are presented or interpreted in a way that leads to the wrong conclusion. Dashboards make these traps more dangerous, because numbers are shown every day until people believe them without question.
Trap 1: Percentages from a small base
Percentage changes on very small numbers look far bigger than they are. Always show the actual figure alongside the percentage, and set a minimum threshold, for example not calculating growth if the base is below an agreed number.
Trap 2: Averages hide the distribution
Average order value may rise because a few corporate customers placed large orders, while most customers bought less. Use the median alongside it, or show the distribution as a histogram.
Trap 3: Simpson's paradox
This is when the trend within each subgroup goes one way, but the combined total goes the other, because the mix between groups has shifted. For example, conversion improves for both new and returning customers, yet overall conversion falls because this month a larger share of visitors were new customers, who convert at a lower rate. To guard against it, break down by key segments whenever the total moves sharply.
Trap 4: Comparing the wrong periods
Thai businesses have strong seasonality: Songkran, New Year, or marketplace double-date campaigns. Month-on-month comparisons can cause needless alarm. Compare with the same period last year and mark campaign periods on the chart.
Trap 5: Goodhart's law
In short, this principle says that when a measure becomes a target, it ceases to be a good measure. If a call centre team is measured on talk time per call, agents may hang up quickly even when the customer's problem is not solved. The fix is to pair counterbalancing metrics, such as talk time with first-contact resolution rate.
Trap 6: Vanity metrics and cumulative charts
Cumulative followers or downloads only go up, so they always look good, but they do not tell you whether the business is actually improving. Look at per-period figures, such as weekly active users, and numbers tied to revenue.
Example: an online electronics store
The dashboard showed the overall return rate falling, so the team assumed service quality had improved. Broken down by product category, every category's return rate was the same or higher. The total fell because that month the store sold unusually many items from a low-return category. The team added a permanent by-category chart to the dashboard.
Numbers do not lie, but the way we look at them can.
Using AI to stress-test your dashboard
- Give AI the summary table and ask: "Why might this conclusion be wrong? Which dimensions should I break the data down by?"
- Have AI check whether every KPI has a counterbalancing metric.
- Have AI play a sceptical executive who questions every number that looks too good.
Watch out for
- AI can fall into these traps too, especially when summarising from averages or totals. Do not make AI your only reviewer.
- Breaking data down too finely makes each group so small that random noise dominates, leading you back into the small-base trap.
- If you change a metric's definition, record the date of the change on the dashboard, otherwise the chart compares different things.
What to do next
- Pick the 3 KPIs used most often for decisions.
- Check each against all 6 traps.
- Add actual figures next to percentages, add medians and the segment breakdowns you need.
- Pair each KPI with a counterbalancing metric.
- Have AI and teammates challenge the numbers before you present.
To present verified numbers in a way that drives action, read Data Storytelling: Turning Dashboards into Action, and strengthen your foundations with the course When anyone can do analytics in the Gen AI era on Plants.work Learn. To build a career in this field, browse data roles on the jobs page.