An AI-assisted campaign dashboard puts two rates side by side. Campaign A shows 10%; Campaign B shows 20%, highlighted in green. The summary says B performed better, making a budget shift look tempting.
Before changing that budget, I'd want to know what each percentage counts.
The dashboard and campaigns are fictional. AI could have assembled the charts, calculated both rates and produced the summary without making an arithmetic error. The problem is whether the comparison is valid.
The two numbers that look comparable
The cards use these figures:
| Dashboard card | Purchases | Denominator | Displayed rate |
|---|---|---|---|
| Campaign A | 40 | 400 landing-page visits | 10% |
| Campaign B | 36 | 180 initiated checkouts | 20% |
The arithmetic is correct. The comparison isn't.
Campaign A counts purchases per landing-page visit. Campaign B counts purchases among people who started checkout. Labeling both figures conversion rate makes B look twice as successful, though the rates describe different steps in a funnel.
Suppose the same fictional data also records 600 landing-page visits for Campaign B during the reporting period. Using A's purchase-per-visit definition, B's rate becomes 36 out of 600, or 6%.
On the same purchase-per-visit basis, A stands at 10% and B at 6%. The apparent ranking reverses even though neither campaign's purchase count changed.
A finished-looking chart can make the definitions seem settled, when the metric label alone doesn't disclose which events or populations were counted.
What a metric label leaves out
Google Analytics defines GA4 engagement rate as engaged sessions divided by sessions. Its documentation specifies what qualifies as an engaged session; without that rule, engagement is only a label.
GA4 also distinguishes traffic-source dimensions by user, session and event. A user's first acquisition source and the source associated with a particular session answer different questions, even when both are shown under familiar campaign labels.
For a card that may influence a consequential decision, I'd want the counted event, eligible population, denominator, reporting period, scope, filters, attribution rules and exclusions to be inspectable. All that detail needn't clutter the dashboard, but it should be available to the person acting on it.
Approved definitions and automated validation can enforce many of these checks. Recalculating a figure by hand would not catch a comparison built from incompatible definitions.
Fixing the denominator doesn't prove the cause
Using purchases per visit, Campaign A has the higher rate: 10% against B's 6%. That describes the fictional reporting period; it does not show that Campaign A caused better conversion. The campaigns may have reached different audiences, run under different conditions or used different attribution rules.
The Stanford Encyclopedia of Philosophy's discussion of Simpson's paradox shows how an association in aggregated data can reverse when the data are separated into relevant groups. We have no subgroup data for these fictional campaigns, so this is a caution about comparing populations, not an explanation of their rates.
A descriptive difference may justify investigating the funnel or audience. If a decision depends on a causal claim, it may require an appropriate experiment or observational design. Better chart formatting cannot supply that missing evidence.
The part I wouldn't delegate by accident
AI can calculate approved measures, build charts and explain results. A well-designed workflow could also flag incompatible definitions and make the queries behind a result inspectable. I'd rather have those checks in place than require someone to redo every calculation by hand.
Before letting a report guide a decision, I'd separate three responsibilities:
- Define the question. What decision are we considering, and which measure could reasonably inform it?
- Verify the comparison. Do the events, populations, reporting periods, scopes and attribution rules align? What remains uncertain?
- Own the action. Does the evidence warrant shifting the budget, investigating further, running a controlled test or holding off?
Definitions and calculations can be checked automatically, without requiring a manual signature on every chart. Whoever is accountable for the decision still needs to see what was defined, what was checked and what is unknown.
A large or hard-to-reverse budget shift deserves stronger evidence than an exploratory chart. The stakes and reversibility of a decision should set that standard, not how finished the screen looks.
A human-built spreadsheet can compare the wrong denominators, too. AI can produce a polished report quickly, but a finished screen doesn't show that its conclusion is sound. If a dashboard points me toward moving the budget, I'd want to know why its numbers support that move and which assumptions still need checking.
// End of transmission. Check the denominator first — ZYANE
