← LOGS

The Dashboard Was Built by AI. Who Chose the Questions?

A finished AI-assisted dashboard can still leave its metric definitions, comparisons and decision claims unresolved.

A repeating grid of diamond-shaped tiles with one bright, upright square standing out among them.
The contrast is unmistakable. What follows from it is not.

IN BRIEF

An AI-assisted dashboard can calculate and display two percentages correctly while encouraging a comparison those numbers cannot support. If one campaign card counts purchases per visit and the other counts purchases per checkout, the displayed ranking is misleading even before anyone asks what caused the difference. Matching definitions and denominators repairs the comparison, but it does not establish causation or authorize a budget decision. The person accountable for acting on a dashboard still needs to know which question was measured, what was checked, and what remains uncertain.

An AI-assisted campaign dashboard puts two rates side by side. Campaign A shows 10%; Campaign B shows 20%, highlighted in green. The summary says B performed better, making a budget shift look tempting.

Before changing that budget, I'd want to know what each percentage counts.

The dashboard and campaigns are fictional. AI could have assembled the charts, calculated both rates and produced the summary without making an arithmetic error. The problem is whether the comparison is valid.


The two numbers that look comparable

The cards use these figures:

Dashboard card Purchases Denominator Displayed rate
Campaign A 40 400 landing-page visits 10%
Campaign B 36 180 initiated checkouts 20%

The arithmetic is correct. The comparison isn't.

Campaign A counts purchases per landing-page visit. Campaign B counts purchases among people who started checkout. Labeling both figures conversion rate makes B look twice as successful, though the rates describe different steps in a funnel.

Suppose the same fictional data also records 600 landing-page visits for Campaign B during the reporting period. Using A's purchase-per-visit definition, B's rate becomes 36 out of 600, or 6%.

On the same purchase-per-visit basis, A stands at 10% and B at 6%. The apparent ranking reverses even though neither campaign's purchase count changed.

A finished-looking chart can make the definitions seem settled, when the metric label alone doesn't disclose which events or populations were counted.


What a metric label leaves out

Google Analytics defines GA4 engagement rate as engaged sessions divided by sessions. Its documentation specifies what qualifies as an engaged session; without that rule, engagement is only a label.

GA4 also distinguishes traffic-source dimensions by user, session and event. A user's first acquisition source and the source associated with a particular session answer different questions, even when both are shown under familiar campaign labels.

For a card that may influence a consequential decision, I'd want the counted event, eligible population, denominator, reporting period, scope, filters, attribution rules and exclusions to be inspectable. All that detail needn't clutter the dashboard, but it should be available to the person acting on it.

Approved definitions and automated validation can enforce many of these checks. Recalculating a figure by hand would not catch a comparison built from incompatible definitions.


Fixing the denominator doesn't prove the cause

Using purchases per visit, Campaign A has the higher rate: 10% against B's 6%. That describes the fictional reporting period; it does not show that Campaign A caused better conversion. The campaigns may have reached different audiences, run under different conditions or used different attribution rules.

The Stanford Encyclopedia of Philosophy's discussion of Simpson's paradox shows how an association in aggregated data can reverse when the data are separated into relevant groups. We have no subgroup data for these fictional campaigns, so this is a caution about comparing populations, not an explanation of their rates.

A descriptive difference may justify investigating the funnel or audience. If a decision depends on a causal claim, it may require an appropriate experiment or observational design. Better chart formatting cannot supply that missing evidence.


The part I wouldn't delegate by accident

AI can calculate approved measures, build charts and explain results. A well-designed workflow could also flag incompatible definitions and make the queries behind a result inspectable. I'd rather have those checks in place than require someone to redo every calculation by hand.

Before letting a report guide a decision, I'd separate three responsibilities:

  1. Define the question. What decision are we considering, and which measure could reasonably inform it?
  2. Verify the comparison. Do the events, populations, reporting periods, scopes and attribution rules align? What remains uncertain?
  3. Own the action. Does the evidence warrant shifting the budget, investigating further, running a controlled test or holding off?

Definitions and calculations can be checked automatically, without requiring a manual signature on every chart. Whoever is accountable for the decision still needs to see what was defined, what was checked and what is unknown.

A large or hard-to-reverse budget shift deserves stronger evidence than an exploratory chart. The stakes and reversibility of a decision should set that standard, not how finished the screen looks.

A human-built spreadsheet can compare the wrong denominators, too. AI can produce a polished report quickly, but a finished screen doesn't show that its conclusion is sound. If a dashboard points me toward moving the budget, I'd want to know why its numbers support that move and which assumptions still need checking.


// End of transmission. Check the denominator first — ZYANE