Measuring marketing without fooling yourself

Most marketing measurement is not wrong, it is unfalsifiable. A dashboard that cannot show you were mistaken is not a measurement system; it is a way of reporting what happened in a format that looks quantitative.

The three questions a metric has to survive

1. Compared with what? A number without a baseline is a description. Conversions went up — from what, over what period, against what would have happened anyway?
2. Attributable to what? What else changed in the same week? Seasonality, a competitor leaving the market, a price change, a release.
3. What decision would change? If the number moved 20 per cent in either direction and nothing you would do differently changes, it is not a decision metric.

The metrics that look good and mean little

• Total sessions. A rise can come from a single syndicated link and disappear next month. It also says nothing about whether anyone found what they needed.
• Follower count. A number that goes up when you are mentioned by someone larger, and down when a platform cleans bots. It is a score, not an outcome.
• Engagement rate as an average. Averaging across posts hides the distribution, and content performance is almost always a distribution rather than a mean.
• Email open rate. No longer reliable as a signal of attention, because of privacy features that pre-fetch images. It measures delivery at best.

Attribution, and why it flatters everything

Every attribution model overstates what it can see. Last-click credits the last thing before the purchase and ignores everything that made the person ready. First-click does the reverse. Multi-touch models are better and still assign credit inside the set of channels you happen to measure.

The consequence is structural: channels that are easy to track get credit for work that harder-to-track channels did. Brand, word of mouth and content usually lose under any attribution model, and lose most under last-click.

The workable response is not a better model but a stated one. Say which model you use, use it consistently, and refuse to compare a number from one model with a number from another, which is the commonest invisible error in marketing reporting.

The measurement that survives scrutiny

Cohorts rather than averages. Compare people who arrived in the same period through the same route. Averages mix old and new customers and move for reasons that have nothing to do with your work.

A holdout, where you can get one. Some proportion of the audience gets nothing, so you can see what happened without you. This is the only design that produces a causal answer, and it is often easier to run than people assume.

The metric you would be embarrassed to report. Whatever it is, it is usually the honest one: retention, repeat purchase, whether the content was read rather than visited.

Reporting that does not mislead

Report the baseline, the period, the comparison, and what changed alongside. Where a result is ambiguous, say so: an ambiguous result reported honestly is more useful than a clean one that required a choice of dates.

And name the measurement model every time. The general framework for deciding what is worth measuring at all is in market research without a budget; the specific question of content paying back over time is in content that compounds.

Choosing a small number of metrics

A team tracking fifteen numbers tracks none of them, because nobody can hold fifteen in mind when making a decision. The workable set is one primary metric, a small number of inputs that move it, and a guardrail that must not get worse.

The guardrail is the part most teams omit. Optimising signups while refunds rise is not growth, and a guardrail makes the trade visible before the quarter is over rather than after.

The measure that survives a change of management

Marketing metrics are frequently replaced when a new person takes over, which destroys the comparability that made them useful. The defence is a definition written down: exactly what is counted, from which source, over which period, excluding what.

A defined metric can be compared across years and across people. An undefined one silently changes meaning, and the chart continues to look fine while measuring something else, which is the most expensive kind of error in this area.

Seasonality and the baseline

Most businesses have a shape across the year, and a comparison against last month is usually a comparison against a different season. Where the business is seasonal, the honest comparison is the same period a year earlier, and even that needs a note about what changed in between.

Where there is no history, say so rather than constructing a baseline from a favourable month. The habit of choosing the comparison after seeing the result is how honest people produce dishonest reports.