Measuring what your ad actually did

A pizza shop owner in Manhattan tried turning off their social media ad last week. It had been running every week, and the report said it was pulling in $1,000 a week.

When they actually killed it, revenue only dropped $400.

The owner is stuck. "So what did the ad actually make? A thousand? Four hundred? Something else?"

Measuring ads starts with that question.

Quick recap of the last post

This one picks up directly from the last post. If you haven't read it, go read it first. The last post was "why bother measuring." This one is "how to actually measure." I'll keep the recap light.

The core was this. To really measure an ad's effect, you have to picture a counterfactual (what revenue would've been if the ad hadn't run), the imaginary parallel world where the ad never happened.

The problem is that the parallel universe doesn't actually exist. So the fallback: the lottery-box trick. Shuffle 1,000 lottery tickets, split them into two boxes of 500 each. Individual tickets are different, but the two boxes end up with nearly identical average winning odds, average prize, average print date. Same with people. Shuffle a crowd well enough and split them, and the two groups come out balanced on average age, average income, average purchase intent, even variables we can't measure. Show the ad (= the treatment) to one group only, and the revenue gap afterward is the ad's effect.

In reality, though, we can never build "a true parallel universe = two perfectly identical groups." The lottery-box picture is clean, but shuffling 1,000 real users cleanly is hard in the ad world (more on why in a sec). So we fall back to a compromise: "perfect measurement is off the table, let's aim for less wrong." This post walks through the concrete tools for "less wrong." None of them are perfect, but all of them get you much closer to the truth than doing nothing.

So this post is really about how to build those two similar groups.

Two main roads

There are two main roads to measuring an ad's effect.

One, build it with an experiment. Split the two groups randomly up front, show the ad to one side only. Cleanest. Though as you'll see, it's trickier than it sounds.

Two, build it from historical data. Running a random experiment is harder than it sounds. The campaign is already a few months in and you can't turn it off now. Or you could, but the lost revenue stings too much. In those cases you dig into the historical data and find "similar pairs" to compare. Cheaper, but less clean.

These two roads aren't the only methods. Each road has variants. For now, let's keep it to these two broad paths, which cover the ones people actually reach for.

Method 1. Randomized Controlled Trial (RCT)

Think of picking dodgeball teams in gym class. If the teacher says "split yourselves," friends cluster, and the teams come out uneven. If instead you roll a die and use odd/even, the two sides come out mixed and roughly matched. Drag that straight into ads and you get the Randomized Controlled Trial (RCT).

Grab 1,000 users. Flip a coin for each one. Heads 500 go to Group A (saw ad), tails 500 go to Group B (no ad). Two weeks later, compare revenue between the two groups. That's it.

Why this is clean: back to the lottery boxes. Pour 1,000 tickets into one bin, shuffle, split 500 into each of two boxes. Individual tickets are different, but the two boxes come out nearly identical on average prize, average winning odds, average print date. Flip a coin for 1,000 users and you get the same thing. Age, income, purchase behavior, even variables we don't know about, all balanced out on average. So if revenue is different two weeks later, there's only one explanation left: the ad. No other excuses hold.

One thing to watch for. It has to be actually random. Harder than it looks. "Split on odd/even user IDs" feels random, but sign-up timing or the platform's ID assignment logic can carry hidden patterns. "Monday signups to A, Tuesday to B" isn't random either. User types shift by day. For it to be really random, you basically have to flip a coin per user, in real time, at the point they enter. Lots of traps that look random and aren't.

Same shape as a drug trial. Randomize subjects into "new drug" and "placebo" groups, compare results a few weeks later. Scientists have been doing this for decades.

Of all the measurement methods we have access to, RCT is the closest to the ad's true effect. Everything else later in this post is "Plan B when you can't run an RCT."

But in advertising, it's really hard to run

The catch is that "if you can run the experiment" basically never holds in ads. Reasons stack up.

Reason 1, giving up revenue. Not showing 500 people the ad for two weeks means giving up two weeks of their revenue. "Let's eat the quarter and run an experiment first" isn't an easy call for an operator. When your CEO asks for the number at month end, "I'm running an experiment" doesn't land great.

Reason 2, ads are built on targeting. Honestly, this is the biggest one. Ad platforms look at each user and let an algorithm decide "show them the ad or not." The criterion is "people likely to buy." Likely buyers get the ad, unlikely ones don't.

So "people who saw the ad vs people who didn't" isn't two coin-flipped groups. It's two groups the algorithm already sorted into "you look like a buyer, here's an ad / you don't, skip." If you look at the revenue gap in that setup, you're not measuring the ad's effect. You're measuring "the gap between people who were likely to buy anyway and people who weren't." Is the ad working, or did the algorithm just collect buyers on one side? Can't tell.

(Targeting is where almost every trap in ad measurement lives. A whole post coming on it later.)

Reason 3, platform limits. To run an RCT on ads, the platform has to open up two things. One, truly random assignment of users to two groups (strict random, no sneaky patterns like the ones above). Two, show the ad to one group only, and completely hide it from the other. Both have to work for the experiment to make sense. The built-in lift test some major platforms offer (sometimes called Conversion Lift) is the canonical example. Most other platforms don't expose this kind of thing.

These three stack up, and running an RCT every time is basically impossible. So people shift from "randomize people" to "randomize regions."

Method 2. Geo Experiment

Instead of splitting at the user level, split at the regional level. Smarter than it sounds.

Find a twin city. Two cities right next to each other, like New York and Jersey City, similar population, similar spending patterns. Turn the ad off in one, leave it on in the other. Compare total revenue two months later. Not just "revenue from people who saw the ad," but the sum of everything people in that region bought through any path (ad, organic search, or just thinking of it).

The famous eBay experiment is exactly this. Randomly picked 68 of 210 US regions and turned ads off completely, left the other 142 alone. Compared revenue two months later. Result: basically no difference. (The original paper is Blake, Nosko, Tadelis, 2015, Econometrica.)

Why it's easier than randomizing people

To stop showing ads to 500 users for two weeks, the platform has to let you carve out that specific user group. Regional targeting is different. All you do is flip off the platform's geographic targeting for that region. "Don't show ads in Jersey City," one setting, done. You don't need a big platform for this.

Still not perfect

Geo experiments have two weak spots.

One: twin cities aren't actually twins. Are New York and Jersey City really the same? Population, income, median age, spending patterns, brand preferences. All a bit different. So "revenue gap between the two cities = ad effect" always leaves a little residue of doubt.

One thing softens that, though. Regions pack hundreds of thousands of people inside. Big samples mean random noise averages out. Compare 500 users vs 500 users and the "did we just get lucky with who we picked?" worry is loud. Compare an 8M-person New York average vs a 300k Jersey City average, and the small random variations wash out. Subtle differences between regions still hang around, but you're no longer getting rattled by "which 500 did we pick."

Two: inside a region, the algorithm is still picking people. The targeting trap from the user-level experiment doesn't completely go away at the regional level either. Inside New York, the algorithm doesn't spray the ad across every New Yorker. It still picks "likely buyers" and shows the ad only to that slice. Everyone else in New York doesn't see the ad.

Still, Geo looks at the region's total revenue, which dilutes that bias a lot. The buying-intent bonus from the algorithm-picked slice gets smeared across all 8M New Yorkers' revenue and shrinks to almost nothing.

The dashboard way of just summing revenue from everyone who clicked the ad exposes that bias head-on (that's the "$10M in ad revenue" trap from the last post). Geo sidesteps it.

But you can't run an experiment every time

That's it for the "build with an experiment" side. Clean, when it works. But in practice operators run into things like:

• Campaign has been running for months. Can't shut it down now.
• Turning off the ad means giving up revenue, and this quarter's numbers are tight.
• No experiment infrastructure, and the platform doesn't open those features anyway.

So what then? Just give up on measuring? That'd be a waste. This is when "pair-finding in historical data" steps in.

Method 3. Matching

Campaign is already running, historical data has piled up. N people saw the ad, M didn't. In there, pair up similar people. Compare revenue between the paired groups and you get the ad's effect.

Sounds plausible. One problem.

How do you define "similar"?

Example. Sarah is on her phone 2 hours a day with a $30,000 salary. We want to find someone similar to Sarah among people who didn't see the ad. Two candidates.

• Alex, 1 hr/day on phone, $32,000 salary
• Chris, 14 hrs/day on phone, $31,000 salary

Which one is more like Sarah?

Add up raw numeric differences and Chris wins. Phone diff 12 hours, salary diff $1,000. Total 1,012. Alex's phone diff is 1 hour, salary diff $2,000. Total 2,001. On paper, Chris looks "closer."

Which is obviously nonsense. Someone who's on their phone 2 hours a day and someone who's on it 14 hours are not similar. Phone-addiction levels are completely different.

The reason you get this distortion: salary and phone time are on wildly different scales. Salary is a five-digit number like "30,000," phone time is a one-digit number like "2." So a salary difference of "1,000" sounds big, but it's basically nothing per month. A phone-time difference of "12" looks like a small number, but 2 hours vs 14 hours a day is a completely different lifestyle.

Just add the two together and the bigger-scale number swallows the comparison. Salary dominates "similarity" all by itself. It's like adding "3 apples vs 4 apples" (difference of 1) to "$3 vs $4" (difference of 1) and calling it 2. Both are "1," but one is an apple and the other is a dollar.

In stats textbooks this is called a scale problem in Euclidean distance. Fancy name, same point: the bigger-scale variable eats the comparison.

Method 3-1. PSM, basically what dating apps do

The trap we hit with Sarah-Alex-Chris. How do you dodge it? Do what dating apps do.

Think about a dating app. Each person has height, age, job, hobbies, personality, lifestyle. Tons of attributes. If you compare one variable at a time, you hit the "big-scale variable dominates" trap. So dating apps let a model compress all those attributes into one "compatibility score." Then match people with similar scores. That's it.

Ad measurement is exactly the same. Just as a dating app spits out a "compatibility score," an ad model spits out one score per person (roughly "how likely this person is to receive the ad"). Phone time, salary, gender, age, past purchases, whatever, all fed in, model compresses into one output. Sarah 0.30, Alex 0.35, Chris 0.90, like that.

The next step is the dating-app step. From the people who saw the ad and the people who didn't, pair up those with similar scores. The ones without a match (extreme people like Chris) get dropped.

Then how do we compare? Pair by pair, compare revenue. Like this:

• Sarah (saw ad, $200 revenue) ↔ Alex (no ad, $150 revenue) → estimated ad effect for this pair: +$50
• Jake (saw ad, $300) ↔ Mike (no ad, $280) → +$20
• Lia (saw ad, $500) ↔ Emma (no ad, $420) → +$80

The average of those pair-level gaps (($50 + $20 + $80) / 3 = $50) is PSM's estimate of the ad effect. Chris, with no pair, doesn't enter the calculation.

Why this works: similar scores means two similar people. One of them happened to see the ad, the other didn't. So the revenue gap between them can be chalked up to "did they see the ad or not," not to "they were different people to begin with." It's a randomized experiment imitated on top of historical data.

This method has a name. Propensity Score Matching, or PSM. The name is heavy. The idea inside is exactly what we just walked through. Dating apps call the score "compatibility." Advertising calls it "likelihood of being shown the ad." Same shape.

Why it's smart

• Compressing many variables into one score kills the "big-scale variable dominates" problem. The model handles "someone on their phone 14 hours is a very different kind of person" and puts that into the score itself.
• People who can't find a pair (extreme ones like Chris) just get dropped. No forced pairing. Honest.
• All you need is historical data. No experiment, no revenue sacrifice.

Rosenbaum and Rubin, two statisticians, proposed this in 1983. It's been used across medicine, education, policy, advertising ever since. The default move when you can't run an experiment.

Still, watch out

Two weaknesses to know before using PSM.

Weakness one, the model is everything. How well that "compatibility score" is built decides the whole result. If it's sloppy, two people who come out "similar" might be nothing alike. Back to the dating app. Say the compatibility score overweights "height." You'll get "same height, totally different personalities" couples. PSM does the same thing. It'll confidently pair two people as "similar" when it's really a surface-level similarity. The scary part is that whoever's running the analysis often can't easily tell how shallow the matching was. That's the hidden trap in PSM.

Weakness two, variables not in the model don't get captured. Back to dating. Say the app bases its compatibility score on height, age, and job. But in reality, "sense of humor" is what makes couples click, and the app has no data on humor at all. Doesn't matter how smart the app is, it can't factor it in. You'll end up with couples "matched by score" who have completely different senses of humor.

Same thing in advertising. Say you fed phone time, salary, age, past purchases into the model. But the real driver of revenue is "did they move recently" (people who just moved buy TVs, appliances, furniture all at once). "Moved recently" isn't in your data, so it's not in the model. PSM then pairs a "didn't move" person with a "moved" person because their scores happen to look similar. The revenue gap is "moved vs didn't move," not "ad vs no ad." PSM counts it as ad effect anyway.

Miss one key variable and PSM's answer can be totally off. So the first question with PSM is always "did every variable that drives revenue get into the model?" And since no one has every variable in the world, you can never be fully sure.

So honestly, PSM is not as clean as a randomized experiment, and you should hold its answers with real skepticism. It's easy to run and the raw material (historical data) is usually sitting right there, which is why people reach for it. But the next methods, RD and DiD, can be cleaner than PSM for the right situation.

Method 4. Compare across the cutoff (Regression Discontinuity)

There are more ways to measure ad effect from historical data besides PSM. The most intuitive one is Regression Discontinuity, or RD. The name is heavy, the idea is absurdly simple.

Think of a scholarship cutoff. A school hands out a $1,000 scholarship to students with a GPA of 3.50 or higher. GPA 3.50 gets it, GPA 3.49 doesn't. Now look at these two students. Can you really say their abilities are "different"?

A 0.01-point gap could be a bad day, a lucky guess on one exam, that kind of thing. Their abilities are basically identical. But the outcomes split completely. One gets the scholarship, the other doesn't. A few years later, if you compare how they're doing financially, the gap can be read as "the effect of getting the scholarship itself." Their abilities were nearly the same beforehand. Other variables (home life, personality, talent) are also roughly balanced across the two groups, because they're right next to the cutoff. The only thing left to explain the gap is "scholarship or no scholarship."

How this works in ads: ad delivery rules often have the shape "only show the ad to users who crossed condition X." Examples:

• "Only retarget users who've been installed 7+ days"
• "Only show premium-product ads to users who spent $100+ last month"
• "Only trigger re-purchase ads for users 30+ days after signup"

Each of these has a clear boundary. 7 days, $100, 30 days. Compare users right across the boundary.

Say the rule is "retarget only users with 7+ days in the app." Compare day-6 users vs day-7 users. The two groups differ by a single day, so their characteristics are basically the same. Similar signup timing, similar app usage, similar purchase behavior. Only one real difference: did they see the ad. So the revenue gap between the two groups can be read as the ad's effect. Same logic as the scholarship example.

What's good about RD,

• Really intuitive. "People just across the boundary are nearly identical" is one piece of common sense that buys you a clean comparison.
• Historical data is enough. No experiment needed.
• The "hidden variable" trap is smaller than with PSM. Right across the cutoff, most variables balance out automatically.

What's limiting,

• The ad delivery rule has to have a clear cutoff. If the algorithm is doing something fuzzy like "73% chance of showing to this user," RD doesn't apply.
• You're only looking at people near the cutoff, so the effect you estimate is a "near-cutoff effect," not a whole-user-base effect. Extrapolating it to all users is a mistake.

Method 5. Difference-in-Differences

One more that comes up a lot. Difference-in-Differences, or DiD. Reads literally: "the difference of differences."

Straight to the ad example. The New York location decides to run a new ad starting in October. The Jersey City location keeps everything the same. Pull revenue before and after the ad for both.

• New York (ad ON): pre-ad $10,000 → post-ad $10,800 (+8%)
• Jersey City (ad OFF): pre-ad $10,000 → post-ad $10,200 (+2%)

New York went up 8%, so "the ad lifted revenue 8%," right? No. The whole neighborhood's spending might have risen 2% in that stretch. Jersey City is showing you that. If ad-free Jersey City naturally went up 2%, then without the ad, New York probably would've gone up 2% too. So the ad's pure extra contribution is 8% - 2% = 6%.

That's the heart of DiD. Jersey City's 2% stands in for "what New York would've done naturally, without the ad." Subtract that from New York's 8%, and you're left with the ad's extra contribution.

Same thing in numbers. (Here "September" is the month before the ad started and "November" is the month after. Both cities are measured across the same two months. The only difference is that New York turned the ad on in October.)

	September	November	Change
NYC (ad ON from October)	$10,000	$10,800	+8%
Jersey City (ad OFF the whole time)	$10,000	$10,200	+2%
Ad effect			8% - 2% = 6%

Right there in the name: Difference-in-Differences, difference (NYC change) minus difference (Jersey City change). The gap between two gaps is the treatment effect.

What's good about DiD, comes up when you can't run a Geo experiment and PSM feels iffy. The idea that "a control group absorbs the natural movement" is intuitive.

What's limiting, it assumes "natural movement is the same in both groups." If a competitor suddenly moved into the city with the ad, or if the two regions' spending patterns were already drifting differently, DiD won't catch it.

These five aren't the whole universe

The five above are the ones people reach for most in practice. The world of ad measurement is wider than that. A few others, briefly. One line each is fine.

• Synthetic Control, for when "there's no twin city." Mix together a few non-ad regions to assemble a "virtual twin." If there's no twin, cook one up from a recipe.

• Instrumental Variable, finding "accidental natural experiments" in your data. Say a telecom outage knocked out ad delivery in some region for a few days. That outage shook ad exposure but didn't directly touch revenue. You can use it like a random experiment nature handed you.

• Bayesian structural model, for when "there isn't enough data." Encode "past similar campaigns landed roughly in this range" as prior experience, then combine with the data. Historical experience props up thin data.

All three are specialty tools, for when experiments, PSM, RD, and DiD are all off the table. Each is a whole post on its own, so for now, name and idea.

So how do you pick?

The five methods, each with its own niche.

The combos that come up most often in practice:

• Native ads on a major platform: use the built-in RCT tool when it's offered (sometimes called Conversion Lift)
• Geo campaigns or off-platform: Geo Experiment
• Only historical data, no experiment possible: PSM
• Delivery rule gates ads by a condition: RD
• Before/after data exists: DiD

Some big companies run two or three methods at the same time. If different angles land on similar numbers, "okay, that's probably near the real effect" gets more trust. "Don't trust one method, triangulate" is the modern measurement best practice.

That said, RCT is still the best. Worth saying one more time. Everything else (Geo, PSM, RD, DiD, plus the three extras) comes with the burden of proving "why this method is OK to use on this particular data." "Can we assume these two regions are similar on average?" "Does the model-produced score really reflect similarity?" "Are these two groups across the boundary actually alike on everything else?" Every time the data situation changes, you have to re-prove those assumptions, and the proofs aren't easy. RCT finishes the proof at the design stage. If it's truly random, you're done. So "running something other than RCT when RCT was possible" basically never happens. Everything else is Plan B.

You can't make a perfect counterfactual, but you can make a less-wrong one

Pulling the last line from the first post, straight:

Anyone who tells you they can pin causation down perfectly is either lying, delusional, or divine. We are none of those three. So "give up perfect, take smart compromise" is the better path.

Every method in this post is a "less-wrong" measurement tool. None opens up the parallel universe for you. But all of them get much closer to the truth than a plain dashboard number. Good enough to use.

Back to the pizza shop owner. The report said $1,000. The actual drop was $400. The owner's question: "a thousand, four hundred, or something else?"

Honestly, it's not exactly $400 either. $400 is one single measurement from killing the ad for one week. That week could've had unusually bad weather, a new menu drop from the shop next door, any number of things mixed in. The "real effect" lives in the parallel universe and will stay out of reach forever.

Still, that $1,000 is three kinds of revenue mixed together. (a) Revenue the ad actually convinced, (b) revenue from people who would've bought anyway, (c) the extra from people the ad nudged into buying a bit more. What the ad actually created is (a) plus the extra in (c). Killing the ad and seeing only $400 drop out of the original $1,000 means a big chunk is (b) and the baseline part of (c), revenue that would've come in without the ad.

So the honest answer this post can land on is "Not all of the reported $1,000 is the ad's work. The real effect is a lot smaller than that." The exact number, one week of killing the ad can't tell you. The five methods are all tools for narrowing down where the real number actually sits.

Thinking about going deeper next on why RCT is so hard to pull off in advertising, and why most of the ad experiments actually running out there are more broken than you'd think. This whole post leaned on "RCT is best," but even "best" breaks down fast in the real ad world. After that: how the industry is dealing with it today, and how it should be. Also want to pick apart Marketing Mix Modeling (MMM) and Multi-Touch Attribution (MTA), one at a time.

One last line. The counterfactual lives in your imagination forever. But the quality of that imagination is something you can improve. Everything the five methods in this post do, in the end, is help you build a more believable imagined world.