Honestly, even the platform doesn't know your ad's real impact
In the last post, the pizza shop owner landed on a conclusion: a randomized experiment (RCT, randomized controlled trial) is the best way to go. You split users randomly into two groups, show the ad to one group but not the other, and compare the sales difference. It's a mouthful, so from here on we'll just call it RCT.
If you're new here, the one-line summary: the dozens of methods floating around in ad measurement are mostly attempts to get a little closer to RCT. That's how clean RCT is for measuring "real ad impact." The two previous posts cover why RCT is the gold standard, Post 1: The causal mindset every marketer must know (and quietly skips) / Post 2: Measuring what your ad actually did.
So the pizza shop owner decides, "fine, next time I'll go with that."
She walks over to the big ad platform running her ads and asks for an RCT. Normally only huge advertisers get this, but let's say she got lucky and they opened it up. (In the real world, unless you spend serious money, this kind of measurement isn't available to you. You just stare at the dashboard numbers.) The ad platform picks a random slice of users, hides her ad from them for two weeks, and then compares the sales difference between the two groups.
Looked perfect, right? The ad platform handles everything.
Two weeks later, she gets the results. But she ran the same ad three times (one result could just be luck). Here's what came back:
She's confused and asks her rep. "How does the same campaign come back with a different number every time? Sometimes +18%, sometimes +7%, sometimes even negative?"
The rep calmly replies, "Experimental results are influenced by many factors. If you run it for a longer period and increase the sample size, the numbers will stabilize. Please use this as a reference metric and evaluate holistically alongside other signals."
A textbook ad-platform-rep answer. Nothing wrong with any of it, except it doesn't move her one inch closer to the question she actually asked: "so what's the ad's impact?"
She goes home and thinks: "They said RCT was the best. I ran the best. Why are the numbers all over the place?" Sometimes plus, sometimes minus, sometimes basically zero. "I don't even know what I learned. So what's the real ad impact?"
That's the question for this post. Why RCTs are so hard in advertising. Last post we kept repeating "RCT is the most reliable method," but in advertising, even that best option isn't simple to run. An advertiser can't run it alone, and even the ad platform, which decides who sees which ad, has a hard time with it. Let's go through why.
Ads aren't clinical trials
The idea of RCT comes from clinical trials. It's still the gold standard for testing new drugs. Take 200 patients, split them randomly, give one group the drug and the other a sugar pill (a placebo). Both groups sit in a controlled environment where a researcher records who took what with precision. "People who got the drug" vs "people who didn't" are cleanly separated. The pair you compare is clean.
Ads work nothing like this. Every time a user scrolls their feed, an ad slot (a rectangular ad box inside an app or webpage) opens, and an automated rule inside the ad platform (the industry calls it an "algorithm") instantly decides what to show there. This decision happens billions of times per second across the world. The pizza shop owner can't directly control whether "this user sees my ad or not."
So you might ask, "fine, can't we just ask the ad platform to randomly pick 20% of users and hide the ad from them, then compare? Isn't that the measurement tool the ad platform offers?"
Yes. That's exactly the tool they offer. But that's where the numbers start dancing. There are four traps. One at a time.
(From here, when I say "sales" I mean the pizza shop owner's own sales. Every measurement is "money coming into her shop from the ad-exposed side vs money coming into her shop from the no-ad side." Competitor sales aren't counted by her and don't enter the measurement.)
Trap 1. The second the parking spot opens up, a competitor's car pulls in
The ad platform randomly sets aside 20% of users and hides the pizza shop owner's ad from them. When those users open their feed, the slot where her ad would have shown is empty. Does it stay empty?
Nope. Within a second, another advertiser's ad fills it. An ad slot is like a parking spot at a mall. The moment one opens up, the car that was waiting pulls right in.
Here's the more important point. Those 20% users whose slot "would have shown her ad" are exactly the users the ad platform's algorithm decided (for example) "probably likes pizza", which is why it was going to serve them a pizza ad in the first place.
And the ad platform doesn't treat that "this person likes pizza" signal as a secret only she knows. It tells other similar pizza-shop advertisers the same thing. So among the advertisers waiting for that slot, there's a decent chance a competing pizza shop is the one that gets in.
Here's where the trap kicks in. She pulled 20% of users to see "what would my sales look like without my ad?" In her head, "the world without her ad" means "everything else the same, only her ad is gone" (economists call this ceteris paribus, the assumption that all other conditions are held equal). If anything else changes, it's not a fair comparison.
But what those 20% users actually experienced is "her ad gone, and a competing pizza shop's ad in its place." A world where the competitor got one more chance to show up.
What she wants to know is "holding everything else equal, what's the difference between my ad being shown and not shown." In that scenario, "sales in the ad group - sales in the no-ad group" is the real ad effect.
But that's not how the measurement actually works. What the no-ad group users experienced is "her ad gone, replaced by a competitor's ad." Some of them see the competitor's pizza ad and buy the competitor's pizza. So the no-ad side's sales for her shop shrink more than they would in a hypothetical world where only her ad vanished.
The math makes the gap obvious.
On the left (the hypothetical) the true effect is 5 buyers, which is what she wanted to see. On the right (the real measurement), 3 people from the no-ad side switched to the competitor's pizza, so her buyers shrink to 7. The measurement reads 15 - 7 = 8 buyers. The measured 8 is bigger than the true 5. (1) real sales the ad actually created + (2) sales lost to the competitor when the ad wasn't there get mixed together and look like "a big ad effect."
Trap 2. The ad budget piles up on the people who are left
The parking-lot problem isn't the end of it. The second trap is subtler.
Say she set a daily budget of 1 million won. The ad platform's algorithm burns through it within the day. The approach varies by ad platform, some spread it evenly, some front-load the hours with better performance, but either way the baseline assumption is "spend this money today." Leftover budget is a loss for the ad platform too (lost commission on her ad).
So what happens the moment she sets "hide the ad from 20% of users"? The ad platform spends nothing on that 20%. The full 1 million won has to flow into the remaining 80%. A pizza that was supposed to feed 10 people is now feeding 8. Each person gets a bigger slice.
Users in the ad group end up seeing the ad more often than they normally would. Their sales naturally bump up. She thinks, "wow, the ad really works," but part of that bump is actually a "I saw this ad more than usual" bonus. The ad effect looks bigger than it is.
What we want to know is "when I run the ad normally, how much does it lift sales?" Not "when I pull out 20% of users and the budget bunches up on everyone else, how much does it lift sales?" Those are different numbers. The measurement itself is testing an abnormal situation (a bunching one).
Trap 1 (parking lot) and Trap 2 (bunching) look similar but have different causes.
- Trap 1, parking lot: sales on the no-ad side drop more than usual because another advertiser fills her slot.
- Trap 2, bunching: sales on the ad side rise more than usual because her own ad budget piles up on the remaining users.
Both traps push in the same direction. Both make the measured effect look bigger than reality. Parking lot bakes "sales lost to a competitor" into the ad effect; bunching bakes "sales from above-normal exposure" into the ad effect.
Natural follow-up question: "so if she just cuts her budget by 20%, wouldn't that avoid the bunching? Keep things at normal levels." Unfortunately, that's not easy either. The ad platform's algorithm doesn't disclose how many times it serves ads to each user, which hours it bunches up on, any of its internal rules. Cutting the budget by 20% doesn't mean impressions drop by exactly 20%, so she has no way to know whether a 15% cut or a 25% cut would land her back at normal. And even if she knew, the ad platform's advertiser controls aren't that flexible anyway.
Trap 3. 10,000 impressions doesn't mean 10,000 people saw it
Trap number three. Even when the ad platform logs "ad shown 10,000 times," the number of people the ad actually registered with is way smaller.
P&G, one of the world's largest advertisers, shared a number at dmexco (a digital marketing industry conference) back in 2017. Average viewing time for a mobile feed ad: 1.7 seconds. P&G's chief brand officer at the time, Marc Pritchard, called it "barely the blink of an eye." Most of the time, the ad technically appears on screen but the user has already scrolled past to the next piece of content.
Rough industry estimate: when the ad platform counts an ad as "shown" 10,000 times, the ad only lingered on someone's eyes for a full second about 1,500 times, and "actually focused on" (3+ seconds, enough to remember what the ad was) is maybe 150 times. (Exact numbers vary a lot by ad type and user, but "it shrinks dramatically" is universal.)
Think about it, this is obvious. Of all the ads that passed in front of your eyes today, how many can you actually recall? Most showed up but you glanced past them. The overwhelming majority of what the ad platform counts as "impressions" are those glanced-past ads.
Why is this a problem? Her "ad group" is padded with people who never actually saw the ad. The ad showed up in the feed, sure, but the user wasn't looking at the screen, or scrolled past, or was zoning out.
The ad platform still files them under "ad group," but the ad made zero impact on them. When the sales of these "never actually saw it" people get mixed into the ad group's average, the real ad effect gets diluted and looks smaller.
"Then why not just use the 150 who actually focused and compare them to the no-ad group?" Good question. Here's where it gets stuck.
In the ad group, you can at least find "people whose eyes lingered for 1 second" (the ad platform has that data). But on the no-ad side, how do you find the matching people?
The core of RCT is comparing two groups with the same characteristics. The 150 on the ad side have a clear profile: "the ad was shown and they focused on it." The matching group on the no-ad side would be "people who would have focused on the ad if it had been shown." But since the ad was never shown, we have no way to know who would have focused and who wouldn't.
School analogy: imagine you want to compare "students who like math," but one side is "kids who took the math class and said they liked it," and the other side is kids who never took the class at all. You have no way to know whether the second group would have liked math if they had taken it.
So you end up stuck with the crude total comparison (all 10,000 in the ad group vs all 10,000 in the no-ad group), which drags "never actually saw it" people into the ad-group average. The real ad effect on 150 people gets averaged with 9,850 people whose behavior didn't budge, and the whole thing comes out diluted.
Trap 4. Can't we just fill the slot with an unrelated ad?
Traps 1, 2, and 3 are all cases where the measurement gets distorted automatically. Trap 4 is a new trap the ad platforms created by cleverly trying to avoid Trap 1.
Back in the mid-2000s, ad platforms came up with a smart idea to fix Trap 1 (parking lot).
Remember the parking-lot problem? Competing pizza shops grab the no-ad side's slot, those users see competitor ads, and the comparison breaks.
Fix: "What if the ad platform pre-fills the no-ad side's slot with an ad unrelated to her business (say, a public service announcement (PSA) about saving sea turtles)? PSAs aren't in the advertiser auction, so a competing pizza shop can't get in, and both sides are symmetric: they both 'saw an ad.'"
Sounded reasonable at first. Then came the catch.
The ad platform's algorithm "optimizes" the PSA too. From the ad platform's side this is obvious. Serving a PSA to random people hurts user experience (if the feed is full of irrelevant ads, people stop scrolling the feed at all), so it'll serve the sea-turtle PSA to "people who'll actually respond to it," say, people who care about environmental issues or watch nature documentaries.
Meanwhile the ad group (users the algorithm pegged as "probably likes pizza") gets her pizza ad as normal.
The result is that each side is populated by completely different kinds of people. For example:
- Ad group: people the algorithm picked as "probably likes pizza" (orders late-night food, uses delivery often)
- No-ad group: people who got the PSA (cares about the environment, watches nature docs)
To put it precisely, the ad effect and the "difference between the two groups of people themselves" get mixed into the sales gap, so you can't isolate the ad effect anymore. The PSA trick dodged the "parking lot problem" but created a bigger new problem in exchange. As long as the ad platform's algorithm "optimizes" every ad it serves, this trap is unavoidable.
Four traps, in summary
| Trap | What happens | Measurement result |
|---|---|---|
| 1. Parking lot | Competitor fills the no-ad side's slot | Effect looks bigger |
| 2. Bunching | Ad budget piles up on remaining users, overexposure | Effect looks bigger |
| 3. Viewability | Shown 10,000 times, actually seen by few | Effect diluted |
| 4. Different crowds | Different ads served to each side, the people themselves differ | Comparison breaks |
Honestly these four are just the headline cases. There are many more. The root cause behind all of them: ad platforms have a huge algorithmic black-box layer in the middle, and neither advertisers nor the ad platforms themselves can precisely predict how it behaves in every situation. The algorithm reacts differently per user, per time of day, per competitive dynamic. So when you dig into "what gets distorted when you run an RCT," new traps keep appearing.
So the ad platforms tried something smarter
After Trap 4 (the PSA problem), smart researchers came up with something cleverer. It's called Ghost Ads, and in 2017 the Google measurement team and outside researchers published a paper on it (Johnson, Lewis, Nubbemeyer 2017).
For the record, this isn't the only proposed answer. It's just the most famous of many methods people have tried for ad-platform measurement over the years. Ghost Ads itself is already a pretty old approach, and in practice ad platforms mix and match this with simpler or more creative methods. Still, it's a good example of "how an ad platform tried to solve this problem," so let's walk through it.
To understand it you first need to know how an ad slot gets filled.
When a user scrolls their feed and an ad slot opens, the ad platform runs a lightning-fast "auction" in under a second. The industry calls this real-time bidding (RTB). Dozens or hundreds of advertisers bid "I'll pay X per impression if you show my ad to this user." The ad platform picks a winner, and that ad shows up.
The user doesn't notice any of this, the feed just looks like a feed. But this auction is running for every slot, every second, across the world. Billions of times a day.
Ghost Ads is a clever trick layered on top of this auction.
For the no-ad side users too, the ad platform still runs the auction, her ad included. But if her ad wins this virtual auction? Instead of actually showing it, the ad platform quietly logs "her ad would have appeared for this user." The user sees a different ad on screen (the auction's second-place ad takes the slot).
The clever part is what comes next. When they analyze, they compare:
- Users on the ad side who "actually saw the ad" (won the auction, ad really shown)
- Users on the no-ad side who "would have seen it" (won the virtual auction, only a log entry left)
Both sides are populated by users who, by the same auction criteria, "qualified to receive her ad." They're similarly "pizza-ish" people, except one group actually saw the ad and the other didn't.
The most important thing in measurement is always "comparing two groups with the same characteristics." The PSA trick (Trap 4) failed because it broke that "same characteristics." Ghost Ads preserves it via the virtual auction.
She didn't actually show the ad to the no-ad group, but she can still pinpoint "people who would have seen it if it had run" and compare them fairly to the people who actually did. Pretty smart. The authors reported that Google Display Network was running this method over 100 million times a day.
But Ghost Ads isn't perfect either
Even this smart method didn't become "the standard" in practice. A few reasons.
- It's operationally heavy. You need a separate system running a virtual auction for every single ad slot, a storage layer for the per-user "would-have-seen" flag, and an analysis pipeline. Server costs, analysts, design time, all nontrivial. Worth running for big advertisers, not for small ones.
- Traps 1 and 2 don't fully disappear. The virtual auction identifies "qualified people," but the ad that actually appears for them is the auction's second-place ad, not hers. And the advertisers bidding for that slot tend to be competing pizza shops targeting the same kind of user (because the ad platform shares the same "likely likes pizza" signal with other pizza advertisers). So the slot is likely to end up with a competitor, and her sales can still leak over. Ghost Ads solved the "clean comparison group" problem, but not the "competitor steals the sale" problem.
Three uncomfortable truths about ad-platform-run measurement
Ghost Ads or any other method, everything we've described, the advertiser (she) can't run alone. The virtual auction happens inside the ad platform, the "would-have-seen" flags can only be set by the ad platform, and the analysis stays inside the ad platform. The advertiser just receives a number like "+12.3% effect" at the end.
Honestly, the advertiser has no choice. Trust the ad platform's measurement or trust nothing.
Three uncomfortable truths follow.
One, big ad platforms only run this for big advertisers. This measurement eats real ad-platform resources. Just think about Ghost Ads: a virtual auction per ad slot, per-user flags, pipelines, analysts. Server cost, headcount, design time, all of it. Honestly, from the ad platform's point of view, running this for small advertisers doesn't move ad-platform revenue much. So usually they'll only offer "we'll run an RCT for you" to advertisers who spend serious money, and small advertisers don't get it at all. They just stare at the dashboard (your sales went up Nx, you got N conversions) and call it a day. For most advertisers, RCT is a pipe dream.
Two, even when the ad platform diligently runs it, nobody knows the exact truth. When the ad platform hands you "+12.3% effect," that number is the result of one specific methodology the ad platform chose. Ghost Ads gives you X, PSA gives you Y, a plain comparison gives you Z, there are dozens more, and the number varies a lot by method. You've already felt this going through the traps one by one. Which method the ad platform used, which assumptions sit inside it, is barely disclosed to advertisers. Which number is closest to the real effect, nobody can say for sure, not the ad platform, not the advertiser, not even measurement researchers. The real effect does exist out there somewhere, we just don't have a method that measures it perfectly.
Three, honestly, the ad platform wants to pick the method that favors it. This isn't a conspiracy theory, it's a normal incentive problem. The ad platform is in the business of selling ads. "Our ads are highly effective" ties directly to ad-platform revenue. There's a natural pull toward picking the method with the bigger number. Whether a specific ad platform actually does this varies, but the structure itself, where the ad platform alone designs, runs, and publishes the measurement, makes this incentive impossible to avoid.
The industry knows this. In November 2025, the Interactive Advertising Bureau (IAB), the ad industry's standards body, released its Guidelines for Incremental Measurement in Commerce Media (official doc). The guidelines sort measurement methods into causal-strength tiers, and classify platform-reported attribution as a weak hybrid proxy, the lowest tier that relies on platform-specific modeling instead of independent control groups. Big ad platforms were sitting at the table that drafted it (Meta, Google, TikTok, Criteo, Walmart, etc.). The ad platforms themselves effectively signed off on "the numbers we provide are the weakest evidence."
The structure, summarized:
- The advertiser can't run RCT alone
- The ad platform can, but only opens it to big advertisers
- Even then, the number depends on the method the ad platform chose
- The ad platform picks the method by itself (and the incentive leans toward its own number)
- The ad platform itself admits "this is the weakest evidence"
So where does that leave us
Last post I kept saying "RCT is the best." Still true. The fact that RCT is the cleanest measurement method hasn't changed.
It's just that in advertising, running that best isn't easy. The advertiser can't run it alone, the ad platform can't measure cleanly through traps like parking lot, bunching, viewability, and different crowds, and even then it only opens up to big advertisers and the result wobbles based on the method. Honestly, nobody knows exactly what the "real ad effect" is. Not the ad platform, not the advertiser.
Back to the pizza shop owner. The same campaign that returned +18%, +7%, -3% across three runs. Why? The four traps we covered (parking lot, bunching, viewability, different crowds), plus other traps we didn't cover, plus the methodology limits of whatever measurement the platform runs, stack in different sizes and different directions and distort each measurement in a different way. On top of that, a small advertiser like her gets a small user sample, so "sample luck" (who happens to get drawn into each run) adds its own wobble. The +18, +7, -3 is all of that piled up: some runs the traps push the effect bigger, other runs the sample luck swings the other way and even goes negative. That's why the ad platform rep's answer, "run it longer, with a bigger sample, and the numbers will stabilize," makes sense. Bigger samples cut the sample-luck wobble, but the traps and methodology limits are still there.
That's not even the end of it. There's a bigger problem.
An actual advertiser doesn't run their campaigns on just one ad platform. She runs 10 pizza ads on platform A, 20 on platform B, 15 on platform C, 5 on platform D, dozens or hundreds of campaigns simultaneously. And since similar targets like "people who like pizza" are something every ad platform chases, one user ends up seeing a mix of ads across multiple ad platforms in a single day. Then they walk into the store and buy pizza. How do you figure out which ad platform's which campaign caused that sale?
If one RCT on one campaign already breaks like this, measuring dozens of campaigns that influence each other at the same time is effectively impossible. Her sales come in as one lump, and nobody knows exactly how to split that credit across all the ads.
Inside the ad platform, multiple traps fire automatically. Outside the ad platform, ad platforms get mixed together. The question "so what's the ad's impact?" doesn't have one clean answer that anyone can hand you. Not the ad platform, not the advertiser.
But she can't just stop marketing. Daily budgets have to be spent, decisions have to be made on where to put the money, winning ads need more spend and losing ones have to be cut. "No answer, let's sit still" isn't an option.
So the industry gave up on "the perfect truth" and built "a shared set of measurement rules everyone can use." Not a complete answer, but if everyone uses the same yardstick, you can at least do relative comparisons like "ad A is better than ad B" inside those rules. The operators who run and measure under this "shared rule" showed up, and they're called MMPs (Mobile Measurement Partners).
An MMP stands outside the ad platforms, on the advertiser's behalf, and applies the same rule to every ad platform (a common standard for attributing a given sale to a given ad) to tally "how much revenue did platform A's ads create, platform B's ads, and so on." They de-duplicate users who overlap across ad platforms (one person saw ads on multiple ad platforms), filter out ad fraud, and handle things like mobile app install tracking. Ad-effect measurement is one of several things MMPs do.
The next post will cover things like this.
- What exactly MMPs do (why they exist, which measurement rule they use (e.g., last-touch), and why that rule became "the industry standard even though it's completely wrong")
- How advertisers actually make decisions off those numbers (not a perfect answer, but what calls do they make with it, how)
Last line. RCT being the best hasn't changed. But in advertising you can't run that best alone. Inside the ad platform you hit four traps, outside the ad platform you need a separate referee, and even the referee doesn't have a perfect answer. That's how ad measurement works: on the path to one clean number, there's barely any safe ground.