Supermartech
Ad Measurement
Measuring Ad Effectiveness Part 6

Would that install have happened without the ad?

In the last post, "Change the rule, and the winner changes", we went through eight rules for deciding how much of one install each ad gets when a user saw several ads before installing. Splitting installs across ads is called attribution, and the share each ad gets is its credit. One install is worth 1 credit, and the ads the user saw split that 1 between them.

We're back with the pizza shop owner. The shop takes orders through its app and runs ads on several platforms. A measurement company called a Mobile Measurement Partner (MMP) tallies which ad drove each install and sends the owner a report. The MMP's default rule is last-touch attribution: the last ad the user tapped before installing takes the full 1 credit. We'll just call it last-touch.

This month's MMP report credits the four ads with 1,400 installs in total (every number here is made up for the example).

  • Apple Search Ads, 600 installs: the ad at the top of App Store results when someone searches the shop's app name
  • AppLovin ads, 400 installs: ads that pop up while someone is using another app, like a game
  • Snapchat ads, 300 installs: ads between posts in the Snapchat feed
  • TikTok ads, 100 installs: video ads between short TikTok videos

The platform names are only there to make each scene easy to picture. The numbers have nothing to do with how those platforms actually perform.

The report has one more line besides those 1,400: installs no ad got credit for, usually called organic. This post asks whether the installs credited to ads really came from ads, so we'll set organic aside and look only at the 1,400.

This month's MMP report: 1,400 ad installs Apple Search 600 AppLovin 400 Snapchat 300 TikTok 100

The owner's read is simple. "Apple Search Ads brings in the most. It must be the most efficient. Let's raise its budget."

Two assumptions under the MMP report

"Let's raise the Apple Search Ads budget" only holds up if Apple Search Ads really brought in the 600 people on the report. But the eight attribution rules from last time, last-touch included, all split credit on top of two assumptions.

Assumption 1. At least one of the ads the user saw influenced the install.

Assumption 2. The install is entirely thanks to the ads.

All eight rules take these two assumptions as given and only decide how the ads split the 1. The numbers in the MMP report do the same. But do the assumptions actually hold? Each one turns into a question:

  • For assumption 1: Did the ad the user saw really influence the install?
  • For assumption 2: Is that one install really all thanks to the ads?

This post follows those two questions one at a time.

Question 1: did the ad really have an effect?

Meet Emma. Emma's friend told her the pizza from this shop is great and she should order through the app. After hearing that, Emma was 100% set on installing it. She searched the app name in the App Store, the owner's Apple Search ad showed up at the top, and she tapped it and installed.

Last-touch gives the full 1 credit for this install to Apple Search Ads, because it's the last ad Emma tapped. But what if the ad hadn't been there? Emma had already made up her mind, so she'd have tapped the shop's app in the regular result right below and installed it the same way. Her friend made this install, not the ad. That's assumption 1 failing. The ad got 1 credit and didn't really do anything for this install.

By the way, an ad that shows up when people search your own brand name is called a brand search ad. With brand search ads, it's common for people like Emma, who would install anyway, to tap the ad. They already know the name and searched for it, so without the ad they'd just tap the regular result below. eBay once stopped its own ads on searches for "ebay" on a few search engines. 99.5% of the clicks those ads used to bring in moved to the regular search results right below. Turning the ads off lost almost no clicks (the paper).

Question 2: is one install all thanks to the ads?

Now Mia. Mia also heard the shop's app name somewhere and searched for it. But unlike Emma, she hadn't made up her mind when she searched. She was only half sure she wanted to install it. The owner's ad sat right at the top of the results, hard to miss, and she tapped it thinking, "Fine, let's get this."

In Mia's case, the ad did have an effect. She was on the fence, and the ad that caught her eye is what made her say "Fine, let's get this." So for Mia, assumption 1 holds.

But is this one install entirely thanks to the ad? Picture it without the ad. Mia was already half planning to install. Without the ad, she might have installed from the regular result below, or she might have moved on to another pizza app and never installed.

The ad did give a hesitant Mia a nudge. But since she might have installed without it, you can't say the ad made this install on its own. So the ad's share is more than 0 but less than the full 1. Maybe it should have gotten about 0.5 instead of 1. Yet the MMP report gives the full 1 to the ad for Mia's install too. Assumption 2 starts from "installs are entirely thanks to the ads," so if Mia saw only one ad, that ad gets the whole 1, no matter what.

So exactly how much of Mia's install belongs to the ad? There's no way to know. On the report, Mia's install and Emma's install look identical: "1 Apple Search Ads install." Both tapped the ad, and both installed. From the outside, you can't tell whether someone installed because of the ad or was going to install anyway. We'll get to why in a bit.

The other way around: ads the report undercounts

Some ads bring in more installs than the MMP report shows. Olivia was scrolling TikTok and saw the shop's app ad. She scrolled past it, but a few days later she was craving pizza, searched the app name she remembered, and installed by tapping the Apple Search ad at the top.

TikTok brought her in, Apple Search got the credit TikTok ad sees the app name Days later searches the name Apple Search taps the ad Install Report: Apple Search, 1 Brought in by: TikTok

Olivia's install shows up in the MMP report as 1 for Apple Search Ads and 0 for TikTok, because the last ad she tapped was the Apple Search ad. But without the TikTok ad, Olivia wouldn't have known the app name, so she wouldn't have searched or installed. The Apple Search ad she tapped last helped a little, but TikTok did most of the work on this install. And TikTok's share in the report is 0.

So the MMP report overcounts some ads (Apple Search Ads, with Emma and Mia) and undercounts others (TikTok, with Olivia). That goes for all four ads and all 1,400 installs, not just Apple Search's 600.

So what the owner really wants to know isn't the report number. It's this: how many installs did this ad actually cause?

An ad "actually caused" an install if, without the ad, the install wouldn't have happened. Let's check with Emma and Olivia. Emma's install would have happened even with Apple Search Ads off, so Apple Search Ads didn't cause it. Olivia's install disappears if you turn off the TikTok ad, so the TikTok ad caused it. From here on, we'll call installs like Olivia's, the ones that would have disappeared with the ad off, installs the ad caused. Installs like Emma's, the ones that would have happened even with the ad off, are installs that would have happened anyway.

Here's what the report looks like through that lens. The report shows 600 Apple Search Ads installs. If all 600 were really thanks to the ad, turning the ad off should cut installs by 600. But people like Emma would install anyway, so the real drop might be, say, just 60. Then 60 of the 600 are installs the ad caused, and the other 540 would have happened anyway. This difference in installs between ad on and ad off, the number of installs the ad caused, is called incrementality.

It can also go the other way, with the ad causing more installs than the report shows. Olivia's install was caused by the TikTok ad but credited to Apple Search Ads. So turning off the TikTok ad can cut more installs than the report's TikTok number.

What about Mia's install? Did the ad cause it, or would it have happened anyway? This is where we get stuck. We can't know what Mia would have done without the ad. To be honest, we could answer for Emma and Olivia only because this post made up their backstories. In real life, nobody's install can answer this question one person at a time.

Why you can't measure one person at a time

To know whether Mia's install came from the ad, you need something to compare against: the same Mia, on the same day, at the same moment, without seeing the ad. If that Mia installs too, it wasn't the ad. If that Mia doesn't, it was.

The problem is that a Mia who didn't see the ad doesn't exist. Mia already saw it. You can't rewind time and replay that moment with only the ad taken out.

What if you show Mia the same search screen the next day, minus the ad? That doesn't work either. The next-day Mia has seen the app name one more time in yesterday's ad, so the app feels more familiar. Plus she's already installed it. Her situation differs from yesterday's Mia, so she can't be the comparison.

Asking Mia directly doesn't work either. You can't ask the thousands of people who installed one by one, and even if you did, you couldn't fully trust the answers. People don't know exactly why they installed an app, and memory gets fuzzy.

You only ever see one world per person Mia, saw the ad → installs (what happened) Mia, didn't see it → ??? (can't be seen) Mia

For any one person, you only ever see one of two worlds: "saw the ad" or "didn't see the ad." Like a parallel universe, the other world stays out of view. So you can't measure an ad's effect one person at a time.

So is there no way? There is. Instead of one person, look at a group of many people.

No one person can stand in for another. Nobody else is exactly like Mia. But things change when you gather lots of people. Mix people thoroughly and split them carefully into two groups of 10,000, and even though each person is different, the two groups as a whole can look almost the same. The share of pizza lovers, the share of people who open delivery apps a lot: both groups end up similar. Then the share of each group that installs the app, the install rate, comes out almost the same too. But a big headcount alone isn't enough. How you split matters, and we'll get to that shortly.

Coins make this easy to see. Flip a coin once and you can't say whether it'll land heads or tails. But flip it 10,000 times, twice over, and both runs come out close to half heads. Each flip goes its own way, but in big batches, the batches look alike.

Now say you split people well like this and got two groups that are alike in everything except whether they see the ad. Age, pizza taste, how often they open delivery apps, even things we don't know about: on average, the two groups match. Show the ad to only one group. Then the install rate of the group that didn't see the ad is the install rate the ad group "would have had without the ad," because the groups differ only in the ad. You can't build a parallel universe for one person, but you can build one for a group. Two groups this alike, where one can show you the other's "without the ad," are called two comparable groups.

Then the ad's effect is the gap between the two groups' install rates. Say 5 in 100 (5%) of the ad group installed and 1 in 100 (1%) of the no-ad group did. The ad got 4 more people per 100 to install. You'll never know whether Mia herself installed because of the ad, but you can know how much the ad raised the install rate.

We'll also write that rate as "per 10,000 people." 4 per 100 is 400 per 10,000. The 10,000 doesn't matter in itself. It's just an easier unit for reading the rate.

Now flip it around. What if the ad group has lots of pizza lovers and the no-ad group has lots of people who don't care much for pizza? Even if you get the same 5 vs. 1 per 100, you can't tell whether the gap comes from the ad or from loving pizza in the first place. Without the ad, the pizza-loving side would have installed more anyway. Here the no-ad group can't be the ad group's parallel universe. These two groups aren't comparable.

This is the most important idea in the post, so here it is once more.

You can't measure an ad's effect from one person. But you can by comparing a group that saw the ad with a group that didn't. The catch: the two groups have to be alike in everything but the ad, which is to say comparable. Every measurement method from here on differs only in how it builds those two comparable groups.

How do you build two comparable groups?

The hard part is getting two comparable groups. There are two broad routes.

  • Experimental methods: before the ad runs, the owner deliberately splits people into two comparable groups, shows the ad to one, and keeps it off for the other.
  • Observational methods: when you can't run an experiment, you take the data that piled up after the ad ran, pick similar people from those who saw it and those who didn't, build two groups, and compare them.

Among experiments, the most trustworthy is splitting people at random. If that's hard, you split regions or dates instead of people. If experiments themselves are hard, you go observational. We'll go in that order.

1. Splitting people at random

The best way is to split people with a coin flip. Show the ad to heads, and not to tails. This is a randomized experiment, also called a randomized controlled trial (RCT). The side you hold back from the ad is the holdout.

Why a coin? If the owner picks, the pick gets lopsided. Usually it's "show it first to people who look likely to buy." Then the ad side is packed with people who'd install anyway, and the ad looks bigger than it is. Pick the other way, "skip the regulars, they'll install anyway," and the effect looks smaller. When people pick, you can't even tell which way or how far the result got bent. A coin knows none of that. So people who'd install and people who wouldn't, pizza lovers and pizza haters, land evenly on team heads and team tails. That goes for traits you can see in data, like age or income, and for traits nobody records, like that day's mood. With enough people, the two groups end up nearly identical except for the ad. And since both groups run at the same time, outside stuff like weather or a competitor's promo hits both sides equally. One coin gives you two comparable groups.

Here's an example. Take 20,000 people the platform picked to show the owner's ad to, split them in half at random, and show the ad to only one half, 10,000 people, for a month. Count every install from each group, no matter which ad the MMP report credited.

  • 500 of the 10,000 who saw the ad installed. That's 5 in 100.
  • 100 of the 10,000 who didn't see it installed too. That's 1 in 100.
Split 20,000 at random, show the ad to one half Saw ad: 10,000 5 in 100 500 installed No ad: 10,000 1 in 100 100 installed Gap of 4 per 100 = 400 installs the ad caused

Even the no-ad side had 1 in 100 install. Whether from a friend's tip or interest they already had, that's how many people install without the ad. The two groups match in everything but the ad, so the ad side must have had about 100 such people per 10,000 too. That means the ad side's 500 break down like this:

  • Would have installed without the ad: 100
  • Installed because of the ad: 400

What you couldn't measure from one person, you can measure once you gather a group. The ad added about 4 installs per 100 people, and since 10,000 people saw it, the ad caused 400 installs. If 20,000 had seen it, that would have been 800. What the experiment tells you is the rate. The count depends on how many people saw the ad.

The weak spot of a randomized experiment is that an advertiser can't really run one alone. The platform decides who sees the ad, and platforms don't show ads at random. They pick people who look likely to install, and they make money when those people click and install (more on this in the observational section). So the owner can't flip the coin on people directly. Instead, big platforms will run this kind of experiment for advertisers who spend a fair amount and ask for it.

But platform-run experiments have two drawbacks. First, the advertiser can't check the math. The platform tells you the result, like "the ad caused 400 installs," but usually doesn't hand over the raw data, like who was in which group and how it was calculated. The advertiser has little choice but to trust the platform's grade on its own ads, along with its word that it ran the test properly. Second, a platform only measures its own ads. A TikTok-run experiment only tells you about TikTok ads. To measure all four ads, you'd need a separate one on each platform (we covered this in post 3, "Honestly, even the platform doesn't know your ad's real impact").

2. Can't split people? Split regions or dates

If you can't split people, split something else. What the owner can decide directly is "which regions get the ad" and "which days get the ad."

Splitting by region. The simplest way is to find two similar cities. If City A and City B are alike in population, weather, and ordering habits, turn the ad off in City A only, leave City B alone, and compare app installs in the two cities. This is a geo experiment, also called a geo lift test.

Having a city that closely matches City A is great, but such a city is often hard to find. In that case, you blend several cities into a "synthetic City A." Say that over the few months before you turned the ad off, a blend of 50% City B, 30% City C, and 20% City D moved up and down almost exactly like City A's installs. If one week City B had 30 installs, City C 20, and City D 40, then 15 + 6 + 8 = 29 is that week's synthetic City A. A comparison built by blending several regions like this is called a synthetic control.

Ad off in City A only Ad off Synthetic A Real A Caused by ad Ad on Real A Synthetic A (B 50% + C 30% + D 20%)

While the ad was on, the real City A and the synthetic City A moved almost on top of each other. After the ad went off in City A, the real City A dropped below the synthetic one. That gap is the installs the ad caused in City A.

Geo experiments have limits too.

  • No two cities are exactly alike. Even the closest match differs a bit. The same goes for a synthetic City A built from several cities. It only matches the trend from before the ad went off, so if a festival happens only in City A during the test, or a competing app runs a promo, the result wobbles.
  • You don't know how the platform runs ads in each city. Turn the ad off in City A and the platform may spend the leftover budget in City B. Then City B's installs rise above normal and the ad's effect looks bigger. Some users also turn off location, so splitting ads cleanly along city lines is hard.

Splitting by date. Flip a coin each day to decide whether the ad runs that day, then compare installs on ad-on days and ad-off days. This is a switchback test. Why flip a coin every day? Why not just turn the ad off for a week and compare last week, with the ad on, to this week, with it off? Say last week averaged 80 installs a day and this week averaged 75. Are those 5 missing installs the ones the ad caused?

Not necessarily. Last week and this week differ in more than the ad. Say it rained all week this week and more people went looking for delivery apps. Turning the ad off cost 10 installs, but the rain added 5 back, landing at 75. Then the ad actually caused 10 installs, but the numbers only show 5 fewer, so the ad looks smaller than it is.

Or say a competing pizza app handed out half-off coupons this week and people went there. Turning off the ad cost 10 and the coupons cost another 10, landing at 60. Now the whole 20-install drop looks like the ad's doing, and the ad looks bigger than it is.

Two different weeks differ in weather, events, competitors, and so on, so they're usually not two comparable groups.

That's why you don't switch off a whole week, and instead flip a coin each day to decide whether the ad runs. Rainy days and coupon days then land evenly on ad-on and ad-off days. On the same principle as flipping a coin on people, the on days and off days become two comparable groups. Say you did this for three months, and ad-on days averaged 22 installs while ad-off days averaged 20. Then the ad caused 2 installs a day. DoorDash measured its app store search ads this way. It couldn't split people at random or split regions precisely, so it randomly picked each day whether the ads ran and compared app downloads between on days and off days (DoorDash engineering blog).

Flip a coin each day: ad on or off First two weeks; each cell shows that day's installs Mon Tue Wed Thu Fri Sat Sun Wk 1 22 20 23 rain 25 rain 21 17 coupon 19 coupon Wk 2 19 23 21 22 20 22 20 Ad-on day Ad-off day On days: 22 avg Off days: 20 avg On days get 2 more installs a day = installs the ad caused

Switchback tests have limits too.

  • They miss people who install days after seeing the ad. Say someone sees the ad on Monday, an ad-on day, and installs on Wednesday. If Wednesday happens to be an off day, the install counts toward the off day, even though the ad drove it. The more people like this, the more the off days get padded, and the gap between on days and off days comes out smaller than it really is. DoorDash, describing its own test, said the method fits cases where people install right after tapping the ad.
  • Switching ads on and off every day is hard. First, it's a lot of hands-on work. Someone has to turn the ad on or off every single day. There's a platform problem too. As the platform runs your ad, it gradually learns who to show it to so people install (this is usually called optimization). Turn the ad off and on again, and what it learned can get scrambled, or it may have to start learning from scratch. Then ad-on days perform worse than usual, and the ad's effect can come out smaller than it really is. Last, it takes a long time. An experiment that splits people can gather thousands of people in a few days, but in a switchback test, one day is one data point. Collecting enough on days and off days to compare can take months.

3. Can't experiment? Build comparable groups from existing data

When you can't run an experiment, you use data that piled up after the ad already ran. Instead of deliberately switching the ad on and off, you build two groups that are as comparable as possible from data on what already happened, and compare them. This is the observational method from earlier. First, the easiest version: what happens if you just split people into those who saw the ad and those who didn't, and compare?

Take the same ad we measured with the randomized experiment, and this time measure it with existing data and no experiment. You just split people into those who actually saw the ad and those who didn't, and compare install rates. 500 of 10,000 people who saw the ad installed, and 20 of 10,000 who didn't. The gap is 480 per 10,000. Same ad, but the randomized experiment said 400. Why is this one bigger?

Because the two groups weren't comparable from the start. Platforms don't show ads to just anyone. To make sure the owner's paid ad turns into installs, they pick people who look likely to install, like someone who's liked lots of pizza photos. So the group that didn't see the ad is full of people the platform filtered out as "unlikely to install," people who weren't that into pizza apps to begin with.

Put simply, the ad group is packed with people who'd install anyway, and the no-ad group is packed with people who wouldn't. Even without the ad, the two groups' installs would already have differed. A plain comparison credits that built-in gap to the ad too, so the ad looks bigger than it is. The 80 between 480 and 400 is exactly that share. Not a fair comparison.

So people came up with matching. For each person who saw the ad, you pick the most similar person among those who didn't and pair them up. Then you keep only the paired people and form two new groups. Each pair is similar, so the two new groups are similar too. Then you compare those two groups.

But what does "similar" mean? Say you need to find a match for Alex, who saw the ad, between Ben and Chris, who didn't.

Ad Age Delivery app opens per week
Alex Saw it 27 5
Ben Didn't 41 5
Chris Didn't 28 1

By age, Chris is closer. By delivery app use, Ben is. With just two criteria, it's already hard to say who's more similar. And in practice it doesn't stop at two. You have to match every trait that could affect whether someone sees the ad and whether they install. Miss one and you're left with the same problem as the "group of nothing but pizza lovers" from earlier. Neighborhood, phone model, recent orders, the time of day they use apps: there's so much to cover that the list of criteria quickly grows into the dozens.

So you hand the job to a computer. Feed in age, delivery app opens, and everything else, and the computer looks at the existing data and squeezes it all into one score between 0 and 1. The more likely the platform would show this person the ad, the higher the score. This is the propensity score. If Alex scores 0.62, Ben 0.58, and Chris 0.21, then Ben, whose score is close to Alex's, is the match.

Everyone else gets paired up the same way, by similar scores, and goes into the two groups. People with no match are left out of the comparison.

Pair people with similar scores into two groups Saw the ad Didn't see it Dana 0.35 Fiona 0.33 Alex 0.62 Ben 0.58 Evan 0.81 Gina 0.79 Chris 0.21 Dropped Compare only the matched pairs

This method is called propensity score matching (PSM). Say comparing the two groups built this way gives a gap of 430 per 10,000.

Same ad, different answer by method Plain comparison 480 (+80) Matching (PSM) 430 (+30) Randomized (RCT) 400 Overstated Installs the ad caused, per 10,000 people

That's much closer to the randomized experiment (400) than the plain comparison (480) was. The overstated share shrank from 80 to 30. Still not a perfect match, though. Why?

A propensity score can only be built from information in the owner's data. Traits in the data, like age or delivery app opens, can be matched. Traits that aren't in the data can't.

Take income. The owner's data has no income info. But the platform, using its own data, might show the ad more to people who look high-income and likely to order delivery often. Then even Alex and Ben, with close scores of 0.62 and 0.58, might differ: Alex, who saw the ad, has a high income and orders delivery a lot anyway, and Ben, who didn't, doesn't. With many pairs like this, if the ad group installs more, you can't separate whether that's the ad or the fact that they order more delivery anyway. The same problem as the "group of nothing but pizza lovers" is still there, a little, in the part the data can't see. That leftover is the 30 between 430 and 400.

This isn't just a matching problem. It's a limit of observational methods as a whole. Whatever the approach, observational methods can only match traits in the data. A randomized experiment doesn't have this worry, since the coin spreads every trait evenly, in the data or not. In fact, a study that measured 663 ads on a large social platform with both randomized experiments and several observational methods found that the observational methods often put the ad's effect higher (Gordon, Moakler, Zettelmeyer 2023).

That doesn't make observational methods useless. Experiments are so hard to run in advertising that observational methods are often the realistic choice. You don't have to turn ads off, you can try it right away with data you already have, and it gets much closer than a plain comparison (430 vs. 480). As long as you know the number may run a little high, it beats measuring nothing.

Here are the methods side by side.

Method How it builds two comparable groups How much to trust it
Randomized experiment Splits people with a coin flip The most trustworthy. But you need the platform's help
Geo experiment Picks similar cities and turns the ad off in one Fairly trustworthy. But the cities still differ a bit beyond the ad, and results wobble if the platform shifts budget to other cities
Switchback test Flips a coin each day to pick ad-on and ad-off days Fairly trustworthy. But it misses people who install days after seeing the ad, and switching daily disrupts the platform's optimization
Observational Pairs similar people in existing data to form two groups No experiment needed, and it works right away on data you have. But it can't match differences that aren't in the data, so the ad's effect can run high

So how should the credit be split?

Now let's revisit the MMP report with numbers measured by these methods.

Think back to the randomized experiment. It measured 400 installs the ad actually caused. Say the MMP report gave that ad 500 credits for the same period. Then only 0.8 of each credited install was caused by the ad. We'll call that 0.8, the number measured by an experiment or observational method divided by the report number, the calibration ratio.

Calibration ratio = installs the ad caused, measured by an experiment or observational method ÷ the credit that ad got in the report for the same period

A ratio of 1 means the report is accurate. Below 1, the report overcounts. Above 1, it undercounts. Once you have it, you can multiply the report number by it to estimate the installs the ad caused, as in 500 × 0.8 = 400.

Say the owner measured each of the four ads with whatever method was possible and got the numbers below. Divide each ad's measured installs caused by its report credit, and you get its calibration ratio.

Ad Report credit Measured installs the ad caused Calibration ratio
Apple Search Ads 600 60 60 ÷ 600 = 0.1
AppLovin 400 200 200 ÷ 400 = 0.5
Snapchat 300 240 240 ÷ 300 = 0.8
TikTok 100 150 150 ÷ 100 = 1.5

Apple Search Ads has the lowest ratio, 0.1. That means only 60 of its 600 report installs were caused by the ad. Lots of people who'd install anyway, like Emma, were mixed into the report.

TikTok's ratio is 1.5. That means turning off the TikTok ads cuts installs by 150, whatever the report says. That's more than TikTok's report credit (100). It's because installs from people like Olivia, who found the app on TikTok and installed through a different ad, drop too.

Now back to this month's 1,400 report installs. Following assumption 2, the last-touch report split all 1,400 among the four ads. So how many of the 1,400 did the ads really cause, and how many would have happened anyway? You can estimate it in two steps.

  1. Add up the installs the four ads really caused. From the table, 60 + 200 + 240 + 150 = 650. That's roughly how many installs the four ads caused between them.
  2. Subtract 650 from 1,400. The remaining 750 wouldn't have disappeared no matter which ad you turned off. In other words, they're installs that would have happened anyway. People like Emma.

To sum up, the last-touch report had to split all 1,400 among the four ads one way or another. But the ads actually caused about 650, and the other 750 would have installed even without any ad. The report still counted those 750 as the ads' share.

Same 1,400 installs, but 750 weren't caused by ads Last-touch report 600 400 300 100 Installs the ad caused 200 240 150 750 Apple Search 600 → 60 AppLovin 400 → 200 Snapchat 300 → 240 TikTok 100 → 150 Would happen anyway 0 → 750

These 750 are easy to confuse with the organic installs we set aside at the start, but they sit in a different spot on the report.

  • Organic: installs the report marked as "not from ads" from the start. They mix people who installed without seeing any ad and people who saw an ad but weren't tracked, so no record was left.
  • The 750: installs inside the 1,400 the report credited to ads. Like Emma, these people tapped an ad, but they'd have installed even without one.

So the 750 are installs the report credited to ads that the ads didn't actually cause.

Finally, back to the owner's original call. This month's ad spend was $1,800 on Apple Search Ads, $2,400 on AppLovin, $3,000 on Snapchat, and $1,500 on TikTok. Ad spend divided by installs, the cost of getting one install, is the cost per install (CPI). Let's compute CPI two ways: once dividing spend by report credit, and once dividing it by installs the ad caused, the installs that would have disappeared with the ad off.

Ad CPI by report credit CPI by installs the ad caused
Apple Search Ads $1,800 ÷ 600 = $3 $1,800 ÷ 60 = $30
AppLovin $2,400 ÷ 400 = $6 $2,400 ÷ 200 = $12
Snapchat $3,000 ÷ 300 = $10 $3,000 ÷ 240 = $12.50
TikTok $1,500 ÷ 100 = $15 $1,500 ÷ 150 = $10
Cost per install (CPI): the ranking flips By report Caused by ad Cheapest first ↓ Apple Search $3 Apple Search $30 AppLovin $6 AppLovin $12 Snapchat $10 Snapchat $12.50 TikTok $15 TikTok $10

By report credit, Apple Search Ads is the cheapest and TikTok the most expensive. But by installs the ad caused, meaning what each install the ad really brought in cost you, TikTok is the cheapest and Apple Search Ads the most expensive (in our example, anyway). The owner's "Apple Search Ads is efficient, let's raise its budget" deserves a second look after this table.

That doesn't mean you should turn off Apple Search Ads tomorrow and pile the budget onto TikTok. Ads usually cost more per install the more you spend. So if you raise TikTok's budget a lot, its current $10 CPI may not hold. Shift budget a little at a time, and re-measure after each shift.

What this means in practice

Next to each ad's number in the MMP report, add one more line: "installs the ad caused," the installs that would have disappeared with that ad off. The same 1,400 installs look completely different. But the calibration ratio isn't a measure-once number. It shifts whenever the creative, the budget, or the season changes, or when the platform changes how it picks who sees the ad. So don't trust one measurement forever. Re-measure every few months, and re-split with the new ratios.

Every method so far needed one of two things. Experiments need a group you deliberately hold back from the ad, and in advertising that's hard. Observational methods pair people who saw the ad with people who didn't, so they need a per-person record of who saw the ad and who installed. But sometimes iPhone users block app tracking, and with something like TV, there's no record of who saw it at all. In those cases, you estimate the ad's effect from how ad spend and total installs moved together, week by week or month by month. That's called marketing mix modeling (MMM), and it's the next post.