Supermartech
Ad Measurement
Measuring Ad Effectiveness Part 7

Your report can't see these ads. Are they really not working?

We're back with the pizza shop owner. A quick recap first.

The owner takes pizza orders through an app and runs four ads to get more installs: Apple Search Ads, which sit at the top of App Store search results; AppLovin ads, which show up while people use other apps like games; and ads on Snapchat and TikTok. A measurement company counts which ad led to each install and sends a report every month. Companies that do this are called Mobile Measurement Partners (MMPs). An MMP usually gives the whole install to the last ad the user tapped before installing.

In the last post, "Would that install have happened without the ad?", we saw that the installs a report credits to an ad and the installs the ad actually caused aren't the same number. The report said Apple Search Ads brought in 600 installs, for example, but plenty of those people would have installed the app anyway. So instead of going by the report, we measured installs caused by the ad, meaning the installs that would have disappeared if that ad had been off. There were two ways to measure them:

  • Experiments: deliberately split people at random into a group that sees the ad and a group that doesn't, then compare installs between the two.
  • Observational methods: from records you already have, pair up similar people who did and didn't see the ad, and compare them.

The results looked like this (every number in this post is an illustration).

Ad Installs in the report Installs caused by the ad (experiments and observation)
Apple Search Ads 600 60
AppLovin 400 200
Snapchat 300 240
TikTok 100 150
Total 1,400 650

Two ads stand out: Apple Search Ads and TikTok.

  • Apple Search Ads looked far bigger in the report than it really was. The report showed 600 installs, but the ad caused only 60, a tenth of that. Many people who search the app's name in the App Store were already planning to install it before they searched. If an ad sits at the top of the results, they tap it and install. If there were no ad, they'd tap the regular result right below it and install anyway. The install happens either way, but because they tapped the ad, the report credits it to Apple Search Ads.
  • TikTok looked smaller in the report than it really was. The report showed 100 installs, but the ad caused 150. Say someone first hears about the app from a TikTok ad but doesn't install it on the spot. A few days later they search the app's name in the App Store and tap the Apple Search Ads result at the top. TikTok is what got them to install, but the report gives the install to the last ad tapped, so it goes to Apple Search Ads.

So report numbers lean toward whichever ad was tapped right before the install. We'll come back to this table later.

The October report looks off

In June this year, the owner put up subway billboards. In August, they also started a YouTuber sponsorship.

  • Subway billboards: big posters with the app's name and a pizza photo next to the exit stairs at six subway stations near the shop, with a QR code in the corner that takes you straight to the app. All six cost $200 a week, or $800 for a four-week month.
  • YouTuber sponsorship: a food YouTuber with lots of subscribers posts one video a month eating the owner's pizza. The sponsorship costs $1,800 a month. The video description also has a link to install the app. It's not a plain Urchin Tracking Module (UTM) link but a tracking link issued by the MMP. Web marketers often add UTM tags to the end of a link to mark where traffic came from. But an app install goes through the App Store, and those tags usually get lost along the way, so a plain UTM link often can't tell who installed the app through it. With an MMP tracking link, the MMP first records who tapped the link, then connects that tap to the install. So anyone who taps it and installs shows up in the report under the YouTuber.

The billboard spots the owner picked were sold by weekly bidding, because another advertiser wanted the same spots. From June through August, the owner bid the same price every week. In weeks the other advertiser bid higher, the owner lost and had no billboards up. In September the other advertiser dropped out, and the owner could run the billboards every week. The table below lays out May through October. The four original ads (Apple Search Ads, AppLovin, Snapchat, TikTok) ran on roughly the same monthly budget the whole time. Only the subway billboards and the YouTuber sponsorship changed.

Month Subway billboards YouTuber sponsorship Notes
May None None About 2,000 installs for the month
June Started. Up only in weeks 2 and 4 (won the bid) None
July Up only in weeks 1 and 3 (won the bid) None Rival app's half-price coupon in week 1
August Up only in weeks 2 and 4 (won the bid) Started. First video in week 3
September Up every week, but at only 3 of the 6 stations (about $400 that month) One video a month 3 stations left empty on purpose to test the billboards (more later)
October All six up every week One video a month 2,200 installs for the month

When October ended, the owner opened the October MMP report.

October MMP report Apple Search Ads 650 AppLovin 400 Snapchat 300 TikTok 100 Subway billboards 10 YouTuber 20 Organic 720 Only QR scans (10) and link taps (20) got counted

Subway billboards: 10. YouTuber: 20. The owner started to worry. "I'm paying $800 a month for the billboards and getting 10 installs. That's $80 an install. Should I take them down?"

The last row in the chart, "Organic 720," is where the report puts installs it couldn't trace to any ad. Installs with no record of an ad tap end up here, like someone who heard about the app from a friend and installed it straight from the App Store. Reports call these installs organic.

Add up the numbers, though, and something's strange. October's report adds up to 2,200 installs. In May, before the billboards went up, it was about 2,000. The four original ads (Apple Search Ads, AppLovin, Snapchat, TikTok) ran about the same as in May. To keep things simple, let's also assume the installs those four ads drive didn't change much as the seasons moved from May to October. Then the extra 200 most likely came from the new subway billboards and the YouTuber. But in the report, the billboards and the YouTuber add up to just 30 (10 + 20). Where did the other 170 go? Compare May's and October's reports line by line.

Report line May (before the billboards and the YouTuber sponsorship) October Increase
Organic 600 720 120
Apple Search Ads 600 650 50
Subway billboards None 10 10
YouTuber sponsorship None 20 20
Total 200

Of the extra 200, only 30 (10 and 20) were credited to the billboards and the YouTuber. The other 170 showed up in the organic and Apple Search Ads lines.

Start with the billboards to see why. People who scanned the QR code at a billboard and installed on the spot get credited to the billboards. That's the 10. But not many people scanned the owner's QR code. Most saw the billboard on their commute, walked past, and searched the app's name in the App Store that evening. Nothing records that these people saw a billboard. So if they tapped the Apple Search Ads result at the top of the search results, the install goes to Apple Search Ads. If they tapped the regular result below it, the install goes to organic.

The YouTuber works the same way. People who tapped the tracking link in the description while watching and installed right away get credited to the YouTuber. That's the 20. Here too, for every person who tapped the link, many more watched the video, moved on, and searched the app's name a few days later when they got a pizza craving. Like the billboard installs, these land in Apple Search Ads or organic.

And the 10 and the 20 in the report don't even count everyone who scanned the QR code or tapped the link. As the note on organic above explains, people who declined tracking on their iPhone can end up in organic even if they installed through the QR code or the link. The 10 and 20 only count the QR and link installs that tracking managed to connect.

So the 120 extra organic installs are a mix of two groups:

  • People who saw an ad, then searched for the app on their own: they saw a billboard or the video, searched the app's name later, and tapped the regular search result instead of the search ad at the top. They never tapped the QR code, the video's link, or a search ad, so the report has no ad record for them.
  • People who scanned the QR code or tapped the link, but whose tracking broke: they declined tracking on their iPhone, so the install never got connected to the billboards or the YouTuber.

In real life, organic installs can also rise on their own for reasons unrelated to ads, like word of mouth or a jump in the App Store rankings. To keep this post simple, we'll say all 120 extra organic installs since May came from the subway billboards and the YouTuber sponsorship.

How many installs did the subway billboards and the YouTuber sponsorship really bring in, and which report lines are they hiding in? In real life, nobody can know the right answer. For this post, though, we'll set the right answer up front.

Ad Really brought in Credited to it (QR, link) In the Apple Search Ads line In the organic line
Subway billboards 80 10 20 50
YouTuber 120 20 30 70
Total 200 30 50 120

In other words, of the 200 installs the billboards and the YouTuber brought in, only 30 were credited to them by name. Another 50 got mixed into the Apple Search Ads line and 120 into the organic line. That's why, comparing May and October, the Apple Search Ads line grew by 50 and the organic line by 120.

Of course, the owner can't see this table. All the owner has is the October MMP report from earlier, and that report alone can't say how many installs the billboards really brought in. The rest of this post is about how the owner fills in the "really brought in" column.

Declined tracking isn't only a problem for the billboards and the YouTuber. People who install by tapping a TikTok or Snapchat ad can also show up as organic if they decline tracking on their iPhone. Even installs from a direct ad tap slip past the report fairly often.

The method that measures installs, including the ones reports miss, is Marketing Mix Modeling (MMM), and it's what this post is about. MMM works for ads people tap to install right away, like TikTok and Snapchat, too. But this post sticks to the cases where MMM is needed most: subway billboards and YouTuber sponsorships. Spending on these ads is growing fast, and they're also the hardest to measure with a report.

Experiments and observational methods both struggle here

Could the two methods from earlier, experiments and observational methods, measure the subway billboards and the YouTuber? First, let's pin down exactly what each one does.

An experiment splits people ahead of time into those who'll see the ad and those who won't. For example, you ask TikTok to split 20,000 people in half with a coin flip and show the owner's ad to only one group of 10,000. A month later, you compare how many people in each group installed the app. The difference is the installs caused by the ad, since the two groups are alike in everything except whether they saw the ad.

To do this with subway billboards, you'd have to flip a coin for everyone walking through the station and let only the people who got heads see the billboard. That's impossible. Anyone who walks past a billboard sees it. Same with the YouTuber's video. TikTok picks who sees a TikTok ad, but anyone who finds the YouTuber's video can watch it. The owner has no way to say "you watch, you don't."

Even if you could split people, another problem remains. To compare installs between the two groups a month later, you'd need to know whether each person who walked through the station installed the app afterward. In a TikTok experiment, TikTok knows who it showed the ad to, so it can match those people to installs. But people walking through a station don't leave a name or a phone number, so there's no way to follow who installed later.

If you can't split people, you can split stations. Take the billboards down at three of the six stations and see how much installs drop in the neighborhoods around them. An experiment that turns ads off in some areas and compares is called a geo experiment. The owner actually ran one in September (more on that later). It has limits, though. Three billboard spots have to sit empty for a whole month. And since the only thing turned off was the billboards, the only thing you learn is how many installs the billboards bring in. The YouTuber sponsorship would need its own measurement, and a geo experiment won't work there. The video can be watched from any neighborhood, so you can't block it in just one area.

Observational methods compare without running an experiment, using records you already have. People who saw an ad and people who didn't are usually a bit different to begin with. So you pair up similar people, say by age or by how often they use delivery apps, to even out those differences, then compare install rates. But to make pairs, you at least need to know "this person saw the ad, that person didn't."

With TikTok ads, TikTok keeps a record of who it showed the ad to, so you know. With the subway billboards and the YouTuber, you don't. The only records come from people who scanned the QR code or tapped the link, and only if they allowed tracking on their iPhone. Some of the people without records never saw a billboard. But plenty of others saw one and walked past, or watched the video and searched for the app days later. Try to build a "didn't see the ad" group and you can't tell how many people in it actually did see the ad. However carefully you pair similar people, the partner might have seen the billboards, so the comparison falls apart.

So the owner needs a method that:

  • doesn't require splitting people,
  • doesn't require knowing everyone who saw an ad, and
  • doesn't require turning ads off on purpose.

There's one more thing. To find out how many installs the billboards and the YouTuber really brought in, you get a more accurate answer by looking at all four original ads too, not just the billboards and the YouTuber. That's because installs the billboards and the YouTuber brought in can leak into other ads' lines. The table above kept things simple by sending them only to the Apple Search Ads and organic lines, but in reality they can land on any line. Say someone sees a billboard on the way to work and thinks, "I'll install that app after work." That evening, scrolling TikTok, they happen to see the owner's ad, remember, and tap it to install. The billboard is why they decided to install, but the report puts them on the TikTok line, since TikTok was the last ad they tapped. If they'd tapped an AppLovin ad while playing a game instead, they'd be on the AppLovin line. The owner doesn't know how many of the billboard and YouTuber installs leaked into which lines. So all six ads, the four original ones (Apple Search Ads, AppLovin, Snapchat, TikTok) plus the subway billboards and the YouTuber sponsorship, have to go on the table together, and October's 2,200 installs have to be split again from scratch. That's how you can move installs mixed into other lines, like Apple Search Ads or organic, back to the billboards or the YouTuber that really brought them in.

You don't know everyone who saw the ads, but you do know weekly spend and weekly installs

This is where MMM proper starts. In one line: MMM splits each week's installs into "installs that would have come in even without ads" and "each ad's share." You don't need to know who saw an ad. A ledger is enough. Let's work through it with the owner's ledger.

The owner does have one record with nothing missing: the shop's ledger. Every week, the owner writes down things like this. This is the ledger for October, week 1.

October week 1 ledger Number Where it comes from
① Ad spend
Apple Search Ads $440 Ads dashboard
AppLovin $630 Ads dashboard
Snapchat $720 Ads dashboard
TikTok $350 Ads dashboard
Subway billboards $200 (all six up) Contract with the ad company
② How much the YouTuber's video was seen
New views that week 80,000 YouTube video stats
③ Result
Installs that week 535 App Store

① is money the owner pays directly, so it's exact. ② is for the YouTuber. For each ad, you just write down the number that best tracks how much that ad reached people that week in the ledger. That's what lets you line it up against installs going up and down. For most ads, that number is spend: spend more this week and the ad gets seen more, and installs move with it. The YouTuber sponsorship is different. The sponsorship is a flat $1,800 a month, so if you split it into $450 a week, every week shows the same number whether the video was watched a lot or a little. So for the YouTuber sponsorship, the ledger records how much the video was seen that week, meaning views, instead of spend. You can see views on YouTube. Views only ever pile up, though, so you write down just the new views that week. And ③, the install count, is fully visible in the App Store, no matter who saw which ad. The ledger has both ad spend and weekly installs in full, whether or not anything was tracked.

So you can change the question. Instead of "who saw the subway billboards and installed?", ask "how many more installs did we get in weeks the billboards were up than in weeks they weren't?" For the YouTuber, it becomes "how many more installs did we get in weeks with lots of new views than in weeks with few?" Instead of following people one by one, you line up weekly ad spend next to weekly installs.

One thing to watch out for. Don't compare just two weeks, last week with the ad up and this week with it down. If it rained all this week, or a rival app ran a coupon, you can't tell whether installs dropped because the ad came down or because of the weather or the coupon. So you look at many weeks, not two. And you add a "special events" column to the ledger for anything other than ads that might have moved installs that week, with entries like "July week 1: rival app ran a half-price coupon" or "September week 3: made the App Store top 10."

Start with the period when the subway billboards were the only new ad, before the YouTuber sponsorship began: the eight weeks of June and July. As the monthly table showed, the billboards were up only in weeks the owner won the bid, and the four original ads got the same spend every week. That means the only thing that changed from week to week was the billboards. Also, someone who sees a billboard might install that week, but they might also install the next week or the week after. To keep it simple here, we'll assume every install the billboards brought in happened in the same week the billboards were up. We'll look at installs that come weeks later separately.

Week Subway billboards Special events Installs that week
June week 1 Not up None 488
June week 2 Up None 512
June week 3 Not up None 491
June week 4 Up None 509
July week 1 Up Rival app ran a half-price coupon 470
July week 2 Not up None 490
July week 3 Up None 511
July week 4 Not up None 489
Weekly installs with and without subway billboards 470 490 510 Billboards up: avg 510 Billboards down: avg 490 488 Wk 1 512 Wk 2 491 Wk 3 509 Wk 4 Rival coupon 470 Wk 1 490 Wk 2 511 Wk 3 489 Wk 4 June July Subway billboards up Subway billboards down Vertical axis starts at 460

The installs in the table aren't the billboards' share. They're all the installs that came in that week. Setting aside July week 1, the coupon week, let's go in three steps.

  1. The four weeks without billboards (June weeks 1 and 3, July weeks 2 and 4): 488, 491, 490, and 489 installs, about 490 on average. That's roughly how many installs come in each week without billboards. It includes installs from the four original ads plus people who would have installed with no ads at all.
  2. The three weeks with billboards (June weeks 2 and 4, July week 3): 512, 509, and 511, about 510 on average.
  3. The difference: 510 − 490 = 20. Since nothing else changed, we can read those 20 as the installs the billboards brought in each week. The billboards that got only 10 installs a month in the October report may really have been bringing in about 20 a week.

Even the no-billboard weeks differ by a few installs, 488 here and 491 there, because people's lives vary a little week to week, like a few more or fewer people happening to want pizza. Averaging many weeks, instead of looking at one, smooths that out.

The table also shows why July week 1 was set aside. The billboards were up that week, yet installs came in at 470, the lowest of all. A rival pizza app ran a half-price coupon that week, and people went there. Include July week 1 in a plain average and the billboard-week average drops to about 500, shrinking the billboard effect to about 10. Something that has nothing to do with the billboards ends up eating into their share.

That's why the ledger needs a special events column. You write down whatever moved installs but wasn't an ad: a rival's coupon, a week the app suddenly climbed the App Store charts, a week of nonstop rain, a week a review of the app spread online. Once a few coupon weeks pile up, you can use the same method to work out that installs drop by about 40 in coupon weeks. Then you don't have to throw July week 1 out. You can explain it as "the billboards added 20 and the coupon took away 40."

Some ups and downs repeat at the same time every year. That's seasonality. People order more pizza at the end of the year, for example. Say December brings about 30 extra installs a week even if nothing about the ads changes. And Thanksgiving week, when lots of people travel home, brings about 20 fewer.

Ignore seasonality and you can fool yourself. Say a shop had no subway billboards in November and put them up for the first time in December.

Week Subway billboards Installs that week
November week 3 Not up 490
December week 1 Up 540

Compare just these two weeks and installs went up by 50, so the billboards seem to bring in 50 a week. But 30 of those would have come anyway because it's December. The billboards brought in only 20. The 30 seasonal installs got counted as the billboards', so the billboards look more than twice as good as they really are.

That's why MMM gathers more than a year of ledger data and works out the seasonal share along with the ad shares. If installs ran about 30 a week higher last December with the ads unchanged, you learn that "December runs about 30 higher." Then the extra 50 in December splits into 30 for the season and the remaining 20 for the billboards. One year alone could be a fluke, so the more years the pattern repeats, the more certain it gets.

For this post, assume the June and July ledger had no seasonal differences like this.

One week's installs in one line

Everything we just did with the June and July ledger fits in one line. It's a simple sum, nothing but adding and subtracting.

June and July's ledger as one sum Base installs 490 every week, even without billboards + Subway billboards 20 in weeks they're up, 0 otherwise − Coupon 40 in rival coupon weeks, 0 otherwise = That week's installs Example: July week 1 is 490 + 20 − 40 = 470

In words: "A week's installs start from the 490 that come in every week even without subway billboards. In billboard weeks, add 20. In weeks a rival app runs a coupon, subtract 40." In this post we'll call that 490 base installs.

Plug in each week from the ledger and it comes very close to the actual installs. Billboard weeks add 20, and July week 1, the coupon week, also subtracts 40.

Week Installs from the sum Actual installs Off by
June week 1 Base 490 488 2
June week 2 Base 490 + billboards 20 = 510 512 2
June week 3 Base 490 491 1
June week 4 Base 490 + billboards 20 = 510 509 1
July week 1 Base 490 + billboards 20 − coupon 40 = 470 470 0
July week 2 Base 490 490 0
July week 3 Base 490 + billboards 20 = 510 511 1
July week 4 Base 490 489 1

The gaps of a few installs are just the week-to-week wobble we saw earlier.

This sum is the backbone of MMM. You split a week's installs into "what comes in even without ads," "each ad's share," and "the share from things that aren't ads," then add and subtract. When the owner starts another ad (say, the YouTuber sponsorship in August), the sum gets one more slot to add that ad's share. When something other than an ad moves installs, like a coupon or the season, the sum gets one more slot to add or subtract that share. The sum just gets more slots, but the adding and subtracting stays the same.

Adding the YouTuber sponsorship

In August, the YouTuber sponsorship started too. The first video went up in August week 3. Now the subway billboards and the YouTuber sponsorship are running together. Let's use the August ledger to split out how many installs each one brought in. The four original ads ran the same for these four weeks, and nothing special like rain or a coupon happened.

Week Subway billboards YouTuber video Installs that week
August week 1 Not up Not posted yet 490
August week 2 Up Not posted yet 510
August week 3 Not up Posted 520
August week 4 Up Posted 540

We rounded the numbers to keep the math easy. A real ledger would wobble by a few installs each week like June and July did, but here we'll assume it doesn't.

Now one week at a time.

  1. August week 1 (neither): no billboards, and no video yet. 490 installs, the same as the 490 base installs from June and July.
  2. August week 3 (YouTuber only): no billboards, just the video. 520 installs, 30 more than the 490 base. The YouTuber was the only change, so we can take the YouTuber's share as about 30 a week.
  3. August week 2 (subway billboards only): billboards up, no video yet. 510 installs, 20 more than base. So we can take the billboards' share as about 20 a week, the same as in June and July.
  4. August week 4 (both): billboards up and the video posted. If the numbers so far are right, this week should be 490 base installs plus 20 from the billboards and 30 from the YouTuber, 540 in total. Actual installs: 540. It checks out, which suggests the 20 and the 30 are about right.

As a sum: "that week's installs = 490 base installs + subway billboards' share (20 in weeks they're up) + YouTuber's share (30 once the video is up)." It's the June and July sum with one more slot for the YouTuber. Without knowing everyone who saw the billboards or watched the video, we split out how many installs each one brought in using only the numbers in the ledger.

How do you find the numbers you don't know?

So far we've found three numbers: 490 base installs, 20 for the billboards, and 30 for the YouTuber. In June and July, we averaged the billboard weeks and the no-billboard weeks; in August, we subtracted one week from another. Both worked because only one thing changed at a time.

What if TikTok spend had also bounced around week to week in June and July? Even if billboard weeks averaged higher, you couldn't tell whether that came from the billboards or from extra TikTok spend that happened to land in those same weeks. In a real ledger, all six ads shift a little together every week, so this happens all the time. So instead of averages, MMM uses a different approach: guess, then check. It works even when all six ads change together every week. Since it's new, though, let's first see how it works on the June and July ledger, where only the billboards changed. Six ads come after that.

In the June and July sum, "that week's installs = base installs + subway billboards' share," there are two things we don't know:

  • Base installs: how many installs come in each week without the billboards
  • Subway billboards' share: how many more come in when the billboards are up

Since we don't know them, start with a guess: "maybe base installs are about 480 and the billboards' share is about 30?" We'll call a guessed pair like this a candidate. If a candidate is right, you can work out how many installs each week should have had. Weeks without billboards have only base installs, so 480. Weeks with billboards add the billboards' share on top: 480 + 30 = 510. That's the prediction for the week. Compare it week by week with the actual installs in the ledger, and count how far off it is.

Leave out July week 1, the coupon week, and use the other seven weeks. First, candidate 1 (base installs 480, billboards 30). It predicts 480 for weeks without billboards and 510 for weeks with them.

Week Billboards Actual Predicted Off by
June week 1 Not up 488 480 8
June week 2 Up 512 510 2
June week 3 Not up 491 480 11
June week 4 Up 509 510 1
July week 2 Not up 490 480 10
July week 3 Up 511 510 1
July week 4 Not up 489 480 9
Total 42

Every week without billboards is off by around 10. The guess of 480 for base installs was too low.

Now candidate 2 (base installs 490, billboards 20). It predicts 490 for weeks without billboards and 510 for weeks with them.

Week Billboards Actual Predicted Off by
June week 1 Not up 488 490 2
June week 2 Up 512 510 2
June week 3 Not up 491 490 1
June week 4 Up 509 510 1
July week 2 Not up 490 490 0
July week 3 Up 511 510 1
July week 4 Not up 489 490 1
Total 8

This time every week is off by only 0 to 2. Do the same for candidate 3 (base installs 500, billboards 10) and it's off by 46 in total. Put the three candidates side by side:

Candidate Base installs Subway billboards' share Total miss over the seven weeks
Candidate 1 480 30 42
Candidate 2 490 20 8
Candidate 3 500 10 46

Candidate 2, "base installs 490, billboards 20," misses the least, so it's the most likely answer.

Plug in a guess, then pick the guess that misses the ledger the least. That's how MMM finds the numbers it doesn't know. We tried only three candidates, but a computer tries thousands or tens of thousands, nudging base installs through 480, 481, 482... and the billboards' share through 10, 11, 12, and so on. Then it picks the candidate that misses the ledger the least.

MMM is a computer doing all of this at once

Now let's widen this to all six ads. Earlier we said that in a real ledger, all six ads shift a little together every week. The owner's real ledger does exactly that. Here are October weeks 1 and 2 side by side.

Ledger October week 1 October week 2 Change
① Ad spend
Apple Search Ads $440 $460 $20 more
AppLovin $630 $570 $60 less
Snapchat $720 $780 $60 more
TikTok $350 $400 $50 more
Subway billboards $200 $200 Same
② How much the YouTuber's video was seen
New views that week 80,000 150,000 70,000 more (new video)
③ Result
Installs that week 535 575 40 more

③ Installs went up by 40, but five things in ① and ② changed a little at the same time. Was the extra 40 thanks to the YouTuber? Snapchat? How much did the AppLovin cut take away? These two weeks alone can't tell you. The June to August ledger was easy because the four original ads got exactly the same spend every week. Usually it doesn't work that way. The owner spends more on TikTok in weeks TikTok is doing well and less on AppLovin in weeks AppLovin isn't. So every week, spend on several ads shifts a little at once, just like October weeks 1 and 2 above. Then comparing two weeks can't tell you what drove the extra installs. On top of that, the ledger covers two years, over 100 weeks. Too many to compare by hand.

So all six ads go into one sum and get solved together. The sum has the same shape as in June and July, with one change.

In June through August, we only asked whether something was on or off: "20 in weeks the billboards are up," "30 once the video is up." But the other ads aren't on or off. Their spend changes a little every week. Apple Search Ads got $440 in October week 1 and $460 in week 2. To handle ads like that too, every ad has to go into the sum the same way.

It works like this: give each ad an unknown number, "installs per $10 spent," and multiply it by how many $10s you spent that week. We don't know this number yet. It's the number we have to find from the ledger.

The subway billboards make this easy to see. Earlier, the billboards brought in 20 installs in weeks the owner spent $200. That's about 1 install per $10. So instead of writing "20 in weeks they're up, 0 otherwise," you can write "1 install per $10" and get the same thing: 20 installs in a $200 week, 0 in a week with no spend.

Written as "installs per $10," ads whose spend changes a little every week, like Apple Search Ads, work the same way. Once you know how many installs Apple Search Ads brings in per $10, you can multiply that by the spend in a $440 week or a $460 week and get that week's share. For each ad, "how many installs it brings in per $10" is exactly the number we have to find from the ledger.

The YouTuber sponsorship works the same way. The ledger records new views that week instead of spend, so you multiply by views instead: "installs per 10,000 views." Before the video goes up, views are 0, so the YouTuber's share is 0. In weeks the video picks up views, the share grows with them.

Now all six ads go in the same way. For each ad, it's "that week's spend (views for the YouTuber) × installs per $10 (per 10,000 views for the YouTuber)."

With all six ads in, the sum looks like this.

The sum with six ads Installs with no ads Installs even with every ad turned off: ? + Apple Search Ads That week's spend × ? per $10 + AppLovin That week's spend × ? per $10 + Snapchat That week's spend × ? per $10 + TikTok That week's spend × ? per $10 + Subway billboards That week's spend × ? per $10 + YouTuber That week's views × ? per 10,000 views ± Special events Coupon week: −?, climbs the charts: +? ± Season Year-end: +?, holiday travel week: −? = That week's installs Finding the numbers behind each ? is what MMM does

The two bottom slots (special events and season) are ones we saw earlier. We said a rival's coupon week loses 40 installs and December adds 30. In reality, you don't know those numbers at first either. Like the ad shares, they have to be found from the ledger, so they get a ? too.

The shape of the sum is set, but there's a problem: we don't know any ad's "installs per $10" yet. We used 1 per $10 for the subway billboards as an example, but we could only get that because in the June and July ledger, the billboards were the only thing that moved. The other ads are still unknown, and so is how many installs would have come in with every ad turned off.

Write the sum out for the October week 2 ledger, and the known and unknown slots split like this. What we know is that week's spend and the 575 actual installs. What we don't know is marked ?. Since each ? is per $10, $460 counts as 46.

October week 2 Spend that week (views) Unknown That ad's share
Installs with no ads None ? ?
Apple Search Ads $460 ? per $10 46 × ?
AppLovin $570 ? per $10 57 × ?
Snapchat $780 ? per $10 78 × ?
TikTok $400 ? per $10 40 × ?
Subway billboards $200 ? per $10 20 × ?
YouTuber 150,000 views ? per 10,000 views 15 × ?
Total Must equal 575

Once every ? is filled in, the shares in the right-hand column should add up to the actual 575 installs. October week 2 had no coupon and no seasonal effect, so the special events and season shares are 0 and left out of the table.

If you knew every ?, you could split the 575 across the ads. The problem is that you don't. But there's one table like this for every week, and the ledger holds 100 weeks of them. Spend and installs differ each week, but we'll assume the numbers behind the ? stay the same every week. If the ad itself doesn't change, the billboards' installs per $10 shouldn't swing from 1 one week to 5 the next. This is just a starting point. Effects that carry into the following week, and installs per $10 that shrink as you spend more, come later, with small changes to the sum.

Finding the ? works just like trying candidates in June and July. There are a lot more of them, though, so this time the computer does all of it.

  1. The computer puts a guess into all seven ?. Something like "installs with every ad turned off: 350, Apple Search Ads: 0.5 per $10, AppLovin: 1 per $10, ..."
  2. Using those guesses, it calculates installs for each of the 100 weeks in the ledger.
  3. It adds up how far the calculated installs are from the actual installs, week by week.
  4. It changes the guesses a little and repeats steps 1 to 3.

It repeats this over and over and picks the guesses with the smallest total miss. The only difference from June and July is how many numbers are unknown. In June and July, we had to guess just two: base installs and the billboards' share. In the October ledger, we have to guess seven at once, as in the table above (one for installs with every ad turned off, plus an "installs per $10" for each of the six ads; add the coupon and season shares and there are even more). Too many for a person, so a computer does it.

Put everything so far together, and that's MMM.

Marketing Mix Modeling (MMM) works like this.

  1. Collect a ledger of what you spent on each ad every week (views for the YouTuber) and how many installs you got that week.
  2. Split each week's installs with a sum: "installs with every ad turned off + each ad's share ± the share from things like coupons or seasons."
  3. For the unknown numbers in the sum (like each ad's installs per $10), plug in guesses and pick the ones that miss the ledger the least.

Even with no record of who saw an ad, a ledger is enough to tell how many installs each ad brought in. It works the same way with revenue instead of installs.

This is where MMM differs most from an MMP report. A report has to link a person to the ad they tapped before it can count the install, so it easily misses people who saw a billboard and walked past without scanning, people who watched a video and searched later, and people who declined tracking on their iPhone. MMM splits up the total number of installs in the App Store each week, so every install can be divided into the ads' shares, whether or not anyone tapped a link or allowed tracking.

Now think back to the "490 base installs" we found from the June and July ledger. Back then, the sum had a slot only for the subway billboards. The four original ads didn't get their own slots, because their spend stayed the same every week and nothing about them changed. So the installs those four ads brought in each week ended up inside the 490 too. The 490 wasn't "installs that would have come with no ads at all," just "installs that would have come without the subway billboards." It mixed two things:

  • installs brought in by the four original ads (Apple Search Ads, AppLovin, Snapchat, TikTok)
  • installs that would have come in with no ads at all

In June and July there was no way to separate them. The four original ads got exactly the same spend every week, so there was no clue to how many installs they brought in.

When the computer looked for the ? above, two things were different from June and July. Besides the subway billboards, the four original ads and the YouTuber sponsorship each had a slot of their own, and instead of the eight weeks of June and July, it used the full two years, over 100 weeks, of ledger data. Over those two years, each ad's weekly spend (views for the YouTuber) varied a little, so the computer can find the "installs per $10" for all six ads together. Then it can separate out all six ads' shares, and what's left at the end is "installs that would have come in with no ads at all."

Say the computer has found every ?, and here's how many installs each ad brought in during October week 2. Take each ad's share out of that week's 575 installs, one at a time:

  • That week's installs: 575
  • The four original ads' share: Apple Search Ads 15, AppLovin 48, Snapchat 62, TikTok 40, 165 in total
  • Subway billboards' share: 20
  • YouTuber's share: 45
  • Left after taking them all out: 575 − 165 − 20 − 45 = 345

Those 345, what's left after removing all six ads' shares, are installs that would have happened without ads. They wouldn't have disappeared even with all six ads turned off. Flip it around, and each ad's share (say, the YouTuber's 45) means the same thing as "installs caused by the ad" from the start of this post: the installs that would have disappeared if that ad had been off.

These 345 are a different number from the organic line in the MMP report earlier. Both look like "installs that aren't any ad's share," but they count different things.

  • The report's organic: "installs with no ad record." It mixes installs from people who really saw no ad with installs from people who saw an ad but left no record.
  • MMM's 345: "installs that would have happened without ads." Installs an ad brought in go to that ad's share, record or no record, and only the installs that truly would have happened without any ad are left.

Take someone who saw a subway billboard on the way to work and searched for the app that evening. The report counts them as organic, but MMM puts them in the subway billboards' share.

The sum itself isn't hard, as you've seen, so you could write your own program to have a computer do this. You don't need to, though. Good free tools already exist: feed in the ledger and they handle the guessing and checking. The best known are Meridian, open-sourced by Google, and Robyn, open-sourced by Meta.

Ad effects can show up late

In the June and July ledger, we simplified one thing to make the math easy. We assumed that in billboard weeks, everyone the billboards would bring in installed that same week, and that in weeks the billboards were down, nobody installed because of them. That's not how it really works.

Someone who sees a billboard on their Monday commute might install that day, but not necessarily. They might pass it a few more times over the next few days and install on Saturday night when they want pizza. The effect sticks around after the billboard week is over. The YouTuber's video even more so. People don't all watch it the week it goes up. Views build over several weeks, and installs keep coming with them. This lingering effect after an ad runs is called adstock.

Say the subway billboards were up for just one week, and the effect came in as 10 installs that week, 6 the next, and 4 the week after.

One week of subway billboards, three weeks of installs 10 Up week 6 +1 week 4 +2 weeks 0 +3 weeks Billboards up Total: 20 installs

Count only the billboard week and it's 10, but across three weeks it's 20. Keep the billboards up continuously and the leftover effect from last week's and the previous week's billboards stacks on top of this week's, so from the third week on, you get about 20 a week. About 80 a month.

Miss this and the math goes wrong. Even in the first week after the billboards come down, 10 extra installs (6 + 4) still come in thanks to earlier billboards, and 4 more the week after. Looking at the ledger, you think "we took the billboards down and installs barely dropped," so the billboards look weaker than they are. Now say that in the very week the billboards came down, the owner happened to raise TikTok spend. The 10 extra installs that week are really thanks to the billboards that ran until last week. But if you don't know effects linger, you think "TikTok is what went up this week, so these 10 must be TikTok's" and credit them to TikTok. Then TikTok looks better than it really is.

So MMM tweaks the sum a little. Here's the sum we built from the June and July ledger:

Before
Base installs
+Subway billboards' share
−Coupon share
=That week's installs

Here, "subway billboards' share" only counted billboards that were up this week. Split that one slot into three:

After
Base installs
+Share from this week's billboards
+Share left over from last week's billboards
+Share left over from billboards two weeks ago
−Coupon share
=That week's installs

Billboards from last week and the week before still bring in installs this week, so they get their own slots. The other slots (base installs, coupon share) stay as they were.

Plug in the numbers from the chart above (10, 6, and 4). In a week when the billboards have been up for several weeks running, this week's subway billboard share comes from three places.

Where the installs came from Installs
This week's billboards 10
Left over from last week's billboards 6
Left over from billboards two weeks ago 4
This week's subway billboard share 20

In the first week after the billboards come down, none went up this week, so the first row is 0. But the 6 and 4 left over from the two weeks before are still there, so the subway billboards' share is 10.

10, 6, and 4 are numbers we picked to make this easier to follow. In reality, they're unknown too. Like the October week 2 table, mark each unknown slot in the new sum with a ?:

Slot in the sum Installs
Base installs ?
+ Share from this week's billboards ?
+ Share left over from last week's billboards ?
+ Share left over from billboards two weeks ago ?
− Coupon share ?
= That week's installs The actual number in the ledger

You find these ? the same way as before. Plug in guesses and pick the one that misses the ledger the least. Take two candidates, for example.

Candidate This week Left from last week Left from 2 weeks ago First week after removal
Candidate A: effect ends the same week 20 0 0 Same as base installs
Candidate B: effect lasts three weeks 10 6 4 10 more than base installs

In weeks the billboards stay up, both candidates give 20, so they fit equally well. The difference shows up in the first week after the billboards come down. If the ledger shows about 10 more installs than base installs even in that week, candidate B fits better.

"Lasts three weeks" is just an example, too. Ads people tap to install right away might be done in a week, while subway billboards or a YouTuber video might linger for four or five. The computer also tries candidates like "lasts one week," "lasts two weeks," and "lasts three weeks," and picks whichever misses the ledger the least.

So how many installs did the subway billboards and the YouTuber bring in?

Back to the owner's original question. The owner gathered two years of weekly ledgers like the October week 1 ledger above and put them into an MMM tool.

October's four weeks of installs came out split like this.

October's installs, split week by week 535 Oct wk 1 575 Oct wk 2 540 Oct wk 3 550 Oct wk 4 Installs with no ads Apple Search Ads AppLovin Snapchat TikTok Subway billboards YouTuber The four weeks add up to the MMM column below

The gray at the bottom shows installs that would have happened without ads. Each ad's share is stacked on top. The subway billboard and YouTuber shares, barely visible in the report, finally show up.

The second bar is October week 2, the one we worked out earlier. Of its 575 installs, the gray 345 would have happened without ads, and the other 230 are ad shares. The YouTuber's share (pink) is 45, bigger than in the other weeks, because a new video went up that week.

Add up the four weeks for each ad and you get October's monthly numbers. The chart only shows bars, but in numbers, the subway billboards were 20 every week, so 80 in total. The YouTuber was 25 in week 1, 45 in week 2, 35 in week 3, and 15 in week 4, so 120 in total. Set those monthly numbers next to the MMP report and you get the table below. The last two columns show what one install cost. Divide monthly spend by the MMP number to get cost per install by the report, or by the MMM number to get the cost of each install the ad really brought in. For the subway billboards, that's $800 ÷ 10 = $80 by the report, and $800 ÷ 80 = $10 by MMM.

Ad and monthly spend MMP report MMM Per install (MMP) Per install (MMM)
Apple Search Ads
$1,800
650 60 about $2.80 $30
AppLovin
$2,400
400 200 $6 $12
Snapchat
$3,000
300 240 $10 $12.50
TikTok
$1,500
100 150 $15 $10
Subway billboards
$800
10 80 $80 $10
YouTuber sponsorship
$1,800
20 120 $90 $15
Organic (report) / Installs with no ads (MMM) 720 1,350
Total 2,200 2,200

The total is the same 2,200 either way. The two methods just split the same 2,200 differently.

  • Subway billboards: 10 in the report, 80 by MMM. Cost per install drops from $80 by the report to $10 by MMM, among the cheapest of the six ads. Taking them down based on the report would have been a mistake.
  • YouTuber sponsorship: 20 in the report, 120 by MMM. By the report, that's $90 an install, which looks like an ad to cut right away. By MMM, it's $15.
  • Apple Search Ads: The biggest in the report at 650, but 60 by MMM. Cost per install goes from about $2.80 to $30, the most expensive of the six. Many of the installs the report credited to it came from people who already planned to install and searched for the app, or who saw another ad and searched later.
  • TikTok: The other way around: MMM (150) is bigger than the report (100). Someone who sees a TikTok ad, searches the app's name days later, and taps an Apple Search Ads result gets credited to Apple Search Ads in the report, but MMM gives that install back to TikTok.
  • 1,350 installs that would have happened without ads: This is what's left of October's 2,200 installs after taking out the six ads' shares that MMM found (60 + 200 + 240 + 150 + 80 + 120 = 850). These installs would have come in during October even with no ads at all.

How accurate are 80 and 120?

The owner immediately wanted to raise the subway billboard and YouTuber budgets. Before that, there's something to check.

The comparison table above gave one number for each, like 80 for the subway billboards and 120 for the YouTuber. But MMM tools like Meridian usually show the answer as a range, not a single number. Something like "the most likely value is 80, but it's probably somewhere between 30 and 130." The wider the range, the less sure the tool is. The owner got these ranges.

Ad Installs brought in per month (range from the MMM tool)
Subway billboards 30 to 130
YouTuber sponsorship 50 to 190

The range is this wide because the records are short. The four original ads have two years of ledger data, but the subway billboards have only twenty-some weeks since June, and the YouTuber only ten weeks since August week 3. With a short ledger, there aren't enough clues to tell "60 is right" from "100 is right." As records build up, the range narrows, which means more confidence in the answer. MMM tools recommend collecting plenty of data for this reason. Robyn recommends at least two years of weekly data (Robyn docs). Meridian recommends at least three years for a ledger like the owner's that keeps everything on one line instead of splitting it by region (Meridian docs). If you keep a separate ledger for each region, like New York and Chicago, you get more clues over the same stretch of time, so Meridian says two years is enough.

But with a range of 30 to 130, it's hard for the owner to decide anything. The billboards cost $800 a month, so here's what one install would cost:

  • If the billboards brought in 30 a month: $800 ÷ 30, about $27 an install. Almost as expensive as Apple Search Ads ($30 an install), the most expensive of the six.
  • If the billboards brought in 130 a month: $800 ÷ 130, about $6 an install. The cheapest of the six.

The same billboards could be "an ad to cut" or "an ad to grow," depending on which end of the range is true. And waiting until two years of records pile up means putting the decision off for more than a year and a half.

When you can't just wait for records to build up, there's a faster way to narrow the range: an experiment. You measure directly how many installs the subway billboards bring in, then tell MMM that number. As it happens, the owner ran an experiment with the billboards in September. It's the geo experiment we saw earlier as "splitting stations."

  • What they did: picked three of the six stations at random and took those billboards down for all of September. The other three stayed up.
  • How installs were counted: counted installs separately for the neighborhoods around the three stations without billboards and the three with them. The App Store doesn't report installs by neighborhood, so the owner used the delivery address on each new user's first order to tell which neighborhood the install came from.
  • The result: neighborhoods without billboards got about 40 fewer installs that month. 40 for three stations suggests about 80 for six.

Meridian and Robyn both have a place to enter experiment results like this. The owner entered "subway billboards: about 80 a month" and ran MMM again. The tool then finds an answer that fits both the ledger and the experiment, and the range narrows from 30 to 130 down to 65 to 95. Adding an experiment's result to MMM so that its answer doesn't stray far from what the experiment measured is called calibration. A narrower range makes the decision much easier. At 30 to 130, the cost per install could be anywhere from $6 to $27. At 65 to 95, it's roughly $8 to $12. Either way it's a cheap ad, so you can say with confidence it isn't one to cut.

Installs the subway billboards brought in per month 0 50 100 150 MMM alone 30 130 80 Geo experiment 80 MMM + experiment 65 95 80

The subway billboards aren't the only ad this helps. Whenever another ad can be tested, adding those results makes MMM's answer more accurate. It helps most with ads that are easy to misread from the ledger alone, like Apple Search Ads. In weeks when lots of people are looking for pizza, search ads get more installs, and their spend also rises on its own, so the ledger makes it easy to mistake the ad for what drove the installs.

A geo experiment doesn't work for the YouTuber. The owner decided not to raise the YouTuber budget yet, and to collect a few more months of records and wait for the range to narrow.

Double the subway billboards, double the installs?

Having learned the subway billboards cost about $10 an install, the owner thought about going from six stations to twelve, or even eighteen. If $800 a month brings in 80, will $1,600 bring in 160, and $2,400 bring in 240?

Usually not. When the owner picked the first six stations, they started with the ones closest to the shop with the most foot traffic. Adding stations means going further from the shop, to stations used by people less likely to order the owner's pizza. So each extra dollar adds fewer installs. Installs growing more slowly the more you spend is called saturation. Online ads work the same way. Platforms like TikTok show ads first to the people most likely to install, so the more you spend, the more the ads reach people who are less and less interested.

So how does MMM figure out how much more slowly installs grow as spend goes up? Earlier, an ad's slot was "that week's spend × ? installs per $10." And we assumed that ? was one number that stayed the same no matter how much you spent. For the subway billboards, 1 install per $10, whether you spent $200 or $600.

To build in saturation, you let that ? change with spend. The first dollars bring in ? installs per $10, and the more you spend, the fewer installs each $10 brings in. That gives you two unknowns:

  • How many installs per $10 at first
  • How fast that number drops as you spend more

You find both the same way as every other ?. Put one candidate curve into the subway billboards' slot in the sum, and use the sum to calculate each week's installs in the ledger. Add up how far off it is from the actual installs. Repeat with different candidates and pick the one that misses the least. The only difference is that this slot now holds two ?.

You don't have to do this yourself either. MMM tools like Meridian and Robyn already assume that "the more you spend, the fewer installs each $10 brings in." So without any extra setup, they also work out how fast that number drops for each ad.

Say this is what came out for the subway billboards.

Subway billboards: more stations, smaller gains $800 $1,600 $2,400 80 128 152 6 stations 12 stations 18 stations +80 (1 per $10) +48 (0.6 per $10) +24 (0.3 per $10) Monthly subway billboard spend Installs from subway billboards
  • First $800 (six stations): 80 installs. 1 per $10.
  • Next $800 (up to twelve stations): 48 more, 128 in total. Down to 0.6 per $10.
  • The $800 after that (up to eighteen stations): 24 more, 152 in total. Only 0.3 per $10.

The more you spend, the fewer installs each $10 brings in.

But there's a catch. The owner's ledger only has subway billboard spending up to $800 a month (six stations). The owner has never spent $1,600 or $2,400.

So of the numbers above, the only one checked against the ledger is "$800 for 80." "$1,600 for 128" and "$2,400 for 152" didn't come from actually spending that much. They're a guess: MMM looked at the records up to $800 and extended the curve, figuring "it'll probably keep flattening out like this." Most ads do flatten out like this, so it's a reasonable guess, but the only way to know is to actually go up to twelve stations.

If the guess holds, the extra $800 for going to twelve stations brings 0.6 installs per $10, or about $17 an install. That's close to the YouTuber sponsorship ($15) and cheaper than Apple Search Ads ($30), so it's worth a try. Going all the way to eighteen, though, the next $800 brings only 0.3 per $10, or about $33 an install, more than Apple Search Ads.

So the owner decided to go up to twelve stations starting in November, and no further. The money would come out of the Apple Search Ads budget, since at $30 an install, Apple Search Ads was the most expensive of the six. Total spend stays the same; it just moves to where it buys more installs. But "twelve stations means 128 a month" is a guess, not something the owner has actually tried. So the plan is to run MMM again in a few months and check whether the six new stations really brought in about 48 more.

Recap

Ads that leave only partial records of who saw them, or none at all, can still be measured. Instead of following people one by one, you line up weekly ad spend next to weekly installs. MMM does four things.

  1. It splits a week's installs into shares. It splits the installs from one week into "installs that would have come in without ads," "installs each ad brought in," and "installs added or lost because of things like coupons or the season," so that adding and subtracting them gives the actual installs that week.
  2. It finds the unknown numbers in the sum by trial and error. You don't know each ad's installs per $10, so you plug in numbers and pick the ones that miss the whole ledger the least.
  3. It builds how ads behave into the sum. Effects that show up late (adstock), and installs that grow more slowly the more you spend (saturation).
  4. It gives the answer as a range. Tools like Meridian and Robyn show "somewhere in here" instead of a single number. A short ledger means a wide range, and more records or experiment results make it narrower.

If you want to try what the owner did, start by keeping a steady weekly log of what you spend on each ad, how many installs you get each week, and special events like coupons or app ranking changes. Tools like Meridian and Robyn handle the math.

Now the owner looks at an MMM table next to the MMP report every month. The report gives installs that left a record to the last ad tapped. MMM uses the weekly ledger to split out how many installs each ad brought in, even for ads that are hard to track, like the subway billboards and the YouTuber sponsorship. When the owner first opened the October report, they wondered, "At $80 an install, should I take the subway billboards down?" Going by the report alone, they would have. After putting the weekly ledger into an MMM tool and splitting out what each ad brought in, the answer flipped: the billboards weren't an ad to cut, but one worth growing. You don't need to know everyone who saw your ads. Keep a steady record of what you spend and how many installs you get each week, and you can make calls like this. That's MMM.