What is the purpose of an incrementality test?
The goal of an incrementality test is to determine what portion of conversions are a result of the retargeting campaign specifically, rather than from your general marketing efforts. The difference between the level of conversions between two groups during the test is called incremental lift and enables you to understand what impact your retargeting provider has on your general marketing results.
How do incrementality tests work?
There are a number of different ways to run an incrementality test, but they all share the same basic approach.
First, the retargeting provider and client define the target group. These users are then split into two groups:
Test Group – These users are delivered ads using the retargeting provider’s methodology.
Control Group – These users are not delivered ads using standard retargeting activities, and are used as a comparison.
Before conducting a test, you need to keep in mind the statistical difference between these groups. Specifically, this means that you need to eliminate the risk that the test results were obtained due to random chance. To do this, we use a P-value, which represents the probability that the results were obtained due to randomness, and variance, which tracks the randomness of the result itself.
Over the course of the campaign, it is possible to determine the difference between the test and control group, you can see some of the results below:
These results are useful because they give you a more accurate picture of your campaigns than a single method, such as attribution, is able to do in isolation.
What do I need to consider before starting an incrementality test?
On the surface, incrementality testing seems like a simple tool to validate your retargeting campaign spend. However, like anything worth doing, an incrementality test requires companies to invest significant time and resources in order to maximize their value.
Before starting an incrementality test, you should consult this checklist to ensure that your business can conduct a test that will produce reliable results.
1. Do you have an analytics team in place?
While your retargeting provider can give you some insights about your incrementality test, it is important to be able to verify the results internally. To do this, you need an analytics team in place with the resources to analyze them.
The team will examine the granular conversion data, and validate partner data against your internal data to confirm that it is correct. Additionally, this team will need to verify that the audience split used is statistically valid. These steps help to ensure that you’re working with valid data that can be used to derive correct insights
2. Is your campaign large enough for an incrementality test?
Data is good, but more data is always better. If your campaign is small, or you have a limited number of transactions, an incrementality test may not provide enough data for your analytics team to draw meaningful insights.
In these cases, you would be better placed using another tool to validate your retargeting strategy.
3. Is your campaign stable?
It can be tempting to run an incrementality test from day one, but doing so can distort the results. This can either be due to technical issues with the campaign itself or because the campaign has not scaled enough to produce statistically significant results.
Generally, we would recommend that you run a normal retargeting campaign for a limited period of time, typically at least two to three months, in order to run your incrementality test with real and stable data.
Assuming that you meet all these criteria, then you can begin looking at what kind of incrementality test you want to run. If not, you can skip straight to our incrementality test alternatives chapter here.
How does RTB House conduct incrementality tests?
There are two industry standard approaches to measure incrementality:
1. RTB House Ghost Ads
The first approach combines industry standard Ghost Ads with our own precise methods of measuring lift. This process involves splitting users into two groups:
A test group – who are shown several ads at a higher frequency rate
A control group – who are shown the same ads but at a limited frequency, typically around 1 impression every 8 days
Our own internal data has demonstrated that the most common ideal split is 80/20. While this has a minor impact on statistical significance compared to a straight 50/50 split, it minimizes the opportunity loss of utilizing a larger control group. We mitigate any reduction in statistical significance by ensuring that the test runs for at least 21–35 days, enabling us to gather more data.
It’s a myth that Ghost Ads can’t be run during a multiple retargeting campaign. Ghost Ads work in both single retargeter and multiple retargeter campaigns, in fact, the more campaigns you run the easier it is to determine incremental lift!
2. The intent-to-treat method
The second industry approach is the intent-to-treat method. As with our Ghost Ads solution, we split users into the control and test groups. When using intent-to-treat, users in the control group are not shown any ads at all. During the test, data is collected from both groups, and then compared to determine the incremental lift of the campaign.
Despite seeming similar at first glance, intent-to-treat comes with a major drawback: noise. During an intent-to-treat test, there is an increased propensity to buy more ads across both exposed and control populations. This increases the background marketing noise, and can make it challenging to actually isolate the real incremental lift from that noise, reducing the overall accuracy of the test compared to Ghost Ads.
Should I use Ghost Ads, or intent-to-treat?
In general, we recommend that our clients use Ghost Ads rather than intent-to-treat tests. The first advantage is that Ghost Ads still display ads to the control base, limiting any opportunity loss. The second, and most important reason, is that Ghost Ads are better at eliminating background noise. Ghost Ads ultimately provide more accurate data, and enable you to accurately validate each partner in your retargeting strategy and best determine how to allocate budget spend.
No matter which kind of incrementality test you chose to use, RTB House will use Deep Learning in order to improve the results of the campaign. Deep Learning is an advanced form of machine learning capable of understanding complicated unstructured data sets and providing meaningful insights from them.
What are the alternatives to incrementality tests?
There are not many alternatives that provide the same granularity of data as an incrementality test. In general, we discourage clients from relying on approaches like time series analysis or comparative market analysis, because they are highly susceptible to outside factors like seasonality.
The only real alternative is to rely on unbiased attribution, for example, Google Analytics which shows the results for all advertising sources using the same methodology. While not as granular as an incrementality test, it will enable your team to make an apples-to-apples comparison. These analytics solutions also have the advantage of being a proven method, used for many years in online marketing.
A test-and-learn mindset is key
Whether you use an incrementality test, or one of the many alternatives, the key is in how you use this data. Adopting a test-and-learn mindset will ensure that you are always looking for ways to improve your marketing campaigns, regardless of what methodology you are using to do so.
With all that said, we think it’s time we lead by example and showcased some of our own data-led retargeting success stories. RTB House takes a proactive approach to retargeting, with a heavy emphasis on using our proprietary Deep Learning technology in order to maximize the benefits of our retargeting campaigns. Our own in-depth internal testing has ensured that we’re able to deliver significant incremental improvements.
One of the best examples of multiple retargeting success can be seen in this case study, detailing how we became Huuuge Games’ most successful retargeting provider after just three months, exceeding ROAS goals by 3-4 times.
Another example is AllBeauty, where the RTB House team was able to help the brand make the most of a spike in online interest through an extensive retargeting campaign. Over the course of the campaign we helped boost the client’s year-on-year revenue from retargeting campaigns by 65%, and the number of conversions by 45%.
You can find all of our retargeting success stories here. If you’d like to learn more about how RTB House can help your brand kickstart a retargeting campaign, or reap the benefits of a multiple retargeting campaign, talk to our team today.
