Test, learn and earn Boosting Performances with Facebook test solutions - Converteo
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Test, learn
and earn
Boosting Performances with Facebook test solutions
Test, learn and earn 1
Test, learn and earn 2
Introduction
Test, learn and earn 3
Table of contents INTRODUCTION p3
Table of contents p4
The contributesters p5
Executive summary p7
Our approach to testing p8
THINK p9
Understand your stakes and needs p10
Get started with hypothesis to verify p11
What is incrementality? p12
Understand the different types of tests p13
Brand Lift p14
Conversion Lift Studies p16
Single-Cell Lift Tests p18
Multi-cell & Split Test p19
Client case examples p20
BUILD p22
Build section guide p23
5 best practices to become a master tester p24
Facebook maturity assessment p25
Tests matrix p26
Cases p27
Creative p28
Audience p32
Set-up optimization p35
Media orchestration p39
RUN p43
Test monitoring p44
Analyzing test results p45
Adapting media strategy p46
BOOST p48
Cross-publisher test p49
Facebook Attribution p50
Go further with offline p51
Marketing Mix Modeling p52
Long-term brand lift p54
APPENDIX p55
Test, learn and earn 4
Converteo
& the contributors
To realise this how-to guide, people from different expertises &
background were united to create the most accurate vision for
advertisers. All the consultants who contributed to this document
are presented below. Thanks also to Converteo’s whole team for
the support and knowledge sharing.
Converteo is a service and consulting firm specialized in data
marketing and related technology, composed of over 180
consultants and seasoned experts. Aiming at providing significant
and lasting business impact, we cover the whole project value
chain, from diagnosis and roadmap to implementation and run. We
are the often-missing link between marketing and IT. We work with
a pragmatic and fiercely independent approach, and help our clients
build in-house capabilities.
For more information about Converteo, please visit converteo.com
Key clients:
Carrefour, Accor, Samsung, L’Oreal, Se Loger, Europcar...
Test, learn and earn 5
Stéphanie Euzen
Project Management
Stéphanie is specialized in Social ads ecosystem & Drive-to-
store challenges. Key contributor, she led the production of
the document
Vincent Colleu
Measurement Expert
As a specialist of Performance Measurement (webanalytics
& ad-centric) & data analysis, Vincent brought his rational
and data-driven vision to the document
Michael Niavet
Facebook Expert
Key expert at Converteo, Michael brought his Social Media
experience to give specific example and make sure the
document was actionable
Ilan Nedjar
Multi-channel Expert
With experiences in Paid Search, Social & Programmatic, Ilan
made sure our Facebook-centric vision taught general and key
insights about media testing
Charline Goret
Media Strategist
Managing large amounts of campaigns each year, Charline
gathered her clients’ problematics in this document and
made sure we were solving key pain points for advertisers
Emmanuel Chomier
Media performance lead
With knowledge of numerous media performance projects,
Emmanuel framed the next steps of the document, from
testing in Facebook to globally improving your strategy.
Merry Mazmanian
Media Strategist
Thanks to her technical & business knowledge of Facebook
campaigns, Merry validated all the set-up and tests
requirements in collaboration with Facebook Teams
+ Marketing Science Team at Facebook
Test, learn and earn 6
Executive summary
Anti-tracking prevention system has been studies. To go further than what you could have
increasingly threatening cookie-based already implemented, we will focus on Multi-
advertising over the last years. As some browsers cell studies. Your hypothesis can be steered to
announced the end of third party cookies by different topics: creatives, audiences, set-up
2022, it’s critical to rethink the way you target optimization, media orchestration. Some testing
and measure media campaigns, and to integrate methods have been detailed in order to guide you
incrementality based tests for better media and allow you to build correctly your test in the
campaign handling. BUILD section.
Developing business is crucial but it demands to Your test is set-up and runs. Keep in mind that
determine clear objectives to reach your goals during tests, optimizations are possible according
and sticks to a testing approach. This document to your test design: for instance, you can run A/B
will help you define every step you have to take test as “normal” campaigns but some variables
into account and how to go further. First of all, the of randomized control tests need to be left
cornerstone of this work is your media strategy. It unchanged. Then, when your campaigns are over,
will allow you to identify the key aspects to focus it’s time to analyze your results and get actionable
on, in order to retrieve specific hypothesis to test. insights in order to adapt your strategy in the
Three main categories of tests can be runned: most efficient way possible. This topic is detailed
A/B test, Brand Lift and Conversion Lift studies; in the RUN section.
according to particular criterias which we will
To go further, some other tests are available
describe more precisely in the THINK section.
to expand your knowledge of your business
On this basis, we will focus only on incrementality and enhance your media impact. Examples are
tests which are Brand Lift and Conversion Lift covered in the BOOST section.
Test, learn and earn 7
Our approach to testing
An impartial and qualified player
Our goal is to use data to help you improve your business performances.
We go hand in hand with you across the full value chain of data:
from collection to exploitation and activation.
We bring value at every step of data project, regardless of the data
solutions you use, we do not favor any actor and try to remain as
neutral as possible.
On top of this data expertise, we also try to popularize and explain
the different data trends and projects.
Going through relevant tests and campaign type
Our main objective is to go through all the possibilities offered by
those test and to identify the specificities of each one.
For each test we tried:
• To describe the test as objectively as possible (criteria,
stakeholders etc)
• To help you define a relevant hypothesis
• To highlight the bias and limits of each test
We also try to go further, by wondering which test you could run
after getting the first insights.
Help you identify the right questions and tests to run
The main outcome of this study is that we can not claim in advance
which test you should run regarding your business issue and what
impact it would have.
Our conviction is that every test is valuable and its significance
depends on the settings, goals and budget you use.
To have significant results you must ask your team the
following questions:
• What is the purpose of your test? What is your hypothesis?
• How can you measure your results? Based on which KPI? What is
the impact wanted?
• What budget and reach you must have in order to be significant?
It’s only by going through those questions, and trying to sharpen as most
as possible your test that you will get satisfying and relevant results.
Test, learn and earn 8
Think
This section helps you define your testing strategy
and identify the optimal type of test (AB Test, Brand
Lift, Conversion Lift) to launch depending on your
hypothesis and media planning.
Test, learn and earn 9
Understand your
stakes and needs
Before you start designing your test plan, it’s crucial to understand
why and how experiments can help your business.
Why should you run Facebook experiments?
Facebook experiment tools may present several advantages:
Facebook automatically Facebook experiments are easy Experiments will enrich
provides randomized, non to set-up and can be easily your media strategies with
overlapping groups of people adapted to business sizes actionable results and best
and maturity practices for future campaigns
Before you get started: connect campaigns to business results
Your testing roadmap should always reflect macro business objectives.
Here is how we recommend to proceed before you get started:
Define measurable business objectives
1 Does your brand suffer from a lack of awareness?
Do you struggle with selling premium products?
Align business objectives with specific KPI
2 Is awareness about brand affinity or ad recall?
Are conversion metrics more important than qualified traffic?
Identify challenges, opportunities & key questions needed
to achieve your business objective
3
Think about budget & time allocation, audience & message strategies.
What questions are often raised before you start a campaign?
Prioritize questions depending on their expected impact on your business
4 Proper planning upfront will help you and your stakeholders to make
the best business decision.
Define your test hypothesis
5
That isolates a single variable to test (see next slide)
Test, learn and earn 10Get started with a
hypothesis to verify
To run an effective experiment on Facebook, you should first choose a hypothesis,
or question, you want the experiment to answer. The hypothesis should be specific,
actionable and relevant to your business objectives. A data-driven, test and learn
approach to advertising can help you better understand the impact of your advertising
Build your hypothesis
Determine business objective and take benefit from your previous
research to build your hypothesis
Define the variable you want to test. Different variables could be
tested to answer your hypothesis:
Optimization for ad Bid strategy Placement Budget
delivery/objective
Targeting Ad format/Content Buying type Campaign launch
Write your test hypothesis.
For example, if you’re trying to understand which ad format will perform the best
with your audience in an upcoming ad campaign, a good hypothesis might state,
“A video format performs better than a carousel format when reaching my
audience about new products.”
Challenge your hypothesis
The actions you take to test this hypothesis can reveal a better understanding of
how your ads perform with your audience.
• Identify relevant KPI to monitor and measure.
Ex: brand awareness, ad recall, add to cart, reach, purchase, traffic on site
• Answer to the hypothesis and analyse the business impact for your brand
• Activate if relevant, these new learnings to your upcoming campaigns
Test, learn and earn 11Choose your test
What is incrementality? How incrementality is measured?
Incrementality helps you understand the impact Generally speaking, the most methodical approach
of your marketing campaigns through your is to apply a holdout to your campaign. The holdout
ads. In other words, Incrementality helps you acts as a control group of users who will not be
identify the additional impact of Facebook ads. exposed to your ads. At the opposite, the test
Measuring incrementality will help you optimize group includes those who will see your ads.
efficiently your marketing strategies and
Once the test is completed, you will take the
campaign scenarios.
results from your test group and compare that
number to your control group. The difference
will be your Lift and represents the incremental
impact. So, the basic question for Incrementality
is: Are the number of increased conversions
caused by your Facebook ads?
Example
For example, Group A and Group B show similar Added to hold out group,
not exposed to a Facebook Ads
behavior, they both convert. The only difference
between these two audiences is that a Facebook Group A
campaign has been runned for Group B. The 100
conversions from Group A will be organic and/ Added to test group,
Conversions
or driven by other activities (excluded Facebook), exposed to a Facebook Ads
so any increase seen in Group B will show you Group B
the incremental lift caused by your ad spend. For Incremental Lift: 120
example, you could have the following results: +20% Conversions
How to increase your 1. Creative:
campaign incrementality ? Design attractive creative assets that are
formatted for a mobile environment
Thanks to Facebook studies, incrementality
testing can be a very powerful tool in measuring 2. Targeting:
your campaign’s impact. Leverage your audience insights
It makes you understand which ads generate 3. Device:
sales or improve brand metrics (and which don’t). Use optimal content based on the device
There are different key takeaways to keep in mind 4. Placements:
so as to increase your campaign Incrementality: Run across multiple placements for
maximum efficiency
5. Optimizations:
Consider using Facebook attribution to
understand users journey and adapt your
marketing budget
Test, learn and earn 12Understand the
different types of tests
A/B tests vs lift tests
There are three criteria that differentiate the different types of tests:
• Incrementality, thanks to hold out groups: a/b tests vs lift tests
• The number of cells: single-cell vs multi-cell
• The KPIs: branding or conversion items
A/B TEST LIFT TESTS
No incremental impact With incremental impact
Conversion lift brand lift
Single-cell Multi-cell Single-cell Multi-cell
Two cells minimum:
What is the What is the best What is the What is the best
Which scenario drives
incremental scenario to drive incremental scenario to drive
the best performances?
impact of the incremental impact of the incremental
campaign on conversions? campaign on impact on brand
conversion? brand metrics? metrics?
Self-served vs managed tests
Depending on your business objective and budget, you can work
with your Facebook representative to create Facebook-managed
tests or work without Facebook representative and benefit from
other ways to measure ad performance using a variety of solutions
right from Ads Manager.
Managed tests Self-served tests
If you work with a Facebook representative If you don’t work with a Facebook representative
A/B tests A/B tests
Single-cell brand-lift study Single-cell brand-lift study
(see requirements & specificities in appendix) (see requirements & specificities in appendix)
Multi-cell brand-lift study Multi-cell brand-lift study
(see requirements & specificities in appendix) (see requirements & specificities in appendix)
Single-cell conversion lift Single-cell conversion lift
(see requirements & specificities in appendix) (see requirements & specificities in appendix)
Multi-cell conversion lift Multi-cell conversion lift
(see requirements & specificities in appendix) (see requirements & specificities in appendix)
Test, learn and earn 13Brand lift study
Brand lift study will help you measure the incremental impact of your campaigns on
branding metrics. They can be declined in single or multi-cell tests.
Brand lift methodology
1 2 3 4 5
Campaign Randomised Deliver ads Polls are served Lift analysis on
targeting control groups to both groups brand metrics
definitition are generated
Test Group
Audiences
targeting
Control Group
Brand lift specificities
Eligibility delivering, after which you will not be able to edit
your brand lift test.
Eligibility depends on multiple factors and
availability is not guaranteed. Can my campaign be edited or paused after my
brand lift test is running?
• Your test should run for at least 14 days
for self served studies and at least 7 days We highly recommend that you hold off on making
recommended for managed studies any changes while your test campaigns are still
• Your total account budget should be important running. Introducing changes to the campaigns
and relative to your campaign duration. For while they’re still running increases the possibility
self served, 28k USD are required (France), for that your results won’t be meaningful.
managed studies you need to contact your How do I access my brand lift test results?
Facebook Representative
Depending on the type of study runned, there
• Creatives should contain your brand name
are 2 differents ways to access to your brand
or logo in the forefront or the first 3 sec
lift results: For managed study, your Facebook
(if it is a video)
Representative will be notified with updates
• For self-served studies, your campaigns’
while the poll is running. When the poll is over,
audience should include people older than
results will be ready to view within 48 hours.
18, because Facebook doesn’t survey people
Your Facebook Representative will notify you
under 18. You may ask for an exception if you
that your report is ready and share. For self-
use managed studies.
served study, the results can be found in the
“experiment” section of Ads Manager.
Questions frequently asked
Can I edit my brand lift test after I set it up? Best practices to get the best results
Your Facebook Representative can edit You can set up Brand lift study at the Ad account
(question, set up..) or cancel your brand lift test or campaign level depending on your client
up until shortly before the campaign starts strategy to better manage audience overlap.
Test, learn and earn 14Brand lift study:
checklist
#1 #5
Confirm your test plan or objective Confirm your budget and estimated power
Align with the underlying brand questions Your chosen ad campaign(s), budget and holdout
or objectives you’re hoping to answer when size contribute to the estimated power calculation
designing your test. Your overall test plan or test for your test. You can try to use past ad
objective should align with this as well. campaign(s) or similar campaigns in your industry
as a point of reference, but every test is different.
#2
Align your test audience and objective #6
Your chosen test audience should align with what
Verify your poll questions and
you’re trying to learn.
chosen response
You can include a wide range of poll questions in
Example:
a Brand Lift test. At least one poll question must
If you’re trying to prospect a new potential audience
be an ad recall question, but you can choose
strategy in a multi-cell test, each audience should minimize
overlap for the best comparative opportunity. If you’re a range of other questions depending on your
running other tests simultaneously, you may also want to business needs.
consider minimizing overlap between your audiences for
each test group depending on your test objectives
#7
Align your test duration and
#3 test objectives
Confirm your creative meets Brand Lift test can only measure what occurs
facebook best practices during the test duration, the period during (and
after) ad campaign(s) delivery where your test
Get familiar with mobile based best practices to
will observe and calculate activity for people in
use when creating your ad campaign, such
the test and control groups.
as having a prominent brand and logo and
clear call-to-action, Since a Brand Lift test is Depending on your type of business and your
measuring metrics around brand awareness and test objective, the post-campaign observation
recall, your ad should highlight the brand you’re period may need to be longer or shorter to reflect
trying to measure in your test to get the best the time needed to measure the appropriate
understanding of your campaign’s impact. brand response
#4 #8
Confirm your holdout size Finish all pending edits to test
Choosing the best holdout size depends on
ad campaigns
expected performance for your audience and It’s critical for no changes to occur to your
campaign type, but should balance between Facebook campaigns tied to your Brand Lift study
the needs for a powerful test configuration while it is in flight. This includes any change to the
and minimize the holdout’s potential impact on test configuration, any change to the campaign(s)
campaign performance. you’re using for your test and planning to run your
campaign for the entire duration.
Test, learn and earn 15Conversion lift studies
Conversion lifts will help you measure the incremental impact on your campaigns on
conversion items. They can be declined in single or multicell tests.
Conversion lift study methodology
1. Randomise Groups 2. Deliver ads 3. Track sales outcomes 4. Analyse Lift
Facebook
ads
Test
Control Compare difference
No Facebook between groups to
ads determine lift
1. 2. 3. 4. 5.
Identify the audience Facebook randomises Ads are delivered You pass your conversion Facebook compares the
you want to target and your audience, then to the test group. data to Facebook via conversions in the test
the business objective separates people into the Facebook pixel(s), and control groups to
you want to measure. test and control groups. a secure data upload or calculate lift, then post
app events. results in Ads Manager.
Conversion lift study specificities
Eligibility What ad formats and platforms does conversion
lift support?
Eligibility is dependent on multiple factors and
availability is not guaranteed. Conversion lift supports all ad formats and
platforms supported by Facebook, including
• Your test should run for at least 3 days for
Facebook, Instagram and Audience Network.
managed (if volume meets requirement) ad
7 days for self-served but we recommend to How does conversion lift measure sales and
make it last longer other conversions?
• At least 100 conversions should be registered Conversion lift leverages the data sources that
in each cell (one cell if single-cell / two & more you’re already using to measure and optimise
if multi-cell) on Facebook. This includes the Facebook pixel
for online actions, the Facebook SDK for in-app
• Your creative should contain your brand name
events, conversion API and the Facebook offline
or logo
events manager for offline actions.
Questions frequently asked
How can I view results from my test?
Is there a self-serve way to create lift tests?
The results of your study are available in Ads
Yes, Facebook offer advertisers the option to Manager. Your Facebook contact will also share
create lift studies for key business questions a dedicated report and follow up with you to
through Test and Learn. discuss the results in greater detail.
Test, learn and earn 16Conversion lift studies:
checklist
#1 We recommend:
Reaffirm your test plan or objective At least 90% estimated power for your
Conversion Lift tests are most effective when they Conversion Lift test or
are designed to answer a specific question related A budget significant enough for At least 50 (or
to a test plan or objective; something that you can ideally 100) weekly actions for each event you’re
confirm or deny from the results of your test and planning to observe for your Conversion Lift test.
then act on to improve your advertising efforts.
#5
#2 Verify all budget-appropriate conversion
Minimize variables and align test events are included
audience relevance
You can include a wide range of conversion event
Test results are more actionable when results types in a Conversion Lift test. However, any
for each cell (or test group) are based on a single conversion events included must plan to observe
test variable or dimension. at least 100 events in the test group or in each cell
of your test, otherwise you won’t be able to see
Example:
If you’re running a multi-cell test, choosing a clear variable results for those conversion events.
within each cell or group will help keep results clear and
You can estimate which events will see more than
easier to interpret. If you create cells or groups with
100 observed events based on past performance
multiple changes, each with their own unique combination
of placements, bidding strategies, and creatives, then it be in your ad campaign(s) for those event types.
difficult to identify business recommendations from your
best performing cell or group.
#6
Align your test duration
and test objectives
#3
Confirm your holdout size A Conversion Lift test can only measure what
occurs during the test duration, the period during
For self-served conversion lift, there is a required
(and after) ad campaign(s) delivery where your
holdout size of 10% that isn’t customizable. For
test will observe and calculate activity for people
managed studies, holdout threshold is 50%
in the test and control groups.
and is customizable. Choosing the best holdout
size depends on expected performance for We recommend tests run long enough to account
your audience and campaign type, but should for at least 1-2 customer conversion cycles, or at
balance between the needs for a powerful test least 2 weeks.
configuration and minimize holdout’s potential
impact on campaign performance.
#7
Finish all pending edits to test ad campaigns
#4 It’s critical for very few changes to occur to
Confirm your budget and estimated power your conversion Lift test while it is in flight. This
Your chosen ad campaign(s), budget and includes any changes to the ad campaign(s)
holdout size contribute to the estimated power you’re using for your test and planning to run
calculation for your test. You can try to use past ad your campaign for the entire duration.
campaign(s) or similar campaigns in your industry
as a point of reference, but every test is different.
Test, learn and earn 17Precision about
holdout groups ?
Single-cells Single cell conversion lift studies
There are two types of single-cell studies you can Single-cell conversion lift study can help you
implement: understand the global impact of your strategy on
conversions, be they traffic or sales actions.
Brand lift study: if you want to measure the
impact of your ad on brand metrics such as ad Conversion lift will compare conversions between
recall, familiarity, consideration… your audience exposed and the hold out to
measure the lift. Contrary to brand lift, there is no
Conversion lift: if you want to measure the impact
survey question.
of your ad on lower funnel items such as view
contents, add to carts or sales.
Did my campaign drive incremental conversions?
Both test results are established in comparison to
a hold out group, meaning that they both relate
to incremental results. Our booklet will have a INCREMENTALITY
greater focus on multi-cell lifts.
CONVERSIONS
Single cell brand lift studies
CAMPAIGN
Single-cell brand lift study can help you
understand the global impact of your strategy on HOLD OUT
brand metrics. Single-cell studies compare people
exposed to your ad to people that are not. But it
doesn’t compare two different strategies. Keep in mind that single-cell studies will show
the global incremental impact of your campaign.
However, it is not showing which parameter made
Did my campaign drive incremental awareness?
tour campaign perform or not. Therefore, you will
need to go multicell and test different hypothesis.
INCREMENTALITY
SURVEY
CAMPAIGN
HOLD OUT
The specificity of the brand lift study is the
inclusion of survey questions chosen by the
advertiser. The answers to this poll are used to
measure lift points between exposed audience
and hold out. Ideally, each poll question will
receive at least 500 responses, though some may
require up to 1,000 responses (such as standard
favorability and action intent questions).
Test, learn and earn 18Multi-cell & split tests
There are two types of split tests: Multi-cell test (conversion or brand lift)
• Tests without incrementality: A/B tests Multicell testing includes a hold out group,
that is an audience that is not exposed to your
• Tests with incrementality: tests with holdout
ad. This control group enables to measure
groups, that can be either brand lift or
incremental results.
conversion lift studies.
Which version will drive the most incremental sales?
Best practices to get the best results:
Incrementality
The easiest way to validate A/B test set up is to make sure
clients are spending enough to get 50 conversions per cell
if it’s an A/B test without holdout or 250 conversions per
VERSION A VERSION B
cell if it’s A/B test with holdout.
HOLD OUT HOLD OUT
A/B test
A/B test is a simple randomization of your variable
in order to define the best version of it. After
Why are incremental results key insights?
choosing the variable you want to test, Facebook Multicell is useful to understand how your ad truly
will divide your budget equally and randomly split impacts your business.
your intended audience between each version of
Let’s take the example of two audiences:
your creative, audience, or placement.
Audience A: retargeting of your visitors, very engaged
Which version will drive the best CTR? Audience B: 35+, wide audience
No incrementality It is very likely that audience A would get better absolute
results with a simple A/B test. However, you might
drive more incremental sales on audience B, because
VERSION A VERSION B audience A would have bought your product anyway.
Test, learn and earn 19Brand lift Case example
A TV industry company used the Facebook brand lift single-cell
study to measure 3 items: advertising memorization, awareness
and purchase intention.
This campaign had positive results on brand metrics:
Advertising recall: Brand awareness: Intent to buy:
15.3 pts 8.5 pts 1.4 pts
(benchmark 6.4pts) (benchmark 2.9pts) (benchmark 1.4pts)
3.41M +15.3pts +595%
Of impacted users Ad recall Product page views
Context:
Date: 5 June – 24 June 2020 Total Reach Impressions Frequency
Objective: Conversions 3.41M 11.9M 3.48
Audience: France, 18 yo + Interest
Products used: Formats:
Instant Experience Facebook Feed
Source: Converteo Client case study
Test, learn and earn 20Brand & Conversion
lift Case example
A telephone company used the Facebook multi-cell brand lift and
conversion lift study to measure not only branding metrics items
but also performance.
This campaign was organized in 2 phases:
A teasing phase from 23/07 to 05/08 (BLS)
A pre-order phase from 05/08 to 21/08 (CLS)
NOT
+4.8 pts SIGNIFICATIVE +4.2%
Standard Ad Recall Attribute Rating Preference
+98% +66.1% +24.5%
Impact view content Shopping card Buy impact
Context:
Date: 23 July – 21 August 2020 Total Reach Impressions Frequency
Objective: Video views, Reach, Lead 10.1M 84.8M 8.4
Audience: France, 18 yo + Interest Techno
Products used:
Formats
Videos Facebook Feed
Page post link
Source: Converteo Client case study
Test, learn and earn 21Build
Presentation of scorecards dedicated to specific tests.
11 tests are presented here to enhance creatives,
audience management, set-up and media planning.
Test, learn and earn 22Build section guide
1
2
3
4
1 Criteria: indicates the type of test we are mentioning
1 Type of test: we focus on 4 types: Audience, Creative, Set-up optimization
and Media orchestration
2 Funnel step: to implement for branding and/or consideration and/or
conversion campaigns
3 Difficulty level: advises complexity level regarding advertiser maturity
4 Business impact: helps assess weigh of expected business impact
5 Available test set-up: single and/or multi-cell Brand and/or Conversion studies
2 Set-up guide #1: presents operational context and requirements
3 Set-up guide #2: helps roll-out by shaping hypothesis and gauging business impact
4 Recommendation: best practices and traps, shaped by our experience advising our
clients on Facebook strategy and implementation
Test, learn and earn 235 best practices to
become a master tester
Use your Facebook strategy to define
1
your testing roadmap
The best way to define your own testing strategy is to focus on
your Facebook business highlights.
Gather your A-team before test implementation
2 Depending on expected test impact, gather a team or at least ask
for fresh eye on your test design: head of acquisition, data analysts,
data scientists or marketing expert.
Define a clear hypothesis
3 It is of utmost importance to clearly define what your are testing
(test frame, dates flights, campaign objective…) to avoid being
disappointed after spending money and time on it.
Stress out bias during post-test analysis
4
Do not rush into conclusion as campaigns results depend on a
multiplicity of external and internal factors, such as economic
situation (lockdown), campaign set-up, period of year, technical site
and pixel implementation...
Do not put all your hope in one single test
5 As stated, you might need to re-run your test to rule out some
bias. Keep in mind that hypothesis definition is key to get relevant
insights from your test.
Test, learn and earn 24Facebook maturity
assessment
Assessing Facebook maturity helps you select test difficulty level to avoid
commiting in a test you operationally can’t run or won’t bring you expected insights.
You can assess it considering three main topics:
Investment Deployed strategy Facebook stack
• How much do I invest on a • Do I consider I get the • Do I have a Facebook
yearly basis? most of Facebook pixel / SDK with solid
• Am I able to meet Brand Business Manager? implementation and
lift / Conversion lift reach • Do I run both ongoing and troubleshooting?
and budget requirement? one shot campaigns? • Do I use offline
• Am I assisted by a • Am I planning campaigns conversions (if relevant)?
Facebook representative? on multiple Facebook • Do I use Facebook
objectives / funnel steps? attribution?
Preferred Difficulty: Easy Preferred Difficulty: Medium Preferred Difficulty: Complex
Low Maturity Moderate Maturity High Maturity
Starter Pack Starter Pack Starter Pack
You can consider you have a You can consider you have a You can consider you have
low Facebook maturity if you moderate Facebook maturity a high Facebook maturity if
match one or several of the if you match one or several of you match one or several of
following items: the following items: the following items:
• You are a Facebook • You have good command • You are Facebook
campaigns beginner of Facebook ad platform campaigns expert
• You have not implemented • You have implemented • You are responsible for
pixel or app software the pixel high yearly budgets
development kit • You have run a few • You have implemented
• You have never run a lift test tests already offline conversions and/or
• You are not acquainted • You have struggles at Facebook attribution
with incrementality topics times to be more efficient • You have already run
• You have struggled at in Facebook campaign many tests
times to find the best management • You would like to go one
campaign set-up step further with testing
strategy
If you have low Facebook
maturity, you should consider
starting with A/B tests
Test, learn and earn 25Tests matrix
We have built this matrix to help you organize your testing approach, but knowing
that every advertiser is indubitably unique, we recommend playing with it. We also
recommend discussing it with your Facebook representative, your media, consulting
and/or creative agency until you find your own testing roadmap.
Low maturity Moderate maturity High maturity
Difficulty Business
Case type Case advertisers advertisers advertisers
level Impact
priority priority priority
A1 - What is the best
Scenario X : 1 Scenario X : 2
A - Creative branding creative Easy Moderate 1
Scenario Y : 2 Scenario Y : 3
format?
A2 - What is the best
Scenario X : 2 Scenario X : 3
A - Creative creative format to Medium Moderate
Scenario Y : 1 Scenario Y : 2
generate conversions ?
A3 - What is the best
Scenario X : 3 Scenario X : 1
A - Creative creative strategy to Medium Moderate
Scenario Y : 3 Scenario Y : 1
reach my different goals?
B1 - Which audience
Scenario X : 1 Scenario X : 1
B - Audience is the best to increase Easy Moderate 2
Scenario Y : 2 Scenario Y : 2
brand awareness?
B2 - Which audience
Scenario X : 2 Scenario X : 2
B - Audience is the best to generate Medium Moderate
Scenario Y : 1 Scenario Y : 1
incremental sales?
C - Set-up C1 - Which campaign Scenario X : 1
Easy Moderate 3 3
optimization type should I implement? Scenario Y : 2
C2 - How to get the most
C - Set-up Scenario X : 2
of my campaign set-up Medium Big 2
optimization Scenario Y : 1
with my agency?
C3 - Is my current
C - Set-up
campaign set-up fully Complex Big 1
optimization
exploited?
D1 - What is the optimal
D - Media Scenario X : 2
media frequency to drive Medium Low 2
orchestration Scenario Y : 2
awareness?
D2 - What is the best
D - Media
media phasing to reach Complex Moderate 3
orchestration
my goal?
D3 - What is the optimal
D - Media Scenario X : 1
budget for each phase of Medium Big 1
orchestration Scenario Y : 1
my media strategy ?
Test, learn and earn 26Notes: Low maturity advertisers: we have discarded
all conversion-related topics as low maturity
First step is to define your main focus based
might be defined by the absence of conversion
on your Facebook strategy and needs. We only
pixel implemented
have rated priorities among each case type,
not between all cases. Indeed, we deem that Moderate maturity advertisers: at least two
this would lead to huge misinterpretation as scenarios are conceivable (of course you can have
we cannot tell you set-up optimization is more similar expectations and questions regarding
important than creative one or reverse way: it your brand and conversion campaigns, in this
depends on your business and your brand. case prioritization shall be made differently):
Keep in mind that you can mix case types to Scenario X:
create your test priorities: for instance it is focused on Brand campaigns and testing
possible to choose the following testing approach: Scenario Y:
focused on Conversion campaigns and testing
1st test: D3
2nd test: A2 High maturity advertisers: we took for granted
3rd test: B2 that this type of advertisers focus on high
business impact and better budget allocation
and that they could have already run most of
the more basic tests. If not, please consider a
different prioritization
Test, learn and earn 27A - Creative
Creative tests rely on creative shape (format, video
length…) as well as content (message, customization…)
or use (placement, broadcast platform…)
Test, learn and earn 28A1 - What is the best branding creative format?
Criteria: Creative
Branding Consideration Conversion
Easy Medium Complex
Low Impact Moderate Impact Big Impact
Single-Cell BLS Multi-Cell BLS Single-Cell CLS Multi-Cell CLS
Set-up Guide
Context Objective Stakeholders
You want to go further than A/B Test different types of creative • Traffic manager
test and measure the impacts of formats to define the best performing • Creative Agency
your strategies on your audience one(s) in terms of brand awareness • Communication planner
• Facebook representative
Facebook Stack Requirements Test implementation
• Check budget and reach Define ahead:
requirement with your • Duration
Facebook representative • Budget
• Set up a frequency per week • Frequency
that will be relevant to analyze • Creative
the results • Auction type
• Prepare Brand Lift • Placement
Hypothesis Business Impact Test examples
Creative A improves the ad recall/ Testing creatives will help you • Is a video improving
the message recall/consideration/… better define your Facebook consideration more
more than creative B branding strategies, depending than a carrousel?
on the section you want to focus • Is a long video better than a
on (ad recall, message recall, short video for ad recall?
consideration, preference)
Question to go further: Is format or message more influential on my results?
Recommendations
Best Practices Traps to avoid
• Choose creatives adapted to social media Bias:
consumption habits and not only a cut of the TV ad. Test results depend on multiple factors: periof of year,
• If you use video creatives, make sure that: bid type, optimization event…
• The brand is visible within 3 secs Limits:
• The key message is integrated within 5 secs When analyzing results, remember that a strategy could
• You ad is clear and effective sound off be inefficient for a specific goal but useful and effective
• Define clear and relevant questions which help you for another goal
to go further in your analysis
See also: Long-term brand lift (Page 50). If any constraint is preventing you from doing brand lift test, you can also compare creative strategies
through A/B testing (Page 18)
Test, learn and earn 29A2 - What is the best creative format to generate conversions?
Criteria: Creative
Branding Consideration Conversion
Easy Medium Complex
Low Impact Moderate Impact Big Impact
Single-Cell BLS Multi-Cell BLS Single-Cell CLS Multi-Cell CLS
Set-up Guide
Context Objective Stakeholders
• Your campaign includes Test different types of creative • Traffic manager
different types of creatives formats to define the best • Creative Agency
• You want to go further than performing one(s) in terms • Communication planner
A/B test and get rid of classic of conversions • Facebook representative
attribution model analysis
Facebook Stack Requirements Test implementation
• Conversion lift: Pixel or SDK Check budget, reach and holdout Define ahead:
• Nice-to-have: Offline Conversions requirement with your • Duration • Creative
Facebook representative • Budget • Auction type
• Frequency • Placement
Hypothesis Business Impact Test examples
Creative A is generating more sales Define a cost-performing media • Is a video generating more sales
than creative B strategy using the most relevant than a carrousel?
creatives that drive business results • Is a customized ad better than
a traditional ad to generate
conversions?
• Is DPA better than traditional
ad to drive sales?
Step to go further: You can test your creatives with specific audiences to analyze
and pick up the most effective ones for each audience
Recommendations
Best Practices Traps to avoid
• Choose creatives adapted to social media Bias:
consumption habits Test results depend on multiple factors: period of year,
• If you use video creatives, make sure that: bid type, optimization event…
• the brand is visible within 3 secs Limits:
• The key message is integrated within 5 secs When analyzing results, remember that an audience
• You ad is clear and effective sound off could be inefficient for a specific goal but useful and
effective for other ones
See also: Offline conversions (Page 48). If any constraint is preventing you from doing conversion lift test, you can also compare creative
strategies through A/B testing (Page 18)
Test, learn and earn 30What is the best creative strategy to reach my different goals?
Criteria: Creative
Branding Consideration Conversion
Easy Medium Complex
Low Impact Moderate Impact Big Impact
Single-Cell BLS Multi-Cell BLS Single-Cell CLS Multi-Cell CLS
Set-up Guide
Context Objective Stakeholders
• Your campaigns include Test different types of creative mix • Head of acquisition
different types of creatives (format, message, placement…) to • Traffic manager
and different phases: branding,
define the best performing ones • Creative Agency
consideration, conversions
• Communication planner
• You need insights about your
• Facebook representative
creative mix
Facebook Stack Requirements Test implementation
• Conversion lift: Pixel or SDK Check budget and reach Define ahead:
requirement with your • Duration • Creative
Facebook representative • Budget • Auction type
• Frequency • Placement
Hypothesis Business Impact Test examples
• Message A is driving more Define a media strategy using the • Is placement optimization
traffic than creative B most relevant creative strategy better than classic placement
• Placement mix had better that fits your goals to generate sales?
performances than dedicated • Should I advertise on both
placements according to Facebook+ Instagram or
campaign goal Instagram only?
• Is personalized message more
efficient than standard one to
drive awareness?
Step to go further: You can test your creatives with different audiences to analyze
if the behaviors are the same for each audience
Recommendations
Best Practices Traps to avoid
• Choose creatives adapted to social media Bias:
consumption habits Test results depend on multiple factors: period of year,
• If you use video creatives, make sure that: bid type, optimization event…
• the brand is visible within 3 secs Limits:
• The key message is integrated within 5 secs When analyzing results, remember that an audience
• You ad is clear and effective sound off could be inefficient for a specific goal but useful and
effective for other ones
See also: Long-term brand lift (Page 50). Offline conversion (Page 48)
Test, learn and earn 31B - Audience
Audience tests help shaping your own best target
audiences according to your goals and stakes
Test, learn and earn 32Which audience is the best to increase brand awareness?
Criteria: Audience
Branding Consideration Conversion
Easy Medium Complex
Low Impact Moderate Impact Big Impact
Single-Cell BLS Multi-Cell BLS Single-Cell CLS Multi-Cell CLS
Set-up Guide
Context Objective Stakeholders
You are not sure you target the best Define the most accurate audience • Head of acquisition
audience to drive brand awareness strategy for awareness purpose • Traffic manager
• Facebook representative
Facebook Stack Requirements Test implementation
• Check budget and reach Define ahead:
requirement with your • Tested • Frequency
Facebook representative audiences • Creative
• Prepare Brand Lift • Duration • Auction type
• Budget • Placement
Hypothesis Business Impact Test examples
Audience A (encompassing users Define a media strategy using the • Does audience A (encompassing
with specific tastes, more likely most relevant audience to improve users with specific tastes,
to fit your products/campaign/ your branding and consideration more likely to fit your products/
brand) will generate more sales/ campaign/brand) generate more
awareness than audience B (broad, sales than audience B (broad,
generic audience) generic audience)?
• Is it better to trust Facebook
algorithm for brand awareness
by creative a big audience in one
adset than multiple ones?
• Is affinity audience more
performing than standard one for a
product launch to raise awareness?
Question to go further: Are result only due to your audience? Or is it better to adopt a several
levers strategy: audience / creative / product?
Recommendations
Best Practices Traps to avoid
Make sure the audience is the only parameter that varies Bias: Test results depend on multiple factors: period of
between the 2 cells year, bid type, optimization event…
Try to avoid other communications at the same time
Limits: When analyzing results, remember that an
audience could be inefficient for a specific goal but
useful and effective for other ones
See also: Long-term brand lift (Page 50). If any constraint is preventing you from doing brand lift test, you can also compare audience strategies
through A/B testing (Page 18)
Test, learn and earn 33Which audience is the best to generate incremental sales?
Criteria: Audience
Branding Consideration Conversion
Easy Medium Complex
Low Impact Moderate Impact Big Impact
Single-Cell BLS Multi-Cell BLS Single-Cell CLS Multi-Cell CLS
Set-up Guide
Context Objective Stakeholders
Your campaign goal is to increase Test different types of audience to • Head of acquisition
sales, regardless of branding and define the best performing ones in • Traffic manager
consideration matters terms of conversions • Facebook representative
Facebook Stack Requirements Test implementation
Facebook Pixel Check budget and reach Define ahead:
requirement with your Facebook • Tested • Frequency
representative audiences • Creative
• Duration • Auction type
• Budget • Placement
Hypothesis Business Impact Test examples
• Audience A improves conversion Define a media strategy using • Is it better to target a specific
more than audience B the most relevant audience that narrow audience or a wide one?
• Audience A will generate increase your sales the most? • Is lookalike an efficient audience
incremental sales if it’s combined to drive sales?
with its lookalike audience • For specific audience, what is
the most performing one (1st
party, Facebook data…) in terms
of incremental sales?
Question to go further: Are result only due to your audience? Or is it better to adopt a several
levers strategy: audience / creative / product?
Recommendations
Best Practices Traps to avoid
Facebook offers a native audience optimization through Bias: Test results depend on multiple factors: period of
its own algorithm year, bid type, optimization event…
Limits: When analyzing results, remember that an
audience could be inefficient for a specific goal but
useful and effective for other ones
See also: Offline conversions (Page 48). If any constraint is preventing you from doing conversion lift test, you can also compare audience
strategies through A/B testing (Page 18)
Test, learn and earn 34C - Set-up
Optimization
Set-up optimization encompasses all dimensions you
can play on when setting-up your campaigns including
campaign type or various adset fields selection
Test, learn and earn 35Which campaign type should I implement?
Criteria: Set-up Optimization
Branding Consideration Conversion
Easy Medium Complex
Low Impact Moderate Impact Big Impact
Single-Cell BLS Multi-Cell BLS Single-Cell CLS Multi-Cell CLS
Set-up Guide
Context Objective Stakeholders
You want to know when to Test campaign type to define the • Traffic manager
implement R&F or classic campaigns best performing one in terms of • Facebook representative
brand awareness or conversion
Facebook Stack Requirements Test implementation
Conversion lift: Pixel or SDK for Check budget and reach Define ahead:
conversion campaigns requirement with your • Duration • Placement
Facebook representative • Budget • Auction type
• Creative
For R&F campaigns:
• Reach / • Frequency
Target
coverage
Hypothesis Business Impact Test examples
Classic campaigns perform Campaign performance depend • It is better to do classic
better than R&F campaign on multiple factors, including campaigns or R&F ones?
as you can optimize them when campaign type. R&F campaigns • Should I implement R&F during
they are running are preferred for busy period but busy period (Black Friday,
testing it would allow to see if it is Sales, Christmas…)?
really relevant. • Should I choose beetween R&F
and classic campaigns or is it
more interesting to mix them?
Question to go further: Is there a huge CPM difference between my campaign types
(and what are its consequences)?
Recommendations
Best Practices Traps to avoid
Set-up your R&F campaigns soon enough (at least 2 Bias: Avoid over-optimizing your classic campaigns as you
weeks before start) to benefit from cheaper CPM may get better results than what you usually would have
Limits: When setting-up an R&F campaign you need to
choose what to set-up before the campaign, it could be
either:
• Reach, Target coverage, Frequency
For instance, choosing to optimize on reach or on
frequency would not generate same outputs
Test, learn and earn 36How to get the most of my campaign set-up with my agency ?
Criteria: Set-up Optimization
Branding Consideration Conversion
Easy Medium Complex
Low Impact Moderate Impact Big Impact
Single-Cell BLS Multi-Cell BLS Single-Cell CLS Multi-Cell CLS
Set-up Guide
Context Objective Stakeholders
Overcome « Facebook is a black Measure algorithm optimization vs. • Head of digital marketing
box » opinion manual one • Head of acquisition
• Traffic manager
• Facebook representative
Facebook Stack Requirements Test implementation
Conversion lift: Pixel or SDK Check budget and reach Define ahead:
requirement with your Facebook • Duration • Auction type
representative • Budget • Placement
• Creative
Hypothesis Business Impact Test examples
Agency manual optimization helps Would help you review budget • Do I earn more money with
me get the best performance out allocation, time spent on manual campaign optimization’s
of my money optimization vs. strategic projects performance increase or with
or set-up agency budget cut?
• Is it better to let algorithm
do the job or to put manual
optimization as much as I can?
• Should manual optimization be
focused on defined topics (to
identify through the test)?
Question to go further: What should be time allocation for each task of my agency/trading team?
Recommendations
Best Practices Traps to avoid
Do not rush at shaping conclusion from this kind of test, • Avoid over-optimizing your campaigns for manual
it would be interesting to run several tests before optimization as you may get better results than
taking decision what you usually would be
• Test results depend on multiple factors: period of
year, bid type, optimization event…
See also: Cross-publisher test (Page 46), Facebook attribution (Page 47), MMM (Page 48)
Test, learn and earn 37Is my current campaign set-up fully exploited?
Criteria: Set-up Optimization
Branding Consideration Conversion
Easy Medium Complex
Low Impact Moderate Impact Big Impact
Single-Cell BLS Multi-Cell BLS Single-Cell CLS Multi-Cell CLS
Set-up Guide
Context Objective Stakeholders
• You want to know when to try Test various campaign • Head of acquisition
different campaign set-up to get optimization to define the best • Traffic manager
the most performance from it performing one in terms of brand • Facebook representative
• Understand Facebook « signal » use awareness or conversion
Facebook Stack Requirements Test implementation
• Conversion lift: Pixel or SDK Check budget and reach Define ahead:
• If relevant: Offline conversions requirement with your • Duration • Auction type
Facebook representative • Budget • Placement
• Creative
Hypothesis Business Impact Test examples
It is better to optimize my For instance in car industry, you • Should I optimize on an event
campaigns on the last online could get a lot of leads not leading which occurs a lot (leads) rather
touchpoint instead of other signals? to in-store conversions. Changing than the one providing real value
the event you optimize on and but few occurrence (sales)?
providing it multiple signals might • Is is better to optimize my
help algorithms identify your most conversion campaigns on X
promising leads and raise your or Y event?
global Facebook ads performance • Is it better to optimize on offline
or online event regarding my
main goal?
Question to go further: Could I integrate more signals to help algorithm optimization?
Recommendations
Best Practices Traps to avoid
Before starting test, identify your main marketing goal Limits:
vs. the event your are optimizing on You shouldn’t switch to another optimization event if:
• Think about the volume of events your have for your • You want to switch to an offline event but can’t
optimization event upload them often
• If interesting to implement offline event to get more • You want to switch to an event which hasn’t got
signals, an offline event pixel set-up should be done enough signals to enhance algorithmic optimization
to begin using it
See also: Offline conversion (Page 48)
Test, learn and earn 38D - Media
Planning
Media planning is the core structure of your media
strategy. It is about choosing the right budget allocation
and the right timing to achieve your strategic goals.
Test, learn and earn 39What is the optimal frequency to drive brand awareness?
Criteria: Media Orchestration
Branding Consideration Conversion
Easy Medium Complex
Low Impact Moderate Impact Big Impact
Single-Cell BLS Multi-Cell BLS Single-Cell CLS Multi-Cell CLS
Set-up Guide
Context Objective Stakeholders
You want to drive brand Understand which campaign • Traffic manager
awareness through reach frequency will drive more awareness • Head of acquisition
& frequency campaign • Data analyst
• Facebook representative
Facebook Stack Requirements Test implementation
• Check budget and reach Define ahead:
requirement with your • Duration types (R&F)
Facebook representative • Creatives • Placement
• Prepare Brand Lift • Auction • Audiences
Hypothesis Business Impact Test examples
Frequency of 3/week will drive • Manage the balance between Is higher frequency really more
more awareness than 1/week reach and frequency and adapt efficient at driving awareness?
your budget accordingly
• Refine advertising pressure
and therefore manage your
brand image
Question to go further: What is the accurate budget allocation for branding phase?
/ What is the best timing to run branding phase?
Recommendations
Best Practices Traps to avoid
Make sure frequencies are significantly different Bias:
between the two cells to get real insights • Test results depend on multiple factors: period of
year, bid type, optimization event…
• Try to avoid other communications (even on other
channels) at the same time.
Limits:
When analyzing results, remember that a strategy
could be the best for a specific goal but results could be
different for another goal. For instance, a high frequency
can help you raise ad recall points but erode purchase
intention due to high advertising pressure.
Test, learn and earn 40You can also read
