A key step in evaluating activity is establishing robust test and control cells.  Here, Darren Burnett, head of data planning at Elvis (Campaign and Marketing's Direct Agency of the Year 2008), provides more detail on how to implement them.

Isolate individual test elements

A number of factors influence consumer behaviour – targeting, propositions, offers, formats, the creative itself and so on. A well-structured test controls all factors outside the element being tested to deliver meaningful results. Each variable should be tested in isolation once within the test design. 

Recognise when it is appropriate to combine factors in a single test. As with any good rule, there are occasional exceptions. There are rare times when it is impossible or inappropriate to separate individual test variables because they are inextricably linked. However, it is important to be honest about whether this is a genuine requirement or just lazy test design.

Be clear about the impact of this test. Why should this combination of variables be judged as a single test element?  On success, will you be able to roll out the tested combination, or will you question which element was truly responsible?

For example: a proposition has, over time, been tailored to be relevant to the core target audience – to simplify, let’s say people aged 50-65. 

Market analysis has identified an opportunity in the under-30s, but the existing proposition is unsuitable. So a new proposition must be tested at the same time as the new audience. This means accepting that it will be unclear how much each element is responsible for the difference in performance compared with the existing champion (breaking our core rule).

However, the longer-term plan should be to test alternative propositions to identify the strongest for this audience – or whether something more generic can be developed that will resonate with both.

Don’t waste budget duplicating tests

As covered in the measurement overview, it is essential to ensure that your control cells are sized to deliver a statistically robust result. This is not as simple as arbitrarily assigning X% of your available volume. The volume required will depend on predicted response rates and how confident you need to be in the result. Speak to your analytical support or your agency about how to calculate these volumes.

As cells are created to test each factor in isolation, your test design will swiftly become very complicated.  Exploring two propositions, offers, formats and creative routes, will generate 16 test cells (2x2x2x2). It is all too easy to end up with a completely unwieldy test design that is difficult to implement and – worse – wastes budget, as valuable contacts are allocated to ‘purist’ test cells that are unnecessary or irrelevant.

While it is critical that each factor is tested in isolation, in most cases it need only be tested once. Review your plan carefully. Are there any inappropriate permutations? For example, a tailored creative tested against the wrong audience. Are any factors being tested multiple times?  Then remove any unnecessary cells.

For example: champion Proposition A is to be tested against new Proposition B, and champion Format 1 against the more interesting, but more expensive, Format 2.

This generates four cells: 

Cell 1: A1 (Proposition A and Format 1) 
Cell 2: A2
Cell 3: B1 
Cell 4: B2

However, both elements are being tested twice. The winning proposition is identified by comparing both cells 1 and 3 AND cells 2 and 4; format with cells 1 and 2 AND 3 and 4.

Removing one cell still provides all the required learning. For example, perhaps we remove cell 4 as this represents the greatest risk (your untested proposition B and the more expensive format 2). 

The winning proposition can still be identified by comparing cells 1 and 3 and formats through cells 1 and 2. However, required test volumes have dropped significantly.

Validate your final plan by listing the cell comparison for each test. This helps identify any remaining duplicates – and whether the plan has been oversimplified, compromising the testing.

Use fallow cells to understand the total impact of your direct communications

No action control cells (in which a representative sample of contacts is withheld from communication) can be used to determine the impact of a single campaign, but how do you understand the total impact of your strategy when each communication is only part of the picture?

A fallow cell is effectively a no-action control for all direct communications (or for a particular channel, such as direct mail). A representative sample of contacts is isolated and suppressed from all communications in the programme. This cell is kept representative of the contacted pool, with an appropriate sample of new recruits being added as they come onto the database.

Over time, this cell can be compared with the contacted base to provide a holistic evaluation of the impact of your activity – including net variations in product holdings, value, retention, as well as softer measures, such as satisfaction or advocacy.

Don’t compromise the plan when it's implemented

The golden rule about every non-tested factor remaining constant across test cells must remain. Ensure you follow the campaign through implementation to identify anything that could compromise your test.

For example: a mailing is evenly split into two tranches, mailed a week apart to manage the flow of inbound responses. If one cell mails with tranche one, and a compared cell in the second, your results will be skewed – and any findings will be unreliable. This can be avoided by ensuring the phasing is consistent across each tested cell – with 50% of each cell in each tranche.

Need tips on evaluating the results? Find out how to successfully evaluate your campaign here