measures of success

fixing marketing measurement

John posted a great comment in response to a recent post:

“Measurement of [TV's] effectiveness is based on samples, not actual viewers, and often the best data you get about the audience is generalized demographic / psycho-graphic information.”

It’s no secret that I have strong opinions when it comes to measurement and research.

My concern is simple: I don’t believe we’re measuring the right things.

It’s time we changed that.

This post outlines an alternative approach to ad measurement, but it still needs some tweaking, so I’d really appreciate your suggestions on how we might improve it.

Let’s begin with some context…

The role of measurement

As early as the mid-1800s, John Wanamaker remarked:

wanamaker ad dollars quote

His fears have been echoed by marketers ever since, and we continue to invest huge sums trying to identify which of our dollars are wasted.

However, this focus on wastage means we’ve been missing the forest for the trees; in order to understand how hard investments are working, we first need to understand whether campaigns are delivering on our objectives.

Advertising success is not just about efficiency; we also need to measure its effectiveness.

To examine these factors in context, we first need to understand  the objectives we hope to address with advertising – in other words, what do we want our brand communications to achieve?

Why do we communicate?

At a fundamental level, communication serves a very simple purpose:

To create a shared understanding between two or more people

It follows that the purpose of brand communications is:

To create a shared understanding between a brand and the people it wishes to influence.

So, in order to measure advertising’s effectiveness, we simply need to determine whether the audience has understood what the brand intended.

To measure the campaign’s efficiency, we need to compare the proportion of the audience that correctly understood  the message with the different campaign elements they’ve experienced, and the cost of those different elements.

In light of the above definitions, it seems logical that both measures centre on the audience’s level of understanding.

So why do we consistently resort to metrics that have so little to do with what really matters?

Flawed measures

Reach (as media agencies currently use the term) is simply a projection of potential audience size. This is the concern that John highlighted in his comment above: reach doesn’t tell you whether anyone actually witnessed your communications, and it gives no indication of whether those who did witness them understood anything.

Meanwhile, frequency is equally limited in its value, informing us of little more than the number of opportunities each individual had to witness the campaign (but again, not telling us if they actually did see things that many times). I’ve talked about frequency’s limitations before, so I’ll avoid going into any more detail here.

The problem with these metrics is that they equate volume with success. However, the more you shout at people, the more they’ll try to ignore you.

shouting ignorance

Even ‘brand health’ metrics are compromised when it comes to determining advertising’s impact, because they tend to look at a brand’s performance in aggregate, rather than just the performance of its communications. This macro view means that we cannot determine the campaign’s influence ceteris paribus, and consequently, we cannot ascribe changes in brand health scores solely to advertising.

So what can we do to improve advertising measurement?

Start out right

A simple improvement would be to focus on objectives.

It’s vital that brands and their agency partners develop campaigns around what the brand wants its audiences to understand – although it’s a widely misused term, I’ll refer to this as ‘the message’.

Only once we’ve agreed this message can we be sure that we’re developing the most efficient and effective communications.

An example might help to put this in context.

A popular brand of face cream wants to grow revenues amongst existing users in Thailand. Traditional ‘brand health’ research shows that the brand is well liked and respected, but that its use is sporadic, with medium and light users applying the product just a few times each month.

Face-to-face conversations with these consumers reveal that they think it does a great job of rehydrating their skin, but the humid climate in Thailand means that this benefit is only relevant on a few days each month.

The brand recognises that it needs to modify people’s perceptions to help them see the brand as an everyday cosmetic, rather than simply as a functional moisturiser. In other words, it needs to tell them that it offers a bonus, and not just a remedy:

benefit baseline

Further research reveals that this audience uses moisturisers to “bring their skin back to life.” When their skin is dry, they think that it looks “flat” and “tired”, but when it’s properly hydrated, they believe it exudes a “radiant glow.” A closer look at the research indicates that this radiance is not only a critical category driver, but that the brand already scores highly on this attribute.

Brand management believes that focusing the audience’s attention on this motivating benefit will encourage them to use the brand more often. They brief a campaign that drives the perception that “using BrandX every day gives my skin a radiant glow.”

This clear statement of what the brand wants people to understand ensures that everybody works towards the same objective. Rather than simply trying to ‘raise awareness’, each agency knows what it must communicate, and can therefore optimise its approach to focus on this outcome.

Measure what matters

In order to assess the campaign’s contribution, the brand needs to measure how strongly the audience associates this statement with the brand. The research must begin well before any communication takes places, enabling the brand to identify baseline scores against which it can compare the scores obtained during and after the campaign.

It’s important to note that research should only canvass the relevant audience. Communications should always be tailored to appeal specifically to the people the brand wants to influence, so there’s little point in measuring their impact on other people.

I’ve mentioned before that I’m not a great believer in advertising pre-testing, but I recognise that many people feel more comfortable using it. If you do use it, make sure that it focuses on assessing the campaign’s ability to influence this core attitude.

Once the campaign launches, we need to measure its impact on the relevant attitude score(s).

Measuring up

The first thing we must identify is whether respondents have witnessed any of the relevant communications.

Historically, brands have placed great importance on metrics such as ‘top-of-mind awareness’ and ‘unaided recall’. However, the audience’s ability to remember the campaign doesn’t say much about whether or not they understood anything.

I’d argue that prompted research is much more indicative. For a start, significant differences between projected reach and prompted awareness suggest an area the brand will want to research further. It also provides:

A reliable indication of actual campaign reach;
A ‘control group’ of respondents who claim not to have witnessed any aspect of the campaign, whose scores we can compare to those of people who claim they have seen at least one part of it.

The research continues by showing each respondent a selection of activity from the campaign, and asking them to identify all that they’ve witnessed. This can be as simple as showing them visuals or playing them snippets of audio from different executions, and asking them to confirm whether or not they’ve seen or heard these activities before.

Your research agencies will be best placed to recommend the maximum number of examples you can show, but make sure that you include all the channels included in the campaign mix at some point in research. Don’t forget to include things like events and sponsorship if they’ve been part of the plan.

Examining scores across people who have seen a different combination of channels allows us to determine the cumulative effect of different activities. Furthermore, by comparing these results to the scores of those who have only seen a few activities, we can begin to infer the impact of specific channels and creative executions.

Comprehension

The next step is to assess whether the respondent has understood what the brand intended. The best way to measure this is to ask it as an open-ended question, such as “what did you understand after seeing this advert?

shared understanding (2)

However, this is often impractical due to respondents’ levels of commitment and involvement, and the need for researchers to record exactly what was said.

As an alternative, you can present respondents with a variety of statements about the brand and the campaign, and allow them to indicate the strength to which they agree with them. It’s best to offer a few variations on what the brand actually intended, along with some other statements that have little to do with the intended message, to allow for more reliable findings.

Once researchers have gathered all the responses, the final step involves interpreting the results. This begins by comparing pre-campaign scores, ‘control group’ scores, and the scores of people who’ve experienced different aspects of the campaign.

Following these steps with similar research a few months after the campaign has finished allows you to assess whether the campaign has permanently modified perceptions, or simply influenced shorter-term attitudes.

And that’s pretty much it: a simple, practical, but powerful approach to improving advertising measurement.

The barriers

So what’s holding us back?

Here are the most common oppositions I encounter when proposing this approach:

“It’s too difficult”
“It would cost too much”

The first point is moot; I’d argue that this process is a lot simpler than the one research companies already use when tracking brand health.

The second point relates to focus: yes, if we ran this process in addition to current measurement, it would add significant cost. However, this approach tells us almost everything we need to know about our advertising, so why would we continue with that other measurement?

A start, not a solution

I don’t pretend that this process is a panacea, and as with all aspects of marketing, you’ll need to adapt it to the specifics of your brand and its context.

However, provided you canvass a sufficient proportion of your audience with focused and relevant questions, this approach should deliver results that are far more informative than most current practices.

What do you think? Which parts need tweaking? What could be added or removed to make it better?

I’d really value your thoughts and comments – please feel free to share them, along with any questions, in the comments section below.

You may also find the following posts useful:

8 steps to better communications
Propagandata
Anjali shared some great thoughts on a similar subject in this recent post

About these ads

16 Responses to “measures of success”


  1. 1 Neil Charles September 29, 2009 at 23:19

    You’re right with wood for the trees, but I think it’s because people lose sight of the ultimate objective of almost all advertising (public information films excepted.) Sell a product.

    Another famous quote: “We sell. Or else” from David Ogilvy.

    All of these other metrics – reach, frequency, awareness… are only used because it’s very difficult to measure how much product a campaign has sold. If you could measure sales effects properly, you’d only ever use other metrics in pre-testing and planning, to try to build a campaign that was likely to work.

    The other metrics you list are needed, but we should never lose sight of the fact that more reach means I sell more product because… (who extra are you talking to?) Comprehension means I sell more product because… (now the consumer understands x and that means they’ll buy.)

    It’s far too common to hear a ‘brand campaign’ proposed, or worse… with no clear idea of why on earth it would make anybody go out and buy.

    • 2 eskimon September 29, 2009 at 23:44

      Great points Neil – understanding what we expect advertising to do is a critical part of assessing its success.

      However, I’m still slightly torn on the ‘sales’ point – surely sales are the focus of every non-support function in a business?

      That’s not to say that advertising is not a key part in driving sales – of course it is – but is it reasonable to expect advertising *alone* to do this?

      As a colleague always points out when it comes to this issue, “it doesn’t matter how many times I see an advert for Porsche, I still can’t afford one!

      Sales are dependent on many more factors than advertising alone.

      Advertising, or rather, ‘promotion’, is only one dimension of the marketing mix. As such, it performs a specific role in achieving overall success.

      Let’s look at this from the perspective of a soccer team: the team only wins if it scores more goals than the opposition. The objective might therefore appear to be scoring as many goals as possible.

      However, a key factor in winning in this context is preventing the other team from scoring as many goals as our team. So, the goalkeeper performs a vital role in ensuring a win, but doesn’t contribute to the foremost objective.

      And so it is with each of the elements in the marketing mix: each of them contributes to overall success, but their specific, distinct role cannot be measured purely in terms of overall sales.

      Tomorrow’s post has a bit more context on this, but I think this deserves further investigation in yet another future post.

      Thanks again for sharing!

  2. 3 Neil Charles September 30, 2009 at 18:11

    Agreed! But just because lots of things cause sales, doesn’t exempt advertising from responsibility. It does make measurement harder though…

    I only really mean keeping the sales objective top of mind – remembering why things like reach & awareness are important.

    Your example works really well for this…

    “it doesn’t matter how many times I see an advert for Porsche, I still can’t afford one!”

    Either:
    – Porsche are wasting money showing him an ad.

    Or

    – Porsche are doing it deliberately so that the people who do buy their cars can know they’ve bought something a lot of people desire.

    If you’ve got front of mind why you’re showing different people the ad in terms of selling product (or you’re trying not to waste money showing them) then you’ll end up with a better campaign.

    Enjoy the blog by the way!

    • 4 eskimon September 30, 2009 at 18:43

      Gotcha – I think we’re saying similar things in different ways! Either way, we both recognise that success comes down to a clear understanding and focus on objectives, and that is probably the biggest bit of the battle here.

      I’m slowly ruminating the whole “ads must drive sales” topic (again) at the moment, so I’ll post more structured thoughts on it soon.

      Have you already covered it on wallpapering fog? I get the feeling that this topic would be a good third installment in the “advertising debates” series (the other two being medium vs message and reasoned response vs classical conditioning), so I’d love to get your thoughts on it too, seeing as I think we’ll approach the subject from different angles.

      Thanks for sharing your thoughts here too!

  3. 5 Neil Charles October 1, 2009 at 00:27

    Not one I’ve done, but I’ll look forward to reading it!

    My very first post was on why trying to tie ads back to sales with econometrics is hard though.

    http://wallpapering-fog.blogspot.com/2008/12/red-queen.html

    It’s a bit like your football defence analogy.

  4. 6 John Barton October 1, 2009 at 23:07

    What a great discussion, it covers so much ground, it took me two cups of coffee and a shot of whiskey to wrap my head around it. I love the process you’ve outlined and I think it makes a lot of sense when you are looking at sampling and measuring awareness.

    Something I keep thinking about is how exactly does this awareness lead to sales? If I were the CMO at Porsche, I would really like to tie a string to each person who sees my ad and follow how many strings lead to the dealership and perhaps more importantly, what path did they take. I don’t want to spend money advertising to people who wouldn’t buy a Porsche, for whatever reason, but it is probably unavoidable at the awareness stage.

    My ultimate job is to put as many qualified people in front of a salesperson, spending as little of my marketing budget as I can to do it. Because if I can keep the cost per lead/opportunity/whatever down, I have a greater ROMI. That greater return and proof that this tactic works better over the other, means I will spend MORE money on advertising and marketing efforts that I know shows results.

    Spend money on the right ad, on the right channel to the right people = increased sales = more money to spend on the right ad to the right people on the right channel.

    Maybe more focused “messaging” happens at “interest” and “preference” levels, where you can somehow engage people in a conversation about Porsche?

    One thing for sure, is that it all requires the right type of measurement at the each level to really understand what’s going on in the market as well as an understanding of what you can and can/should not measure.

    Thanks again! I’ll pass this post around to my colleagues, I’m sure they will be interested.

  5. 7 eskimon October 2, 2009 at 19:30

    This is why comments sections are so useful… John, while I was reading your latest response, I was reminded of something that Neil hinted at above.

    Porsche make cars that not everyone can afford. Part of that is because of the expensive parts and skilled manufacturing, but a big part of it is because they charge a significant brand premium. A large part of the cost is simply a statement.

    It’s partly that inflated cost that makes the brand desirable, because by being so expensive, it’s much more exclusive. And that exclusivity is one of the reason people buy a Porsche; to some people, it feels good to have something others can’t get.

    That’s all fairly straightforward, but the advertising point you make is much more interesting.

    Advertising to an audience that stretches beyond those who can afford the brand is very clever. The hankering, the craving, the ‘lusting’ all make the brand more attractive, because it gives more satisfaction to those who can afford the brand and who find such exclusive brands appealing. Having something other people want gives them a sense of power and achievement.

    I’ll be posting a piece early next week on differentiation, which will touch on this too – look forward to continuing the conversation on that one!

    • 8 John Barton October 3, 2009 at 03:31

      Great point. The influences that make a brand desirable may not be directly related to selling the product, but probably have a real value. The trick is measuring that, which brings us back to your process.

      Thanks again. Looking forward to next weeks blog. This one has spurred a lively offline dissuasion.

  6. 9 phil October 3, 2009 at 20:34

    great stuff, eskimon! i think you raise very valid points – and lots of them.

    1. i believe that marketing is all about selling. advertising – being a part of marketing – is therefore geared towards selling. i don’t think we cannot not talk about selling. advertising is a means towards an end – a purchase. otherwise, why should companies spend on advertising? to make people feel good? no. to make people buy.

    2. i believe that aligning advertising objectives to the business is critical. sure, it’s nice to be recalled and be understood and to have my message be loved. it’s nice to be the ‘number one favored brand’. but if that doesn’t convert into actual sales, what’s the point? at the end of the day, it’s a business.

    3. advertising’s effects on sales can – and should be measured. ok, let me reword that: advertising’s contributions on sales uplifts should be considered. sure, there are other factors in place – the sales team, the final-mile channels, the reputation the brand has amongst a consumer’s referent groups, discounting, the retail-ambience… BUT that doesn’t mean we should not measure advertising’s contribution to sales. it is not a straightforward process – there are intricacies and convolutions in the measurement process. but that doesn’t mean we should not even bother measuring.

    should advertising objectives therefore be all sales driven? should all ad objectives therefore be aimed at quantifiable sales measures? in an ideal world, yes.

    but we don’t live in an ideal world: because metrics are far too simplistic to ensnare the complicated relationships that lead to an actual sale.

    engagement, experience, communication, conversation, dialogues, recommendation, comprehension, excitement, understanding, belief, empathy, conviction, desire, lust, alignment, affinity – all these are means towards an end: to get people to buy. so that a brand sells.

  7. 10 eskimon October 4, 2009 at 13:51

    Haha, the plot thickens! Thanks for these great additions Phil, you’ve highlighted one of the key distinctions in the upcoming ‘sales’ post.

    The traditional view of ‘marketing’ relates to the word’s literal meaning. This definition comes from dictionary.com:

    “1. the act of buying or selling in a market.
    2. the total of activities involved in the transfer of goods from the producer or seller to the consumer or buyer, including advertising, shipping, storing, and selling.”

    The reference to ‘market’ in the above definition guides us to the crux of the issue – this definition again from dictionary.com:

    noun: the course of commercial activity by which the exchange of commodities is accomplished;
    verb: to sell.”

    It’s clear from all these definitions that the act of selling is central to the marketing function.

    However, considering how marketers’ role and scope of influence has evolved, perhaps we should examine things from a broader perspective.

    Our most valuable contribution to the success of the organisations we work with is encouraging specific behaviours.

    That encompasses the classic objective of encouraging people to buy our goods and services, but can also include more subtle changes in behaviour, such as stimulating charitable donations, or even getting people to act in a more considerate or conscientious manner (à la ‘Don’t Drink and Drive’, or Earth Hour).

    The key distinction here is advertising’s role in effecting that behaviour change.

    Advertising is only one of the tools at our disposal. Just as a hammer is important when it comes to building a house, but doesn’t suit all the jobs involved, so it is with the marketing mix. Advertising can’t do everything on its own, even within the brand communication sphere.

    That’s why it’s critical that we understand precisely what we expect advertising to achieve. We need to see everything in the context of the organisation’s overall success, but we also need to accept that specific functions and disciplines will have their own, specific KPIs.

    More on this soon…

  8. 11 Simon October 15, 2009 at 05:13

    I came to read this post in response to a comment on a post I wrote on viral-ness being a measure of effectiveness. I’ll add a similar comment on my own blog, but thought I’d do the decent thing and post a comment here directly too.

    Simon was asking why people keep on trying to find one uber-measure, rather than trying to identify the objectives of the campaign… which reflects the thoughts above.

    I think he’s absolutely right, but it misses something – the theory outlined above is based on the idea that all advertising and communications has a “message” to deliver to people. Which may or may not be true.

    I think this theory is spot on, but I’d take it a step further – ask not what the ‘message’ was, but what the business need/opportunity was – what is the brand/product ambition? And, therefore, was it achieved? How do sales reflect success? What was the impact on other brand measures?

    I’m firmly against having any single measure as the “yes/no” of success. It sounds great, because everyone knows if you passed or failed. But it’s a false idol. We should be looking at a multitude of measures – surround our original business objective with evidence of success and opportunities to improve as the campaign builds… And, as I was thinking when I wrote the original post, why not include a measure of genuine engagement from the audience? That’s a potential measure of capturing interest… amongst others, but not as the only measure.

    • 12 eskimon October 15, 2009 at 12:46

      Aha! I’m actually with you on the ‘message’ point; I mention the potential misuse of the term ‘message’ above, but I admit it’s not very clear:

      It’s vital that brands and their agency partners develop campaigns around what the brand wants its audiences to understand – although it’s a widely misused term, I’ll refer to this as ‘the message’.

      I’ll stick by that assertion – all communications must establish some kind of shared understanding, otherwise they serve no purpose. Communication (of any kind – brand or otherwise) is about stimulating some kind of change, but that ‘change’ can be as subtle as gently reinforcing existing perceptions, beliefs, or behaviour.

      So it makes sense that communications don’t need a ‘message’ in the sense that many people misuse the term: we don’t always need a slogan, tagline, or coherent statement – as long as we’re clear what we want people to think or do as a result of experiencing the communication.

      And it’s that aspect that the brand needs to clarify beforehand – what do we need people to think or do differently in order to achieve the business goal and brand ambition, and what role will communications play in driving that? This ties back to your comments above.

      So thanks very much for that challenge – it’s really helped me to clear up my thinking!

  9. 13 Rob November 24, 2010 at 11:23

    Great post …

    The ‘what should be measured’ issue is massive, with too much time, effort and money spent on justifying points of view rather than what actually are the factors contributing to the business.

    There’s tons of reasons for this, but the “prove to my boss I’ve not done anything to be sacked” does seem to be high on that list, which also explains why so much time, money and effort is spent discussing how to evaluate the process rather than what the process should be delivering.


  1. 1 advertising and sales « eskimon Trackback on October 6, 2009 at 11:53
  2. 2 get the message? « eskimon Trackback on October 15, 2009 at 16:57
  3. 3 groundhog marketing « eskimon Trackback on October 23, 2009 at 12:50

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s





Follow

Get every new post delivered to your Inbox.

Join 46 other followers

%d bloggers like this: