Are Your Metrics Misleading? Avoid These 6 Mistakes of Business Intelligence

I started off intending to write an article on metrics. What I realized quickly was there was no real consensus on specific metrics. In fact, deciding which metrics are most critical is a strategy unto itself. Looking at individual metrics without putting them into the larger context is like looking at an elephant through a microscope. So, think of metrics in terms of your ‘Business Intelligence Strategy’. When developing a Business Intelligence strategy, there are six common mistakes and ways to avoid them:

Mistake #1. Misapplying Metrics.

The most common mistake I have seen over the years is to assign the wrong metrics to a particular campaign or program. For example, evaluating a lead generation effort using ROI metrics like cost per sale, revenue dollars, or ROI. The objective for a lead generation effort is to get someone to raise their hand. Period. Whether a lead converts into a sale is affected by a number of other variables – which should be measured independently – such as response follow up (were they properly nurtured?), the price/value equation, fulfillment requirements, etc.

Often, when this occurs it is a symptom of a larger problem, the campaign design. Marketers sometimes try to accomplish everything in a single communication (from cold prospect to buyer) but the buying process is more complex and drawn out.

A good way to avoid this is to map out the entire campaign’s ecosystem and assign metrics specific to each component’s objective. This makes it possible to isolate where adjustments are needed.

Mistake #2: Reading False Positives.

Direct marketers have known for decades there tends to be an inverse relationship between response and conversion. A communication which is poorly targeted or designed can stimulate a lot of poor responses. But, in terms of response rate, it’s a home run.

In an A/B split test for demand generation it was found that a ‘sweepstake-type’ offer outperformed a voucher offer by a considerable margin. When analyzing responder personas it was found that the sweepstakes offer attracted the worst persona segment, those least likely to convert to a sale or be profitable. By contrast, the voucher attracted the best persona segment which had the highest likelihood of being profitable.

A good way to avoid this is by using a relational database (before, during, and after) to view results rather than a flat file. This makes it possible to apply a single metric, such as conversion rate, to many different variables (segments, subsegments, media mixes, response and purchase channel, offer and message version, etc… ). Doing this helps provide a better understanding of results over time and more holistically.

Mistake #3: Short Term Thinking

Campaigns are episodic with a clear beginning and end. Often the evaluation period ends too soon. This is not to say that you shouldn’t start tracking, measuring, and analyzing at the very beginning. You should. However, the true impact may not be fully evident until sometime later.

For a manufacturer of recreational vehicles it was found that those who responded to a communication and did not convert to a sale were twice as likely to respond and 10 times more likely to convert when remarketed to the following year.

This phenomenon was attributed to brand commitment that was built from the first communication forward. Further research found that the integrated/multichannel campaign was effective at incrementally overcoming barriers to purchase while building brand commitment over time. So when end-users walk into a dealer they were more likely to make a purchase and were less likely to be sold on a competitive brand. Each component in the campaign had a specific role (and KPIs) linked to the others in the overall path to purchase.

A good way to avoid this is by thinking ‘horizontally’ across the entire lifetime of the individual instead of ‘vertically’ within a fixed period of time like a campaign.

Mistake #4: Not Closing the Loop

Many of the aforementioned mistakes can be attributed to not closing the loop. Closing the loop requires being able to track an individual from beginning to end. Actually, there is no ‘end’ if you are measuring lifetime value (LTV), attrition, purchase history, etc..

For a specialty tool manufacturer, product warranty registrations were matched back to the original e-mail solicitation file to close the loop on campaign performance. Even though only 4% to 6% of buyers register for a warranty it is a reliable indicator. In this case, every warranty registration represented 20 purchases (20 purchases X 5% = 1 warranty registration). This made it possible to project total revenue, ROI, cost-per-sale, performance by segment, and many other business intelligence metrics.

There are many ways to close the loop through tactics (e.g. PIN number assignment), technology (offline/online matching), research (phone and email confirmation), data capture, and opt-ins. It doesn’t require capturing 100% of all activity. There just needs to be a statistically significant sample to measure. Keep in mind there is a lot of readily available transaction data – such as warranty cards, drop shipments, customer service activity, parts orders – that identifies an individual who can be matched back to the original campaign.

Mistake #5: Getting Lost in the Weeds

Far too many result analyses are composed of page after page of charts, graphs, tables and numbers and lack proper context or clarity. In fairness to the analysts and those doing the tracking, there needs to be a full accounting of the methodology, the statistical methods, the various aspects of the data and caveats. Dashboards are a good way to pull it out of the weeds. However, they too can lack context and clarity.

Additionally, numbers and percentages are most relevant when indexed to something.

A good way to avoid this is to create a ‘composite dashboard’. For example, a major manufacturer of trucks was able to develop a single view dashboard with a narrative of what happened during the time period that included changes over time from multiple sources and media channels. It also included explanations of why things happened (it was a holiday week, the weather had an impact, there was a major news event, the media campaign kicked off, etc…) and what the implications were.

Mistake #6: Numbers Without Context

Numbers and percentages are of limited value without comparing to something and putting them into context. A metric can be compared to an industry average or other results. However, that can be misleading if the results should be 3 or 4 times an average or, of past results.

A good way to avoid this is to use an index for context. For example, if 2% of buyers came from one segment, is that good? It is if that segment represents only 1% of the total market. In that case, you’re doing twice as good as (an index of 200) the average. By contrast, if that segment represents 4% of the market you’re not doing well (an index of 50).

Business Intelligence Strategy: Best Practices for Defining KPIs

To avoid critical pitfalls and to improve business intelligence, follow these practices:

• Start with a Single View of the business, from the industry level such as:
  • PAM-TAM-SAM-SOM
  • Share of Shop, Share of Spend, Share of Wallet
  • Penetration Rate
• Create a campaign ecosystem that maps the entire campaign and all the paths to purchase with response possibilities.
• Create 1-page campaign dashboards using:
  • Multiple sources
  • Composite Information
  • Contextual Narrative (summary statement)
• Discipline the assignment of metrics to specific objectives and/or behaviors
  • For example, 1^st get hand raisers, 2^nd get engagement, 3^rd get a request for information, etc…
• Create a match-back loop
  • Start with Personas and end with Personas
  1. Use source codes and matching logic to match targets to behaviors
  2. Always, always, always ask for information. Especially with online messages/solicitations to match anonymous identification with PII (personally identifiable information)

As the business environment becomes more complex and more interconnected it becomes increasingly important to have a Business Intelligence strategy that is linear (over an extended period of time), holistic, and comprehensive.

Keep in mind, the real purpose of metrics is to help understand what happened, why, and what to do next. A Business Intelligence strategy will help put individual metrics in the proper context to achieve the overall business objective.