For Manufacturers, There Can Be More to Measuring Digital Marketing Effectiveness Than Meets the Eye

By Neil Ruffolo, Director, Digital Strategy

Making the shift to a “user first” culture can be a high hurdle for some manufacturing organizations. Even so, the potential payoff has many execs moving in this direction. In fact, everyday business language is now peppered with buzzwords like customer-centricity, customer loyalty, end-to-end customer experience (CX), and voice of the customer (VOC). Whatever angle you take, the root goal is the same: Get those who need your product to want your brand… even if they’re not the ones who pay your invoice. Yet leaders who recognize how demand originates with end users and commit to a market-back strategy are often met with frustration in the execution phase. For manufacturers, the question of how to measure the effectiveness of tactics like a digital marketing campaign presents unique challenges not faced by resellers.

After selling through two-step distribution for decades, middle-market manufacturers discover they don’t have the first-party data needed to engage with a user base. And the channel partners who own that data are justifiably reluctant to share it for fear of being circumvented and cut out of the transaction. Distributors in many sectors are still the keepers of almost all identifying buyer information, as well as point-of-purchase data like product purchase history, purchase frequency, average purchase amount, and lifetime customer value (LCV). Given all this, many industrial brands are left to do the best they can with whatever first-party data they can muster. Warranty registrations, drop shipments, replacement part orders, email newsletter subscribers, and website traffic can be good sources of end-user data. But any actionable insights to be gained from this data are only as good as its interpretation. Almost anyone can fall victim to survivorship bias.

Never Go Against a Hungarian Mathematician When Death Is On the Line

Survivorship bias is the logical error of focusing on what is visibly evident to the exclusion of all else. In short, it basically means counting hits, but not misses. And it is a potential problem whenever the raw numbers are limited, obscured, inaccessible, or not knowable. When it comes to learning how to measure the effectiveness of a digital marketing campaign, this error can manifest itself in one’s tendency to assign more relevance to first-party data.

A popular example of survivorship bias has been making the rounds on business blogs for a couple of years now. It is the story of Abraham Wald, a Hungarian-American immigrant who worked on the Applied Mathematics Panel for the U.S. military during World War II. This panel used sophisticated math and data analysis to solve complex problems from the battlefield. In one case the Army Air Force wanted to improve the odds that a bomber and its crew would return safely. They knew their airplanes needed more protective armor, but armoring the entire underbelly would make them too heavy. So the problem was how to optimize the configuration of limited armor in a way that would minimize losses.

The military had recorded where planes returning from enemy territory had sustained damage. This data made it evident that bullet holes tended to be clustered on the wings, down the center, and around the tail. So the top brass assumed that adding more armor to those areas would be the best solution. Wald recognized that the data collected from returning planes, in fact, showed where armor was needed least. He recognized the areas that actually needed protection were the cockpit, engines, ailerons, and stabilizers. That’s where the planes that were shot down had been hit, but they never made it back to be counted.

So if top military strategists are susceptible to data interpretation errors when lives are at stake, it’s not a stretch to believe business strategists can fall into similar traps.

When first-party data steers you wrong: A Lesson for B2B brands from the digital media battlefield

A brand that focuses on users is generally more profitable and inspires more loyalty than a brand that focuses on buyers. That case has been made by others, and it is not the purpose of this article to further argue the point. Though it is worth noting that the lines separating B2B users, buyers, and customers can be fuzzy, depending on your perspective. This causes confusion. So, for many industrial manufacturers, it is better said that brands focused on end-users are more successful than those focused on dealers, distributors, or channel partners.

It can be a long and bumpy road from being a channel-centric brand to a user-centric brand. This is particularly true if you are a manufacturer starting from scratch with little or no end-user data to work with. How to begin is the first obstacle to overcome. Many industrial brands start with a targeted digital media campaign to promote awareness and drive end-users to the brand website for further engagement.

Connecting end-users with dealers is one way many industrial brands engage website visitors. We call this function Where-To-Buy, or WTB for short. In some cases, WTB is an opportunity to convert anonymous web traffic into known contacts via registration. But the friction this introduces is usually not worth the trade-off. (50 to 75 percent of these prospective customers will abandon the process and go elsewhere.) So allowing WTB referrals to go unidentified without registration is common.

For manufacturers, even anonymous WTB referrals are a valuable leading indicator of demand. They measure some heightened level of purchase intent, albeit much weaker than a dealer quote request, or an actual ecommerce transaction. But for brands at an arm’s length from the point-of-sale, this sort of first-party data is valuable. As a KPI for digital media outreach, it would seem WTB referrals play an important role in closing the gap between an industrial brand and its end-users. One might even conclude WTB referrals should be the primary goal for all owned, earned, and paid digital media. But that would be a textbook example of survivorship bias.

Raw WTB referral data is actually a lot like counting bullet holes in airplanes returning from a bombing run. They look important because you can see them, not because they tell you what you want to know. This was apparent in a semi-controlled “live experiment” involving an equipment manufacturer that held the same seasonal promotion every year. Same product category. Same brand. Same duration. Same discount. What changed from one year to the next was the amount budgeted for digital media to promote end-user awareness. The channel mix included paid social, video, display, and search advertising.

Looking back at annual campaign performance was an eye-opener. From Year-1 to Year-3 there was not a strong correlation between WTB referrals and orders as was expected. In fact, when WTB referrals dramatically spiked in Year-2, orders remained basically flat. Yet there was a surprisingly strong correlation between digital ad impressions and orders.

Impressions, or total reach, is often dismissed as a soft metric. In this case, however, orders and impressions were tracking together. So the Year-4 media mix was adjusted accordingly. Less money was spent on search and social, which are the channels that tend to deliver more website traffic (and WTB referrals) at the expense of total reach. Instead, a larger share of the budget was shifted to display and video, the channels that deliver greater reach at the expense of website traffic. As predicted, the Year-4 outcome was the best yet.

From Year-1 through Year-3 the opportunity cost of prioritizing WTB referrals (essentially a channel-focused metric) was overlooked due to survivorship bias. This limited activity, which could be tracked on the brand website, was assumed to be more relevant and valuable than all other off-site activity. It was not. After prioritizing total audience reach (an end-user focused metric) in Year-4, the totality of activity that was stimulated “off the radar”, including showroom visits and direct-to-dealer website traffic, proved to be substantially more valuable.

Measuring Digital Marketing Effectiveness: 3 Ways to Overcome Survivorship Bias

An object you can see is no more real than the space around it. And the bullet holes you cannot see are the ones that actually kill. When you consider these perspectives, then it does not take such a big leap of faith to trust deductive logic and inference as much as hard numbers. But old habits die hard. Here are a few ways user-centric B2B brands can avoid the pitfalls of survivorship bias:

1. Choose KPIs carefully. Always question whether or not a metric really measures what you think it measures. The problem with WTB referrals as described above is not that the metric itself is inherently bad, or invalid. It’s just not a good way to measure digital media effectiveness in this particular case. I would argue that WTB referrals are a better measure of on-site usability, or UI efficiency, than advertising performance.
2. Calculate what you cannot measure. Raw digital marketing KPIs on their own don’t say much about the things that matter to the CFO and Board of Directors. But they do provide a basis for calculating what you cannot measure directly. Even if you’re not a data scientist, technology now makes it possible to attribute revenue, margin, cash flow, and pipeline velocity to investments in omnichannel marketing.
3. Start selling direct. As the saying goes, if you can’t beat ’em, join ’em. This idea might not be as crazy as you think. In recent years, it has become a lot more common for manufacturers to engage in direct sales without ruffling the feathers of channel partners. One way is to focus on serving a niche segment that dealers are ignoring. Instead of leaving this incremental revenue and market share on the table, why not service it in-house? The variety and amount of first-party data, market insights, and direct contacts will help to fuel an end-user centric growth strategy. An ecommerce pilot program is also a smart way to prep for the possibility that Amazon Business will steamroll your existing distribution channel sooner or later.