Does Bigger Data Lead to Better Decisions?

Many scholars, from decision scientists to organizational theorists, have addressed this question from different perspectives, and the answer, as for most complex questions, is “it depends.” Big Data can lead to Big Mistakes. After all, the financial sector has been flooded with big data for decades.

A large body of research shows that decision-makers selectively use data for self-enhancement or to confirm their beliefs or simply to pursue personal goals not necessarily congruent with organizational ones. Not surprisingly, any interpretation of the data becomes as much an evaluation of oneself as much as of the data.

How can organizations avoid such pitfalls and turn “Big Data” into a safe opportunity? Decade-old research provides some pointers. It is not Big that matters, it is Diversity that matters. Big is old – retailers and financial institutions have had big data for decades.

But Diversity is new. Take large retailers. Sure, they have had enormous databases for long time now. But marketers are only now connecting data from loyalty programs in physical stores with data not only about how the same customers behave on the company’s website, but also how the same or similar customers anywhere in the world behave on other websites – ranging from news sites to car sites to movies sites – all tracked using cookies. They can then link this data with in-depth market research as well as social media data from Twitter or Facebook.

This kind of linkage is reaping rich rewards.  A leading Telco company we have worked with was able to increase market share by more than 20% in some countries without increasing the marketing budget by leveraging behavioural and transactional data from social and general media.

Some innovative companies are connecting data traditionally used by banks to assess the credit score of loan applicants with information ranging from mobile phone usage data to online social media relations data, in order to better and faster assess the creditworthiness of a micro-loan applicant. What can a phone bill tell us about the chances that someone will repay a loan?  Or even about the creditworthiness of the people that the applicant is connected to online?

Of course, management scholars and practitioners have long recognized the benefits of diversity.  It’s widely accepted that heterogeneous teams are more creative than homogeneous ones. Diversity, if managed well, yields divergent thinking and the pooling of a broader base of knowledge results often in better strategic choices.

The point we stress here is that diverse data confers similar benefits. And it’s worth noting in this context that in statistics and data science as well the key quality measures in data are not the size of the dataset but metrics like variance and entropy, which effectively capture the data’s diversity.

So perhaps we shouldn’t be talking about Big Data making decisions better, but about Diverse Data connecting the dots using new technologies, processes, and skills. We need to connect the dots or we risk drowning in Big Data.

by Theos Evgeniou, Vibha Gaba and Joerg Niessing HBR