It takes two: understanding customers through big and small data

The terms “Big Data” and “Small Data” have become popular buzzwords over the past five to ten years. Yet, it’s not always clear what either of these terms means or how they help us better understand our customers.

Big Data is data created in untold ways, such as through transactions, clicks, radiofrequency identification readers (RFID), sensors, financial data and the vastly growing number of IoT connected devices.

Small Data, on the other hand, is the data we gather through primary research. It isn’t just gathered from qualitative research – focus groups, in-home ethnographies, online communities, and so on – but also from quantitative survey research. It’s where we ask or observe people directly to uncover their attitudes, motivations, and values.

This paper unpacks what each of these buzzwords means and outlines why it is important to use both types of data to really understand your customer.

What is Big Data?

It may feel obvious what is meant by the term ‘Big Data’ but clearly identifying it is more nebulous. Gartner formed an early, yet still widely cited, definition using three ‘Vs’ (Volume, Velocity, and Variety) to represent the key characteristics of Big Data:

“Big Data is high-volume, high-velocity and/or high-variety
information assets that demand cost-effective, innovative forms of information processing that enables enhanced insight, decision making, and process automation.”

Big Data’s Volume, i.e. the amount of data, is represented in vast quantities; often terabytes, petabytes and beyond. It requires advanced computing power and new processing techniques to manage. The Velocity, that is the speed in which the data appears, is fast and is measured in minutes and hours. Finally, Variety is the way in which the data presents itself. It is both known and unknown, consisting of structured (i.e. date, time, GPS location, database), semi-structured (i.e. clickstream) and unstructured (i.e. text, image, voice, video) detail. More recent definitions include other “V-terms” into the Gartner model, adding Veracity to reflect data accuracy and Value to address the
usefulness of the data.

Big, and growing

We live in the Data Age. We don’t just have vast quantities of existing Big Data but its growth is almost incomprehensible. Consider that in August 1991 there was one website on the internet. Today, there are nearly two billion. All of these web pages contain information in the form of unstructured textual, image, voice and/or video data. Additionally, more data is created when people visit these pages, in the form of clicks, searches, transactions and so on. Perhaps it’s unsurprising then that 90% of the world’s data was created over the last two years, at a rate of 2.5 quintillion bytes per day.

What is Small Data?

The nature of Small Data is less obvious, and this is reflected in the lack of agreement on its definition. Allen Bonde, one of the earliest users of the term around the late 2000s, defines Small Data as “…connecting people with timely, meaningful insights (derived from Big Data and/or ‘local’ sources), organized and packaged (often visually) to be accessible, understandable, and actionable for everyday tasks”. Whereas branding expert Martin Lindstrom defines it as “The tiny clues that uncover huge trends”, which he generally collects through observational data. Perhaps the most commonly used broad definition is “Data that is small enough in size for human comprehension” (original source unknown). This idea of it being ‘small enough for human comprehension’ is key. In comparison to Big Data, Small Data’s volume is more manageable and is measured in megabytes and gigabytes. It can be stored and processed on a single computer, using established techniques developed in the 1950s onwards as the first commercial mainframe computers became available. The speed of the data is slower, being collected over days and weeks. Finally, it consists entirely of known detail, the data is either structured (i.e. numeric) and/or unstructured (i.e. text, images, video).

Small, yet impactful

Small Data has been around as long as humans have been documenting our lives. Think, for example, about the Rosetta Stone, which was carved in 196 B.C and is one of our oldest historical documents. This is a form of Small Data. The documentation of tax and population information gathered over the centuries are other examples. In modern times, social and market Small Data formally started to be collected in the mid-1800s with the creation of the first questionnaire by Sir Francis Galton and work by early social science pioneers Joseph Rowntree and Charles Booth, who conducted pioneering work on poverty that impacted government decisions.

Within market research, a well-known early example of Small Data’s impact is from the 1950s where focus group insights helped revive Betty Crocker’s declining dry cake mix sales. These focus groups uncovered that the “just
add water” recipe was unsatisfying. The new recipe required bakers to add eggs as well, resulting in a better experience.

A powerful equation

As we’ve seen above, Big and Small Data are very different in how they are defined and what the data looks like. Each type, however, is invaluable in its own right. Big Data helps us understand human actions and behaviors, for example, website clicks, sales transactions. We are able to obtain an objective ‘What’. What people did. Small Data, on the other hand, helps us understand the attitudes, motivations, and emotions behind those actions and behaviors. The histories and situations leading to the ‘What’ captured through Big Data. Small Data helps us
uncover the ‘Why’.

Analyzing Big or Small Data independently can provide a good understanding of your customer experience. However, by combining the two data sets, you can bring the full customer story into greater and more insightful focus. As Lindstrom notes, “The best, closest approximation of who we are as humans comes from mixing our online and offline selves, and from combining big data with small data”. By drawing upon both Big and Small Data, you
can gather a holistic picture of your customers’ truth.