Data in Wilderness

Continuing with our series on interesting aspects of data, we will today look at the data of Human Insights; something missed by organizations. Almost universally. A majority of business and IT departments use this data in its rudimentary form, and mainly by accident and without realizing that they are. This data of Human Insights is different from (and NOT a replacement for) Data Insights. We will examine what it is, why it is missed, and how it is different from Big Data.

A concept of “Thick Data” was propagated by Tricia Wang, a technology ethnographer in 1983; she is widely recognized as the birth giver of this field of data science. Thick Data is the science of Human Insights that is vetted by Data Insights. The reason the two must work together is that Human Insights data is based on a very small dataset (and prone to errors as we will see a little later); so, the Big Data has to back up what Human Insights are “recommending”. Wang describes her work with Nokia, where after living the life of a poor Chinese, she came up with her findings based on a sample size of 100. Her idea was rejected because the concept of a smartphone did not fit into Nokia’s Big Data. We all know what happened to Nokia; the very markets that it once dominated are now the largest markets for smartphones.

Another example quoted by her is Netflix, which when presented with a Human Insight, went back to its Big Data, and validated it. And so was born binge-watching. The Human Insight was that people wanted more of the same; and Netflix gave it to them. And of course, we all know where Netflix is today.

Gartner adopted this as “X Analytics”. MIT started researching it extensively and is now creating a whole field of science around this unstructured data. The real challenge is that the sources of this data are typically voice, video, images, IoT, and human artefacts like writings and surveys. And now, parts of the implementation are emerging strongly in areas of Data Visualization (collect data from sources of data), Data Stories, Games Technology (entering the domain of consumer and business technology), Augmented Reality, etc. One can easily imagine it enter the realm of Customer Service, Recommendation Engines, etc.

But most of the examples above are implementations of the concept. The real power of Human Insights however, lies somewhere else. MIT’s Integrated Data Thinking framework talks of a scale as one moves between Big Data, which is quantitative and humongous, and Thick data, which is qualitative and miniscule. As one slides back and forth between these two, you play the field of Discovery, which is the unknown, and of Optimization, which is more explainable. In other words, one needs to increase what is called Explainability by taking the Discovery and vetting it against the Optimization algorithms. This increase in Explainability allows the business to adopt with confidence, because it is comprehensible, succinct, actionable, reusable, accurate, and complete.

Undoubtedly, many Human Insights are wrong or insignificant. There are sufficient examples of “brilliant” insightful goods and services that were rejected by the target consumers. After all, if the Big Data has a set of a billion data rows, and every 100 of them provides a Human Insight, surely not all of them will be a hit. And this is the essence of this model: Thick Data provides Human Insights; Big Data provides Data Insights to back up or reject them. The very essence of Big Data is to ignore Human Insights as “Outliers” when left to itself; and therein lies the danger in stopping at Big Data.

Computational Data Science today has sufficient power to help identify which of the Human Insights are even worth vetting against Data Insights. Unfortunately, far too many organizations get carried away by Big Data as an end in itself. Its insights are meaningful, but they are limited by its own core principles. It loses the human context and emerging trends during its process of normalizing and optimizing. And it misses the stories.

As Wang says, Big Data needs Thick Data. And we say, the reverse is equally true. The science and technology exist to bring the Human Insights out of this wilderness. And it is time to do so; more so in today’s uncertain and spin-on-a-dime times.

The author managed large IT organizations for global players like MasterCard and Reliance, as well as lean IT organizations for startups, with experience in financial and retail technologies