Human beings, and
scientists in particular, are conditioned to look for causes. One of the most
brilliant scientists of our time, Einstein, had his nemesis in what we call
quantum mechanics (or quantum physics as it is also sometimes called). This is
the study of the smallest particles known to man. Einstein’s theory of
relativity and quantum theory are – in many ways – contradictions, which have
still not been satisfactorily reconciled into one cohesive theory.
For example, at the
quantum level, one particle can potentially be at two places at the same time!
It defies all logic and yet it is possible. If you look at a nanoparticle,
it can change from a wave form to a point. This is observable in the lab. Other
types of strange behavior of these smallest nanoparticles – which in turn make
up all things big and small - has been known for a long time and can be
replicated. There is little doubt that this strangeness actually occurs.
Einstein himself called
quantum mechanics “spooky” because we could all see what was
happening, but were unable to explain how it is happening. Today,
these issues are still unresolved.
How does this relate to
Big Data? To understand this, let’s consider:
Until recently we lived
in the era of “Small Data” which was characterized by the premise of exactitude. Databases
were built to very specific formats and rules, so that they could be relied
upon to retrieve accurate results which perfectly matched the queries we
applied against such data. If you used a spreadsheet to compute data, this is what
you did. Data was smaller in size – because among other things, computing and
storage costs were very high. This in turn meant that databases had to be
precise in order to be useful. Databases tended to answer the causal question: What
is happening and why? It took time to use data this way and actionable
intelligence was, in some cases, delayed. But, that was the cost of using small
data and applying data analytics to it.
Today, the costs of data
storage have shrunk dramatically. Computing power is enormous and climbing.
Enter the era of “Big Data”.
One of the earlier
examples of Big Data was in healthcare (as Viktor Mayer-Schönberger and Kenneth
Cukier point out in their bestseller “Big Data”). Back in 2007, the CDC would
go through precise collection of data points, aggregate them and then use that
data to keep track of the spread of flu among the states. They had been doing
the same thing for a while. All of this took time, data had to be collected
from doctors’ offices and hospitals and by the time the CDC statistics were
compiled and published, weeks went by and the data was sometimes already too
old.
Around that time, Google
was testing its algorithms and mathematical models to test search terms. They
found that when users search for 45 terms in specific combinations – that
became a predictor of the flu and was closely mapping the CDC data. Just like
the CDC, Google too could now predict how flu was spreading nationwide, but unlike
the CDC, their data was real-time! And Google did not have to go to
doctors’ offices to get data, they extrapolated it from their users’ searches.
Google’s data model was
not built on the concept of exactitude. Rather it was based on the
use of patterns and correlations to tell us what was happening, not necessarily – how it was happening.
The Bureau of Labor is
responsible for computing CPI or the Consumer Price Index. To do this, they
employ thousands of people who report approximately 80,000 prices on everything
from home rentals to the cost of airline tickets and everything in between. Many
things are dependent on CPI including wages, social security amounts etc. CPI
is very important. Again, by the time these numbers come out, they too are
already old.
Then, two economists from
MIT came up with a Big Data solution for CPI. They collected data on 500,000
products in the US – many more than the Bureau of Labor was collecting - and
did so only using the web. Admittedly, their data was “messy” in that it was
not exact – but it
was possible to figure out a CPI equivalent much more quickly. And because they
had much more data, any individual anomalies were less meaningful. This project
is now a commercial venture and everyday thousands of financial institutions
are making decisions based on this work.
In that sense, Big Data
is the equivalent of quantum mechanics in this storyline. We can see what
is happening, but don’t really know (or maybe even care) how it is exactly
happening? The price of getting actionable intelligence in a timely way, is
an absence of the precise causal questions and answers. Many
are still uncomfortable with this lack of preciseness, and this is only the 2nd or
3rd inning of the Big Data story. More discomfort is
in store.
Today, a significant
percentage of the trades on Wall Street are driven by computerized Big Data orders.
Healthcare, retail, airlines, hotels – almost everything you can imagine – is
being impacted by Big Data. We haven’t seen nothing yet.
In the era of Small Data,
accuracy was preeminent because only small data sets were possible and had to
be exact. But, as we delve deeper and deeper into this new world of Big Data,
we are all making a tradeoff: the quality of the information
becomes less important because the volume of the data evens
things out. It is the speed of
decision-making and the new insights we are learning which are becoming
paramount, and all this is being revolutionized by Big Data driven by patterns
and correlations.
Prof. Einstein would
likely think Big Data too was “spooky” - because just like quantum mechanics -
we know what it does, but don’t go looking for too many
answers on exactly how and
why it got there. You may be disappointed.
