Chris Conway
Chief Architect, Quantiv
By 2025, 70% of organisations will shift their focus from ‘big’ to ‘small and wide’ data, according to predictions by technology research firm Gartner. The company believes this change will provide better operational data, more context for analytics, and make artificial intelligence (AI) less data hungry.
So, what is small and wide data, and how is it different from big data?
But first: what is big data?
If I could describe big data graphically, it would be as a blob. It has no clearly defined shape or even edges, but somewhere within it there could be a hint of a recurring pattern of interesting data.
In the big data set, discovery of a pattern isn’t guaranteed. In other words, the purpose of collecting the data is to find a pattern (in fact, any pattern). The aim isn’t to collect instances of a specific pattern.
Equally, there’s a lot of noise in the data and it may not include all examples of the pattern.
In terms of data collection, to be able to find a pattern you need data to be wide-ranging (i.e. big) because you don’t know what you’re looking for. But you don’t need the data to be complete. Plus, it can be made available long after the event that led to it.
To put it another way, you want to collect as much data as possible to give yourself the best chance of finding the pattern. But you don’t mind if a few examples are missed, or the data is late.
The benefits of small and wide data
That pattern could then form the basis of more deliberate collection. And although you would be looking for the same pattern, its collection would be different.
Once a pattern has been identified, the process of monitoring it is where small and wide data comes into play.
For operational purposes, the data must be complete and timely. But while it needs to feature references to all relevant dimensions (i.e. wide), it shouldn’t include examples that don’t match the pattern (i.e. small/no redundant data).
To collect small and wide data, you need a very different approach from the collection of big data. That’s because you care about what you’re specifically looking for and that you’ve found all examples.
Gartner’s predictions
The Gartner analysts said disruptions such as the Covid-19 pandemic had caused historical data that reflects past conditions to quickly become “obsolete”, which is “breaking” many production AI and machine learning (ML) models.
Jim Hare, Distinguished Vice President at Gartner, urged data and analytics leaders to turn to small and wide data.
He said: “Taken together they are capable of using available data more effectively, either by reducing the required volume or by extracting more value from unstructured, diverse data sources.”
Quantiv: champions of small and wide data
At Quantiv, we’re already advocates of small and wide data.
Our NumberWorks method can help you formalise the definition of the interesting patterns in your data. And our NumberCloud platform uses these definitions to support the collection and exposure of the information that matches these patterns.
Want to find out more? Call us on 0161 927 4000 or email: info@quantiv.com
