My colleague Angie recently published a blog on “5 Surprising Data Trends of 2024.” The first trend was the rise of Small Data. Angie is a Virgo, so she’s all about precision and facts. And while these data insights were compelling, they made me think about how we balance all our data, the small and the big.
As a Libra, I’m supposed to exhibit a number of personality traits: charming, sociable, harmonious. The average Libra respects balance, harmony, and justice. While I take the zodiac with a grain of salt, I admit that the concept of balance resonates with me.
So, it’s with a sense of relief that Small Data is emerging as a trend in 2024, to level the scales that Big Data has dominated over the past few years.
What is Small Data
The best definition I’ve found defines small data as data that’s “small enough for human comprehension.” I like this human-centric explanation because, ultimately, we humans exist in a physical world we’ve manipulated to suit our needs. We’re, by definition, human-centric, and we often struggle with concepts that aren’t. I think this explains why there are so few Theoretical Quantum Optics professors.
Big Data vs Small Data: Context is Key
Big Data might show, using human movement data (mobile phone locations), that approximately 50,000 people from all over New Zealand travelled to Kingsland Auckland on 20 July 2023. But it doesn’t tell you why. Small data, on the other hand, through ticket or merchandise sales, could explain that those 50,000 people were attending the FIFA Women’s World Cup opening game at Eden Park.
We can conclude that small data is often about providing context to big data concepts.
The other important aspect of the definition is the term “comprehension.” As a human, even a charming, sociable, and harmonious Libra, I struggle to make sense of large datasets (it’s why I like maps). But people much smarter than me struggle to comprehend a dataset of 50+ million mobile phone records. Small Data, therefore, needs to be of a volume and format that humans understand. That’s one of the strengths of FME, particularly its ability to manipulate and transform data from various sources into a clean and easy-to-read dataset.
Finding the Balance: Big Data, Small Data, and the Future
In some ways, the focus on Small Data is a return to normalcy before the hype and investment in Big Data. For many people, small data is all we’ve ever known. This is especially true in New Zealand, where true Big Data datasets are few and far between. However, it isn’t quite right to think the landscape hasn’t been changed by Big Data. The use of machine learning and AI has impacted how we analyse data, big or small. And what we’re seeing now is a greater demand by Big Data to add context, a context that often only small data can provide. Without it, Big Data can be difficult to action. It might show a trend or pattern, but deciding what that pattern means or how to create meaningful action from it can still be difficult to understand.
The future of data analytics isn’t one based on big or small data. Instead, it’s about the appropriate combination of the two. Big Data provides trends, Small Data provides context, and tools like FME provide the ability to manipulate, automate, and transform data. Balance is restored, and Libras like me can sleep happy.
