Monday, August 12, 2013

The Top 0.1% - Keeping Pace with the Data Explosion?

I've been thinking a lot lately, as I do, about data.

When you work with something so closely, it is hard not to have the way you think about what you work with impact the way you think about other things in other parts of your life.

For instance, cooks don't think about food the same way once they've worked in the kitchen; bankers don't think about money the same way once they've seen a vault full of it; and analysts don't think about data the same way, once they start constantly analyzing it.

The difference being, of course, that not everything is related to food or money, but everything is data if you know how to think like an analyst.

I remember when I was in elementary school as a young child and a friend of mine described to me the things of which he was afraid. We sat in the field behind the school and stared down at the gravel on the track.

"Think about counting every single pebble of gravel on the track," he said. "That's the sort of thing that really scares me." He did seemed genuinely concerned. "Or think about counting every grain of sand on a beach, and then think about how many beaches there are in the whole world, and counting every grain of sand on every single one of those beaches. That's the sort of thing that frightens me."

The thing that scared my childhood friend was infinity; or perhaps not the infinite, but just very very large numbers - the quantities of the magnitude relating to that thing everyone is talking about these days called Big Data.

And that's the sort of thing I've been thinking about lately.

I don't remember the exact figure, but if you scour the internet to read up on our information age, and in particular our age of "Big Data" you will find statements similar to that below:

.... there has been more information created in the past year than there was in all of recorded history before it.

Which brings me to my point about the Top 0.1%.

Or, perhaps, to be more fair, probably something more like the Top 0.01%.

There is so much information out there. Every day around the world, every milliliter of gas pumped, every transaction at POS, every mouse click on millions of websites on the internet is being recorded, and creating more data.

Our capacity to record and store information has exploded exponentially.

But, perhaps somewhat critically, our ability to work with it and analyze it has not.

In the same way that The Ingenuity Gap talks about how the complexity of problems facing society is ever increasing but our ability to implement solutions is not matching that pace, we might be in danger of similarly finding the amount of information being recorded and stored in our modern world is exponentially increasing but our ability to manage and analyze it is not. Not only from a technological perspective, but also from a human perspective - there is only so much information one person can handle working with and keep "in mind".

I know that many other people are thinking this way as well. This is my crude, data-centric take on what people have been talking about since the 1970's - information overload. And I know that many other authors have touched on this point recently as it is of legitimate concern; for instance - acknowledging that the skill set needed to work with and make sense of these data sets of exponentially increasing size is so specialized that data scientists will not scale.

Will our technology and ability to manage data be able to keep up with the ever increasing explosion of it? Will software and platforms develop and match pace such that those able to make sense of these large data sets are not just a select group of specialists? How will the analyst of tomorrow handle working with and thinking about the analysis of such massive and diverse data sets?

Only time will answer these questions, but the one thing that seems certain is that the data deluge will only continue as storage becomes ever cheaper and of greater density while the volume, velocity and variety of data collected worldwide continues to explode.

In other words: there's more that came from.

No comments:

Post a Comment