How to survive the data explosion

IT organizations everywhere are racing to deal with the onslaught of petabytes. Here's how to meet the challenge

IDC estimates that enterprise data doubles every 18 months. That's an astounding statistic, but somewhat difficult to wrap your head around. A simple analogy may help.
Let's say you're an avid movie buff, and when the American Film Institute's top 100 DVD collection came out in November 1998, you were one of the first to buy it. A collection of 100 DVDs is large enough to be impressive, but small enough to browse easily and find something you want to watch. Weighing in at around 28 pounds and taking up about four feet of space on your bookcase, even the most cramped NYC loft is likely to have space for it. Best of all, "Apocalypse Now" is only a quick 30-second visual search away from your DVD player.
Now, let's apply IDC's enterprise data growth stat to your primo collection. After doubling every 18 months, in November 2015, your collection would have grown to more than 200,000 DVDs weighing over 20 tons and taking up nearly two miles of shelf space. Unless you kept the DVDs scrupulously alphabetized, finding the one you want could take hours. Your collection would have grown to such a massive size, it would almost be useless. It would be a ball and chain dragging behind you until you give up and get rid of most of it.
That's precisely what's happening with our data. Personal, corporate, governmental — it doesn't matter. We're keeping and maintaining way more data than we can possibly ever use. The fact that an 18GB disk available in 1998 is roughly the same size, weight, and cost as a 4TB disk today only obscures the problem and makes us lazy about policing our data growth.
We may be able to store all that data, but when we lose the ability to manage and exploit it effectively, its value decreases. As a result, many businesses are spending more and more time and capital to store data that's worth less and less to the business. Data growth is unavoidable. But it must be accompanied by data management policies that ensure the data created and retained is of real and lasting value.