Storage and the economics of social media analytics

Social Noise – Social media has made some significant “contributions” to big data but even more comes from the growing networks of sensors and monitors.  The great thing about social media though is that so many people volunteer to put in so much information about themselves, their wants/needs, friends and thoughts that enterprises are able to collect/analyze all of the data and use predictive technologies to plan for tomorrow.
As if that wasn’t good enough, all of that DIY work, every click, creates an even bigger digital shadow about you … 8-10 times more than the data you produce.
The data flood (some call it a swamp) is growing because:
–        Storage is cheap, it costs only about $600 for a drive to store all of the world’s music
–        There were 5B mobile phones in use last year and the number is growing
–        30B pieces of stuff are given to Facebook each month
–        Global data will grow in excess of 40 percent per year
–        Global IT investment will only increase 5 percent per year … bummer
That’s huge because it means a clear, steady growing market for storage – hard drives, solid- state, optical. It also means a healthy market for storage services like the cloud because everyone – individuals and companies – finds it more convenient to store the data somewhere accessible.

Keeps On Coming – The constant stream of data in any organization is almost overwhelming.  Without a lot of assistance and high-performance tools, making sense out of the data would be completely impossible.  The tools are there now; all we need is the folks to use them.

Your DIY/shadow stuff (an estimated 75 percent of the content) is in the hands of companies that are liable for it while they’re making sense of it, using it to make their own data:
–        Google processes about 24 PB of data per day
–        AT&T transfers about 19 PB of data through their networks each day
–        Four Large Hadron Collider experiments produce about 15 PB of data per year
–        The German Climate Computing Center (DKRZ) has a storage capacity of 60 PB of climate data
–        As of last June, Isohunt has about 10.8 PB of torrents indexed files
–        The Internet Archive contains about 3 PB of data, growing at the rate of about 100 TB per month
–        World of Warcraft has about 1.3 PB of game storage
–        Avatar required over 1 PB of 3D CGI
No wonder Google, IBM, Facebook, Amazon, MS, Apple, Oracle–everyone is building huge cloud storage farms around the globe.