A sensible approach to big data

This vendor-written tech primer has been edited by Network World to eliminate product promotion, but readers should note it will likely favor the submitter’s approach.

Big data is getting a lot of coverage of late and with good reason. We live in a world that is fast becoming overwhelmed by information. Ninety percent of the world’s data was created in just the last two years. From software applications and social media to Internet search results and the ever-present email, the rate of data creation is growing exponentially with no signs of slowing.

This has resulted in somewhat of a “dog catching the car” scenario. Companies have for years seen information as the Holy Grail of competitive differentiation — if only we had more customer data, if only we knew more about market patterns, if only our equipment could tell us when it was going to fail. In the blink of a cosmic eye, we have gone from thirsting for this information to drowning in it, leaving us to ask the question “now what?” Many see big data as the answer — and the key to making “if only” a reality.

Despite all the focus, the actual meaning of big data and its application aren’t always clearly articulated, but the concept is actually a fairly simple one.

At the most basic level, big data is just as it sounds — volumes of information that are extremely large and growing. That can mean data already being generated within an organization by systems or machines, like manufacturing output or customer buying patterns; or increasingly, data that’s available for companies to buy, provided from sources like social media vehicles or Internet search providers.

The problem comes when the volume gets so big that it can’t be effectively managed in a traditional “batch processing” way. If it takes many hours or even days to process large volumes of data, the information quickly loses its value. Enter big data solutions designed to turn this information tsunami into an information gold mine.

Because of this potential and all the hype, companies understandably want to make sure they don’t miss out on the big data phenomenon. Truth is, we’re far from reaching a point where more than a small percentage of businesses truly have a big data problem.

Big data opportunities, on the other hand, are a zebra of a different stripe. Certainly, the possibility of making information a competitive advantage is a real one, more viable in fact than it has ever been. The question is whether big data holds the key to achieving that goal. For some, the answer will be a resounding yes. For others, however, big data could end up being a costly and unnecessary distraction.

How do you know where your company falls? Some of the best advice about how to address the big data question can probably be summed up by admonitions that we frequently give to our children:

* Set the table. The first step in any data project (big or otherwise) is to clearly establish what you’re trying to accomplish. Additionally, it can be helpful to establish a visual picture or framework that organizes the business around an end game and breaks down the problem into consumable, achievable initiatives. With all the excitement around the big data concept, it’s easy to get pulled into an initiative with objectives that are likely to change as you go along. Defining upfront what you need helps determine if big data is the answer and puts parameters around what you undertake if it is.