Big Data: creating value from the networked economy

Most of us are now aware on some level that information is being collected about us all the time. Whether that's mobile usage, buying behaviour, web-browsing habits…the list really does go on.

We live in an "always on" culture in which everyone is digitally empowered. There are officially more mobile devices than people in the world - over 15 billion of them in fact, connecting us to the people and information we need to share, shop and consume.

And with the internet of things permeating both the workplace and the home, this "networked economy" is set to widen, connecting everything from your fridge, to your car, to the office coffee pot.

As a result, data is becoming more and more deeply embedded in our lives, and is exploding - doubling about every 18 months. It's unavoidable.

Not just our future, but our present

To take one example - try booking a flight abroad without big data. For a boarding pass to be generated, your itinerary needs to pass through a number of massive databases - from ticketing to no-fly lists - before you get confirmation from whatever online booking system you use.

You may then want to check what the weather's like in your intended destination, perhaps organise a hotel, and make sure you'll have wireless access once you arrive. Interestingly, but perhaps not surprisingly, Big Data plays a major role in completing each of these fairly routine tasks.

Now, extrapolate this from the perspective of an individual to cover a whole department, line of business or market unit, and then add customers, suppliers and partners into the equation. All of a sudden you have an astronomical amount of information being generated on a daily basis which, if analysed and understood, can deliver a 360 degree picture of your business at any point in time.

It's important to consider upfront why you are collecting this data, and what it is you want to achieve. Once that's established, the benefit - and the real competitive advantage that can be gained - is that responding to this information as its filtered back into the business enables you to become more agile to change; responding to market fluctuations or an increase in demand.

This presents an opportunity for businesses to rethink the way they operate, and drive unprecedented change. It's also requiring them to move faster than ever. To get - and stay - ahead of the competition, companies must not only sense the present, but see the future and proactively shape it to their advantage. The key to this lies in the deluge of data that they are sitting on.

An age-old practice

Human beings have collected and stored information since the dawn of our species, and it is this habit that has allowed us to pass on knowledge from one generation to the next, so that we can avoid duplicating previous efforts and ultimately, develop beyond our forerunners.

From markings on cave walls, to slate, paper scrolls, punched cards, the ever-faithful floppy disk, USB, and more recently, the cloud. The evolution of data storage to where we are today means that we can now theoretically collect unlimited data - ideal for a world in which many terabytes of information is now being generated every single second.

Structuring the unstructured

For many years, organisations have been accumulating information, but not necessarily doing much with it. This stockpiled information is often rather ominously termed 'dark data' - neglected data that accumulates in log files and archives that nobody knows what to do with.

Although it never sees the light of day, no one wants to get rid of it because it might prove useful at some point. Much of this data is also 'unstructured' - unorganised and raw, in fact, Gartner estimates that roughly 80% of all corporate data is unstructured.