Big Data: creating value from the networked economy

The term Big Data seems an apt description for this rapidly growing supply of information, and if we take Wikipedia's definition, is "a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing application".

This has driven the growth of new technologies to do this job, such as NoSQL, Hadoop and MPP databases. These technologies in turn power the business intelligence and real-time software that, in a nutshell, make sense of big data to turn it into understandable, actionable insights for businesses to make future decisions based upon. Enter, predictive analytics.

Finding order from the chaos

Consumers come across predictive analytics on a daily basis, albeit unknowingly. Between weather forecasts, betting odds, insurance premiums and credit analyses, the touch-points are numerous.

Alongside this trend, businesses too are keen to cash in, and rightly so! Predictive analytics technology is the core 'enabler' of big data, allowing businesses to quickly interpret information - sometimes from many different data sources - in real time, and respond accordingly.

This could be anything from anticipating customer needs, forecasting wider market trends or managing risk, which in turn offer a competitive advantage, the ability to drive new opportunities and ultimately increase revenue.

Take retail for example. Now that we're currently experiencing a summer heatwave, many retailers will be looking at how they can exploit this to their advantage using predictive analysis. If a customer bought a gas barbeque last month, will they need more gas canisters this week?

And what is the purchase frequency to anticipate demand when they do buy? How can promotional cross-sell and upsell be incorporated to increase transaction size, number of items, and revenue?

This information provides the business with the ability to forecast, in real-time, the likelihood that a customer will buy, abandon or indeed go to a competitor - which gives them the power to save the situation by providing targeted deals, or offering informed services if needed.

While currently a fledgling market, the use of predictive analytics is set to rise. With growing data volumes, predictive analytics is firmly on the agenda; in fact 2013 research from SAP found that for 60% of businesses, predictive analytics is already an investment priority. Additionally, over two-thirds of companies think that predictive analytics will be an investment priority for them within the next five years.

The use of this technology is placing data firmly at the heart of organisational decision-making, across all sorts of sectors and industries. Businesses are now able to not only 'crunch the numbers' and make sense of the swathes of information that they are gathering, but also use historical data to make predictions and extrapolate actionable insight as they try to navigate an uncertain future.

Introducing in-memory

When discussing the future of storage in 2006, computer scientist, Jim Gray was famously quoted as saying the "tape is dead". The hard disk was then cited as taking over tape; equally the role of the hard disk was then to be taken over by main memory. Jim made these predictions based on the way in which hardware was continuing to diversify; all of which were the first steps to realising in-memory computing. He was right.

With in-memory, information is stored in the RAM of dedicated servers rather than in databases operating on slower disk drives. In-memory computing helps businesses to quickly detect patterns, analyse massive data volumes on-the-fly, and perform their operations much more quickly than was previously possible.

Existing in-memory technology can process data up to 10,000 times faster than legacy databases. These tremendous advances have increasingly placed pressure on software development; however there are lots of opportunities and directions in-memory can be taken. We're already seeing this in several key industries, for example sport.