Insights or invasion of privacy?
In a recent speech to the Centre for Internet Safety, Australia's privacy commissioner Timothy Pilgrim noted: "When you consider that much of the personal data available online is uploaded by individuals themselves via social media and how readily we trade our personal information in return for convenience or the use of a service, it is not surprising that some people might question whether privacy matters, or if it can even exist."
"In addition to the amount of unchecked personal information shared on social media sites, we also have the issue of internet tracking and profiling, which can create uncannily accurate digital profiles of users."
PayPal for example, processes 70 transactions worth an aggregate $5,000 each second of the day. Associated with each of those transactions there's data about the owner of the PayPal account (120 million plus people use the system) and their purchase history. For PayPal the real test of big data is whether it can use it to predict the future.
Dr. Mok Oh, until recently PayPal's chief scientist, says that by developing special mathematic algorithms it's possible to watch a shopper — for example, a mother buying toys — and then predict what she might do next based on analysis of similar shoppers in similar locations. "Toys R Us would die to know that," he says.
Armed with that insight, merchants could send coupons or adverts to a shopper on their mobile phone at just the right time. "They can serve up the right offer at the right time" says Dr. Oh, because the data can reveal "who's likely to buy what, where, when and how."
In the future, Dr. Oh predicts systems that automatically match PayPal activity to eBay or Facebook accounts, and then map and visualise the data for the marketing team.
How Big Data is being used
Not surprisingly, retailers are keen on the promise of big data.
StreetHawk is an Australian start-up that uses a cloud-based matching engine linked to a smartphone app, which can track shoppers.
When they come within 500m of a shop they frequent, the app can send them an alert that they're near a shop, or a discount coupon.
CEO Natasha Rawlings believes that people will be willing to give up some of their rights to privacy "as long as they are getting something".
The banks think the same, with all of the majors now investigating big data in earnest. The Commonwealth Bank has spent $1.1 billion on a new core computer system that manages all its customer accounts in real time and provides a window on what customers are doing, when and how.
It wants to use the data it has, plus the insights it can glean from social media (CommBank has over 235,000 people who like its Facebook page and over 9,000 followers on Twitter) to tailor and price products targeted at individual customers.
NAB is doing something similar by using the information it has to offer loyal and repeat customers better rates on deposits, and to act as an early warning system about people about to default on their loan payments
Dr. Rami Mukhtar, senior researcher at research institution NICTA, says that to a company or government, big data is like a large 'digital shoebox' in which all sorts of potentially useful information is stuffed.
But he warns: "A big storage devices is not going to give you a crystal ball. We run a lens over the digital shoebox, then you need to apply a science to understand what is happening."
The fact is that there's a lot of science going on because by better understanding what's in their digital shoebox, companies and governments can work out which of their customers and citizens are influential, how much it might cost to process an insurance claim, where there are opportunities to cross-sell and which people represent credit or security risks.
