Big data: how it knows everything you do

By developing special algorithms, it's possible for organisations to comb the digital shoebox looking for subtle patterns.

Earlier this year the New York Times reported how Target had analysed its digital shoebox and worked out how to tell when a customer was pregnant from their purchase patterns.

It used that insight to send coupons to a pregnant teen before she'd told her parents the news. The challenge is balancing what the consumer may find valuable against what they might consider an affront.

Brandscreen grapples with that balance each day. The Sydney-based business deals in what's known as 'programmatic buying'.

In essence, it uses powerful computers and specially designed algorithms to work out who's looking at a web site and what they like based on their past online activities.

It uses that information to auction off an online advertisement slot to the highest bidder.

For advertisers there's a clear benefit in having their advert seen by the right person at the right time, so the bidding for an advertisement slot in a super-fast auction can be conducted while the person is still in the browser.

As Julian Tol, founder and CEO explains, "It's like a stock exchange where the asset is a human being". On sites such as Google, Microsoft, Apple and Amazon, where there can be 2 million visitors a second, the systems have to carve up the opportunities quickly, making over 200 million decisions each second.

Local start-up Adbiddr offers a similar service for companies wanting to load advertisements on digital signs, but instead of tracking what people are doing online, it photographs people as they approach the sign and then auctions an advertisement spot based on the age and gender of the person looking at the sign at that precise moment.

Tol predicts that the ad-auction concept will be extended so that people watching smart TVs will see adverts that remind them that their home insurance is due next month, and if they press the red button now, they'll get a special deal.

It's not the only application of big data that's in the wings. SceneTap has been launched on a trial basis in the US and takes video feeds in clubs and bars.

It analyses the gender ratios and age groups, to be displayed on a mobile phone app to let people select venues that aren't too crowded and seem to have the right mix of people.

EMC, which has sponsored Rick Smolan's big data project, believes that insurers might one day analyse data such as how old you are or where you park the car, then use telematics to measure how you're driving three times a minute and offer you insurance based on that information.

They might also send reminders when your tyres might need replacing, based on your driving history.

Researchers at the University of Adelaide are working on algorithms that can analyse data collected by sensors placed in and around an elderly person's home in order to determine if they're behaving as expected, or if there might be a need to send someone to check them out.

Chris Gartlan, managing director of the Asia Pacific region for GoodData, which provides business intelligence solutions for big data analysis, says that smart analysis allows governments and businesses to refine and improve their services, and tackle problems as they arise in real time.

But he also notes: "It's a new area and consumers aren't as aware as they should be. There are loads of things that you might be able to collect — even your likes on Facebook and Twitter." He recommends consumers and citizens "know the information you are giving over — it's information that can be used to track behaviour".

Applied judiciously, big data has the capacity to provide enormous insights about the human condition and provide a safety net for those at risk.

It can ensure that valuable and useful information reaches us in a timely manner. Applied cynically, big data can invade our privacy, clog our inboxes as well as manipulate our decisions.

So, do you think Big Data is a good thing for consumers? Tell us what you think in the comments section below.