Big data: how it knows everything you do

If you're one of the 68% of Australians aged 15-65 who owns a smartphone, then you're playing a key role in the biggest human experiment on the planet, because you're now a sensor.

Not only does that phone know where you are, it's the conduit to a vault of information about you stored on web sites, in clouds and on social networks.

It can capture biometric information using the camera and authenticate your identity via a voiceprint.

Internationally acclaimed photographer Rick Smolan — the man behind the A Day in the Life of series of books — has of late focused his attention on the rise and exploitation of so-called big data.

He has recently published a book detailing how big data has the capacity to both enrich and challenge the human experience.

"My son is 10 years old. He asked, 'what's big data?'. I told him to imagine if you only ever had one eye, and then suddenly opened the other. It's not just more data — it's a different dimension; you are seeing things that you couldn't see before," says Smolan. "When I started I was sceptical. Now I think that big data will be more important than the internet."

It's a big call.

Smolan counts Marissa Mayer (CEO of Yahoo!) as a friend and says that she describes big data as the early stages of a global nervous system in which every connected consumer or citizen is a sensor. As Smolan notes, "It's something our species has never had before."

How that might play out remains to be seen, but a recent partnership between supercomputer company SGI and the University of Illinois provides one clue.

Together they created the 'Global Twitter Heartbeat', which analyses 50 million tweets a day, their location and sentiment — and then visualises that as an infographic.

It's already been used to monitor the reaction to the US Presidential Election and Hurricane Sandy.

What is Big Data?

To properly understand what big data is, it's perhaps useful to start with what it's not.

It's tempting to think of big data as being just lots more data, but big data embraces information stored not just in databases, spreadsheets and reports; it's also data collected by sensors, CCTV footage, YouTube videos and social networking posts. It's the firehose and every one of us is feeding that firehose constantly.

Technology analyst IDC estimates that by 2015 the 'digital universe' will comprise 8 zettabytes — that's 8 billion terabytes.

Even if you're a relatively discreet user of technology and social networks, your digital footprints still loom large.

Zeebly's Social Me application provides Facebook users with a glimpse of how they're perceived by analytics engines and how they compare to other Facebook users.

For example, I generally post in the mornings, often about science and beverages, and avoid exclamation marks. It's a fair cop, guv'nor. Social Me also knows that I post from an Android smartphone about 20% of the time.

It's not a huge leap to understand how valuable it might be for a company to know what I'm posting about, track my loyalty cards to uncover my spending habits and where I am — easily ascertained if I opt in to allow an app access to my phone's GPS capability.

I'm also a member of the Flybuys loyalty program and as a result Coles knows where I shop, how much I spend and what items I buy. And I'm not alone.

In April, Flybuys relaunched the entire program and sent cards to every household in Australia — 17 million in total — and invited people to register or re-register for the scheme.

As a result, Flybuys cleaned and updated its database, and learned the email addresses of every member. In fact, it probably knows what you're serving up for dinner tonight.