Cloud on clouds: How weather data is a test-bed for new tech

Shall we talk about the weather? The eternal conversation starter is fast becoming a classic test-bed for the spread of big data, the Internet of Things and cloud storage. Data thus far mostly comes from dedicated weather stations, observation systems at airports, meteorological satellites, ocean buoys, ships, aircraft and – most recently – personal weather stations in people's homes that are connected to the internet.

If the latter are on the rise, so are sensor-packed smartphones, which are not only creating more demand for hyper-local weather data, but could themselves help meet that demand.

What role does the IoT play in weather prediction?

The spread of sensor-packed IoT devices is giving the weather industry more real time, hyper-local data that could be used – and already is – to develop more sophisticated weather prediction models. "More and more devices such as smartphones, thermostats or even sensors in vehicles are providing real-time data about temperature, humidity, environmental conditions and other variables," says Hendrik Hamann, manager of the Physical Analytics Group at IBM Research. "This data is the source for improving weather prediction systems."

The IoT clearly offers new ways in which to collect observations, and more cheaply, too. "Existing observation networks are limited in number but highly calibrated, because they're expensive to run and operate," says Charles Ewen, chief information officer at the Met Office. "However, the IoT offers a high density of observations of an unknown quality."

Is crowdsourced weather data reliable?

The forecasting models are incredibly reliable, but crowdsourcing the raw data – often not validated and unverified – comes with obvious risks in terms of accuracy. "We need to build systems which ensure the integrity of the data they feed in is valid, not just assume that more data will automatically make models more accurate," says Nick Clarke, Head of Analytics at data analytics company Tessella. "But ensuring the data which feeds into forecasting models is useful requires careful use of advanced analytics techniques."

Micro-weather data

Introducing the IoT into weather data collection relies on all of us chipping in. "People like to monitor their environment and their performances," says Christelle Manach, Marketing Product Manager for the Paris-based IoT startup Netatmo's Weather Station, which measures temperature, humidity, barometric pressure, CO2, sound and rainfall, and pairs with smartphones.

"They're willing to be surrounded by smart devices if it helps them … it's specifically important to measure air quality as the indoor air pollution has dangerous effects on health conditions including asthma, allergies and respiratory problems."

Netatmo, whose data is displayed on the Netatmo WeatherMap, is partnering with Weather Underground to add its data points to the Weather Underground Personal Weather Station (PWS) Network. The aim? Accurate forecasts for each specific data point location.