Does the internet have a back-up plan?

Forget Tweetstorms

Network Disaster Recovery critical for the first world

At first it looks like the result of a spending spree at Go Outdoors. There's a table piled high with sleeping bags, tents, ration packs, dental kits, steel toecap boots, deckchairs, UHF/VHF radios, Savlon, water purifying tablets and even a bog-in-a-bag, but behind all of this is nothing short of a global telecoms network on wheels.

Seven trailers parked-up in a huge warehouse – some on wheels, others designed for 747 cargo holds – together create a network node capable of handling 15 Terabits per second. This is AT&T's top-secret Global Network Disaster Recovery centre somewhere in the south of England (we can't say where), and its US$16 million custom-built contents can save a city's communications infrastructure.

High stakes

The stakes are high because AT&T's network, while not publicly visible in the UK, constitutes a big slice of global telecoms and internet traffic. Worth $140 billion, it's made-up from over a million miles of network fibre at 155 sites in 60 countries. It transmits 67 petabytes of data every day. It's got over 116 million wireless customers, 649,000 Wi-Fi hotspots and 38 internet data centres.

If even one node of that network goes down during a disaster it's a big problem both for AT&T and for the people in the affected area, who are cut-off just when they most need to communicate. Ready to protect all of this – and keep everyone online and in contact with each other post-apocalypse – is a dedicated team on-call 24/7 for disaster-deployment.

The team – many of them volunteers from within the company – contains veterans of 70 'network events' ranging from the Colorado and California wildfires (2013), the Oklahoma tornado (spring 2013), Super Storm Sandy (autumn 2012), Hurricane Irene (2011), the Alabama/Tennessee tornados (2011) and the Santiago, Chile earthquake (2010).

This vast warehouse can turn a disaster zone into a communications hub. This is the tech behind tragedy.

ATT2
This mobile command centre is designed for overland missions as far as the Middle East

The biggest threats

"Most of the problems that we see are power-related," says Justin Williams, Network Disaster Recovery – International, AT&T Network Emergency Management, Preparedness & Response, who took TechRadar Pro on a tour of the facility, one of six that AT&T has around the globe (one in Europe and four in the US).

"Self-sufficiency is everything, so we take our own power with us," he says, explaining that his NDR team takes generators, fully charged batteries and everything else needed to restore the network in any given scenario.

We're not just talking about natural disasters; unpredictable social and political events are also problematic. Impromptu demos and protests can bring down a node in a network if the call activity goes catastrophically high, though AT&T is constantly probing its network for potential problems.

ATT3
US$16 million-worth of gear is ready for instant deployment

Communications is a major challenge after an 'event'. "You can't rely on on local people on the ground because in a disaster zone they will have their own family issues," says Williams. It's a story of constant network overload. "Calling your loved ones on New Year's Eve is difficult, but try it in an area where the network is toast and you've got no chance," he says.

The NDR team takes its own satellite phones (checked and charged every week) and its own managed data links via satellite because the main problem – as in Chile after the devastating earthquake of 2010 – was a network that was bursting. "In Chile the main problem was congestion levels, and it was very unreliable – and there were aftershocks, too, the impact of which we had to monitor."

The gear

The mobile trailers at AT&T's disposal include Trailer 365 (home to all that survival gear), a five-tonnes command centre and Trailer 180, a dense wavelength division multiplexing (DWDM) truck stuffed with servers, air-con units, generators (and the occasional tea-urn and microwave) alongside an endless number of FSI cables, each with 72 fibre connections.