This Noah’s Ark contains not beasts, but bytes. With archaeologists and conservationists increasingly nervous about the threats from war, natural disasters and tourism to the world's most famous cultural sites, armies of 3D scanners, drones and cameras are out in the field.
Digital preservation has also become the buzzword in museums and exhibitions, and the end result is something essential, but also exciting; 3D virtual reality (VR) and augmented reality (AR) experiences for all. They could one day be the only traces left we have of ancient sites because history is under threat.
3D-printing the past
Perhaps the most famous example of recreating the past with data is Syria's Monumental Arch of Palmyra, which was destroyed by IS militants in 2015. The Institute for Digital Archaeology (IDA) has since been able to 3D-print a smaller version of despite it never being 3D scanned.
"We didn't have the luxury of using 3D surface-scanning tech because the Arch had already been reduced to rubble,” says Dr. Alexy Karenowska, the IDA's Director of Technology. “So we created 3D models using photos from all kinds of sources, including from tourists." This is the art of photogrammetry, and it's usefulness convinced the IDA to initiate the Million Image Database, a hub for photos of heritage sites, including objects.
Digital archaeology
However, photogrammetry has its limits. "Sometimes we use photogrammetry, but the end result of the scan has very complicated geometry that has to be optimised for VR and AR," says Piotr Baczynski, CEO at Immersion, which constructs AR and VR experiences. "We use it as a base, but we often have to do so much work on it that it's sometimes easier to start 3D modelling from scratch."
Digital archaeologists prefer to use a 3D surface-scanning tech called LiDAR (light detection and ranging), a kind of digital radar. "It’s essentially a high fidelity snapshot, in a given point in time, wrapped around a mesh," says Faizaan Ghauri, CEO of dynamic 3D mapping company WRLD. "These implementations are also quite memory intensive, need a high performance PC, and are not capable of running on a broad spectrum of devices and coverage networks."
A big data problem
For many sprawling heritage sites, basic digitisation is the first step, and it's often a hugely time-consuming process. At CyArk's digital preservation project at Ayutthaya in Thailand, staff have had to concentrate only on digitally preserving one key section, Wat Phra Si San Phet.
Here, scans and images (including aerial photos from drones) are slowly being turned into photo-real 3D models for future generations of students, tourists and cultural heritage enthusiasts to experience the site virtually anytime and anywhere.
Backing-up the massive 3D capture data files in the field is a key part of the job, and something that hasn't been easily possible until recently. "It essentially a big data-gathering problem," says Karenowska. "It's about organised data gathering and good data storage."
Drones & disasters
CyArk has chosen Ayutthaya for one simple reason; earthquakes. Ayutthaya was was destroyed in 1767 by a massive earthquake, but it's not the only one that could crumble at any minute. Having amassed digital records of over 200 heritage sites since 2003 (and turned them into 3D models), it's just finished its third field mission to Bagan in Myanmar where it used drones, laser scanners and photogrammetric rigs to capture 4TB of data.
This breathtaking landscape of around 2,200 temples was badly damaged by an 6.8 magnitude earthquake just a year ago. "Drones have become an integral part of our process, allowing our team to cover large portions of the site very efficiently," writes Scott Lee, Director of Production at CyArk, in a recent blog post.
"We used two DJI drones to fly nearly 400,000 square meters of the site in order to generate building footprints of numerous monuments [and] the data collected by the drones is used to generate a 3D model of the monuments captured, which is then used to extract contour lines for incorporation into the site-wide GIS mapping system." For individual structures, Faro laser scanners and robotic photogrammetric camera mounts were used to record individual structures.
The threat of mass tourism
At some particularly delicate heritage sites, digitisation is being used to create virtual reality experiences not for preservation purposes, but to dissuade visitors altogether. "It is a fact that tourism brings damage that is beyond repair to immovable culture relics," says Ding Xiaohong from the Dunhuang Research Academy, whose job it is to 3D-scan all 492 temples and caves at the Dunhuang Mogao caves in Western China.
Together they contain over 5,000 square meters of murals and 2,415 coloured clay sculptures, some made from clay, wood and straw as long ago as 366 AD. They're some of the finest examples of Buddhist art in China, but a surge in tourism is threatening to degrade them.
"Carbon dioxide exhaled from the visitors builds up in the caves, resulting in an increase in temperature and humidity, which further degrades the murals and coloured clay sculptures," says Xiaohong. So it's all being scanned to create an on-site and online VR experience that will hopefully mean tourists linger for less time in the actual caves.
Upping the resolution
"Technology advances day by day and it is playing an increasingly important role in culture relic protection," says Xiaohong, who knows the digital product has to match ever-increasing expectations. "We will keep applying the most cutting-edge technology, including 4K or even 8K (resolutions)."
It's entirely possible to 3D scan at under 1mm per pixel (Google Earth is about 1cm per pixel), but Ghauri doesn't think resolution is critical for VR and AR experiences. "The thing that induces presence is not when objects and landmarks look exactly like they do in the real world, but rather, capture the essence," says Ghauri.
Horizon of history
VR can also add a powerful time-travelling dimension to a geographic location. At the Palace of Culture and Science in Warsaw, Poland, Horizon of History hosts a virtual trip into the past to see what the city looked like in the 11th, 14th, and 16th centuries, as well as in 1944.
"Warsaw is a city that has been destroyed a few times in history – the city as you see it now was completely rebuilt after World War II, but before that it was destroyed by the Russians, the Swedish and many others," says Baczynski, whose team built Horizon of History.
It's the largest permanent VR installation in Europe, featuring eight HTC Vive VR headsets for viewers, but it's not just a passive experience. "You can interact with everyday objects from different time periods, so if you're in the 14th century you can shoot with a bow or try a traditional sword, and if you teleport to 1944 you can shoot with a machine gun as used by members of the resistance in Warsaw," says Baczynski.
Time-travelling in AR
Historical time-travel is a definite trend in AR and VR. The TimeLooper app is about spicing-up walking tours; don either a Google Cardboard-like VR headset or use a phone and you can relive past events in specific places.
For example, go to the Brandenburg Gate and relive the night of November 9, 1989 when the Berlin Wall came down. At London's Tower Bridge Exhibition you can experience the bridge's construction in 1892, while in New York outside City Hall you can 'see' George Washington's inauguration as the first President in 1789 going on around you.
It's not lost on the tech giants that using AR in museums and visitor attractions could be the spark that ignites a surge in AR and VR. The Detroit Institute of Arts (DIA) has a mobile tour called Lumin that uses Google’s Tango technology to give visitors with new ways to engage with its collection, but it's an imminent firmware update from another tech giant that is getting AR developers excited.
Importance of iOS 11
ARKit, a platform for developing AR apps for iPhone and iPad that premiered at WWDC, will come to Apple's suite of devices in a few months as part of iOS 11. Developers already have the beta version, including those at Immersion, who are working on an AR storytelling experience with journalism museum Newseum in Washington D.C.
Its Unabomber Cabin in AR is an attempt to recreate – in excruciating detail – the primitive cabin in Montana that the reclusive 'Unabomber' Ted Kaczynski was eventually discovered in by the FBI.
"We used the photos from the cabin taken on the day they found him and we've recreated every single object in there, including trash," says Baczynski. "Using the new Apple ARKit we've made an AR experience where you can take the cabin and put it in your backyard."
If the does have depth sensors, ARKit could change the way you experience every museum, heritage site and city walking tour.
Cue Microsoft
That might be AR on phones taken care of, but what about AR glasses? "The next step will be AR glasses that would be wirelessly connected to your phone to take advantage of the phone's processing power.
The glasses will just be responsible for the tracking of your location and the image recognition," says Baczynski. Cue rumours about an Apple product. However, Microsoft published a patent last year for a technology that creates a virtual representation of a physical location at a previous time period with 3D virtual data.
It sounds like TimeLooper, but this is all about the HoloLens. “The technology could make it possible to recreate the physical landscape of sites at a historical point in time, or even replay an event that happened there," says Ray Chohan, SVP, Corporate Strategy at global innovation analytics company PatSnap. "This could be a medieval battle scene, or a famous sporting event within the stadium where it took place, or it could be used to preserve world heritage sites."
VR or AR?
It's largely about accessibility. "VR is very immersive so great for telling stories, and it can put you in a highly intensive scene to help you understand historical events better," say Baczynski. "But AR is about to be so much easier to be a part of because of the number of devices compatible with AR. After iOS 11 there will be 300-400 million iPhone and iPads compatible with it."
For now, museums give you audio guides. Coming soon will be apps for phones and tablets and, in a few years, transparent AR glasses. The future may not be written, but the past is increasingly being reduced to data for preservation, of course, but also to help create the entertainment of tomorrow.
- Want to see all we know so far about Apple's AR glasses? Check out: Apple AR glasses release date, news and rumors
