3D photography has never taken off in the same way as the two-dimensional art form, but the Apple Vision Pro and iPhone are aiming to change that. If they succeed, it could change the way we capture and experience memories.
That lofty goal is far from guaranteed. After all, Apple's introduction of the Vision Pro (its "first 3D camera") was simultaneously the most interesting and creepy part of a launch that often felt like a Black Mirror outtake.
The long list of meme-worthy Vision Pro launch moments means it's easy to forget that Apple announced a device that captures "spatial photos and videos in 3D". Considering the company's last camera (outside the iPhone) was the Apple QuickTake in 1994, that's a pretty big deal.
On the other hand, the Vision Pro is a camera you have to wear on your face – and Apple's almost comical demo led to widespread mockery, even from those who helped create it. This doesn't mean 3D photos and video are again doomed to remaining fringe novelties. It just means the Apple Vision Pro needs a device that can capture those 'spatial' memories, while it focuses on the playback experience.
Those 3D cameras will almost certainly be the iPhone and iPad. And while this will all likely come too soon for the iPhone 15, the iPhone 16 – or perhaps next year's iPad Pro – could add the necessary sensors and software to Apple's existing Lidar scanner to become the spatial cameras that the Apple Vision Pro desperately needs. If they do, then 3D photos and video could well become the next big smartphone camera feature.
Memory palace
Apple is far from the only company that's been exploring the possibilities of capturing memories in three dimensions. The idea itself goes all the way back to the first stereoscopes in the 1830s, but more recently we've seen the invite-only Wist (below) promise to let you shoot "immersive memories" on your phone that can then be re-lived on headsets like the Meta Quest 2.
An adjacent trend is the rise of apps like Polycam, which is the most popular 3D scanning app for iOS and Android. Polycam's founder, Chris Heinrich, told us that the Apple Vision Pro's spatial photos are an exciting moment and that current phone cameras will need to evolve to shoot them.
"Regular images and photos appear very flat in 3D, and do not live up to the potential of the hardware compared to 3D or stereoscopic content,' Heinrich says. But the Vision Pro's three-dimensional snaps are also different from full 3D scans. "The spatial photos and videos that they [Apple] announced are not proper 3D models, but rather stereoscopic photos which produce a 3D effect, and perhaps allow for some small head movement," he adds.
Still, while spatial photos and 3D scanning apps are different beasts, Polycam thinks the two will likely co-exist in many apps. "We anticipate that Apple will let third-party developers import spatial photos and videos from the Photos app like they do regular photos and videos," Heinrich adds. "And if this is the case, we will likely support importing, viewing, and editing spatial photos and videos with Polycam", he adds.
Future gazing
For many, the idea of capturing 3D memories will sound like a gimmick. After all, 3D TVs – which the Apple Vision Pro is arguably the natural successor to – were one of tech's biggest flops. And ten years ago, stereo cameras like the Fujifilm FinePix Real 3D W1 similarly failed to leave a cultural mark.
But even before the arrival of the Vision Pro, a growing number of experts were hailing 3D snapping as the next biggest thing since the arrival of digital photography. And now that Apple has entered the game, the question is how quickly might it finally take off?
The obvious conclusion from the Vision Pro launch was that the iPhone 15, tipped to land in September, would surely be able to shoot spatial photos and video for Apple's headset. But it might not be quite that simple, according to Polycam's Chris Heinrich.
He says that spatial photos "may require adding yet another camera to the iPhone that is spaced further (about the distance between our eyes) from the other cameras" and that this "would have cost and design tradeoffs that probably won't justify adding it just yet".
Instead, it looks more likely that 2024 will be the year for spatial memories to take off. "My guess is that they [Apple] will release the headset first, and if spatial photos and videos are a hit, they will consider adding it on the iPhone 16 or later", he adds.
What isn't in question is that the iPhone and iPad are far more suited to shooting spatial photos and videos than the Vision Pro itself. "I think it makes sense for Apple to support capturing spatial photos and videos on the iPhone because, firstly, capturing photos from a hand-held device is much more natural than capturing from a headset and, secondly, it plays into their strengths of owning a multi-device platform", he concludes.
Minority Report
Polycam isn't the only app that's excited about the potential for 3D photos and videos, following the arrival of the Vision Pro. The team behind Halide, still widely considered to be the best camera app around, also told us that it see lots of potential – even if it could mean waiting for the second version of VisionOS.
Ben Sandofsky, co-founder of Halide's development studio Lux, told us: "Over the years, we’ve watched Apple release the building blocks that culminated in the Vision Pro. For example, the iPhone 12 Pro saw the addition of the lidar sensor, unlocking room scanning and 3d photography. While these technologies are merely a tease for what expect to see next year, we’ve already used it to improve our 3D capture capabilities in Halide, and we can’t wait to get more," he said.
But there is a stumbling block for third-party spatial photography apps. "Our only disappointment is that it doesn’t appear the first release of Vision Pro will include capture capabilities for third-party apps," he noted. "We strongly suspect this is a time constraint issue, as Apple has a fairly good track record of supporting third-party camera apps. So assuming these capabilities arrive in a 'VisionOS 2.0', we’re definitely on board," he added.
What about the potential for future iPhones with 3D cameras? Like Polycam, Sandofsky also noted that current iPhones just aren't up to the task. "Lidar is only available on the 'Pro' iPhones starting with the iPhone 12 Pro, and some iPhones use ML [machine learning] to synthesize 3D data," he said. "That’s more than enough power to create effects like 'portrait mode', but when it comes to a realistic full 3D reproduction, we suspect that going from an iPhone to Vision Pro will feel like the move from standard definition to high definition televisions," he added.
In a separate Tweet, the Halide team concluded that "current iPhones can't create proper spatial captures the way the headset [Apple Vision Pro] can. We'll need a significant improvement in terms of processing power and sensor fidelity and quality to make that work" they added.
Given the rate of change on phone cameras – and the relative plateauing of 2D camera features – there is a chance that an iPhone 16 could still become Apple's first phone with stereoscopic lenses for capturing 3D photos and video.
But it's a way off and, alongside the technical challenge of making that happen, Apple will also need to work hard on getting the average person as excited about 'spatial' memories as iOS developers are.
As our hands-on Apple Vision Pro review concluded following a demo of the feature, "it was alike a postcard from Minority Report but instead of pre-cogs, we have a past-cog Vision Pro letting us relive moments like never before".
That definitely sounds like a fun sci-fi experience – but as the last decade of VR headsets has taught us, there's a big difference between making science fiction a reality and turning that into mainstream success.
Update June 15, 2023: we've updated this article with comments given to us about the Vision Pro and spatial photography from the camera app Halide.
