TB Research - Mobile 3D Capture

Alan Melling
Alan Melling - March 16, 2018

Capture true 3D with just your phone?

It was a big week for 3D capture! So many awesome demos!

We're all familiar with true 3D capture from special cameras like Matterport or GeoCV, which capture 3D models of objects in a room using "structured light" depth sensors. These cameras project thousands of infrared points of light to make their measurements. But what if you didn't need to buy a special camera? 


6D.ai's first public sneak-peak of its 3D reconstruction from monoscopic rgb camera


This is where we jump onto a speeding train. There's been an intense amount of work and invention happening in augmented reality, but instead of imagining a wacky pair of smart-glasses that put virtual objects into your field of view, let's think about a different way to use the underlying technology.

Those glasses have to track where they are in 3D space, so to help developers use this new tech, Apple and Google have released toolkits (ARKit and ARCore, respectively) which unite the device's camera with the motion sensors to track where it is in 3D space.

This opens up a whole new realm of possibilities for smartphones.  


AR is a powerful new way to view architecture (above), but can we also use this to capture as well?


Once we know where the camera is, we can start to extract measurements from the scene using a century-old technique called photogrammetry which was initially used by the military starting in the late 1800s to measure distances using hot-air balloons and a newfangled technology called the film camera. Now with fast computers, instead of taking a handful of measurements, we're taking millions or billions! This is how Google made Google Earth, by extracting 3D information from tens of millions of aerial photos.

Tourbuzz experimented last year with photogrammetric 3D reconstruction of entire houses, but while that workflow is feasible for Hollywood movies and video games, it's still impractical for everyday interior capture for real estate marketing. The current capture and processing techniques are just simply a pain in the butt. (Still intrigued? Az Balabanian has a good breakdown)

These existing workflows all have their problems, but things may be changing — and with most things in life, you can blame the smartphone.


Why now? 

  • New positional tracking tech (i.e. ARKit) - ensures we reconstruct a single scene using all available imagery. Without this, there are too many ambiguities in a scene which often cause failure.
  • Better mobile cameras - improvements to sharpness, dynamic range, image stabilization
  • Better processors - mobile GPUs might be just enough compute power to not need a big machine
  • Smarter algorithms - traditional algorithms are brittle and rely on too many assumptions about the world, but newer learning-based approaches mean lower chance of failure. By teaching the computers to see like humans (sort-of), we'll be able to automatically find and fix problems with the 3D reconstructions. AI for the win!


Look to AEC

Photogrammetric capture may have had its breakthroughs in mapmaking, and then in movie visual effects and now in video games, but the Architecture, Engineering, and Construction sector (AEC) is perhaps the closest sibling to Real Estate. Even though their uses tend to have more priority on accuracy than on visual appeal, they deal with many of the same real-world problems of budget, time, and technology. 

AEC has relied on expensive but accurate LiDAR scanners (the price of a luxury car), but shrinking budgets have startups and established companies jumping into mobile 3D capture. If you're an architecture+3D nerd like me, you should check out Spar3D's blog, which is a great source for AEC tech trends that may filter into Real Estate (and vice-versa).


        AEC is looking for low-cost alternatives to expensive laser scanners


Watch out for the hype!

This will take time, probably even more than you may think. As with most innovations, people overestimate the impacts in the short-term and underestimate in the long-term. We're nerds — we get excited.

A necessary de-hype: In the short-term, we won't see completely photorealistic 3D reconstruction of full houses that you can walk around in in VR (sorry). Even with the innovation from AEC, film, and the gaming industries, the visual standards for Real Estate marketing in 3D are just simply higher. Agents and homebuyers expect perfectly photorealistic imagery and have low tolerance for artifacts or errors. It's understandable: "Is that a real hole in the wall, or is it a bad capture artifact?" This is something that Matterport and GeoCV have struggled with, and it continue to be a struggle for all of us, despite all the new shiny tech.

In the meantime, we can anticipate to see 3D mapping show up in augmented reality experiences, so that when your virtual pet runs behind the couch, it'll actually disappear. Even that alone is a triumph worth celebrating, because those moments of delight will build habits in consumers and will attract more investment into the underlying hard technical problems that are still unsolved.


The long-term

There's a next generation of capture+compute+consume coming to the market, and we're going to need a next generation of content producers. There's gold in those hills.

So instead of fearing being replaced by a noob with a cellphone, professional photographers might come to view the glowing rectangle in their pocket as a new superpower. 3D capture opens up entire new ways to tell the story of a house, especially as agents and homebuyers start to adopt immersive media into their lives and will come to expect 3D-first experiences that they can actually walk inside, and not just look around

And, of course, Tourbuzz Research will continue to track these new developments and help navigate all of the possibilities.

With supercomputer cameras in our pockets, we've all become Computational Photographers. I'm excited for the day when we can all be Photogrammetrists as well.