R&D Deepdive: 3D Drone Scanning

Alan Melling
Alan Melling - September 08, 2017

How do you walk through a house that doesn’t exist?

With the aid of a handy drone, a hefty computer, and a little bit of patience, we’ll dive into how to 3D scan a neighborhood. Then we’ll see what it takes to put a 3D house in place and walk through it in virtual reality.

This is a deepdive into how emerging techniques like photogrammetry can give us new ways to capture a property. We’ll combine that with 3D modeling tools to give our homeowner a chance to live a moment in their new home before it’s even built!

Transcript from video:

Hey guys, I'm Alan Melling. I'm a researcher at Tourbuzz, and we've got a really cool project to show you.

The owners of this property designed a custom house, but they've never been able to walk through it – all they have is a floor plan, so I'm going to rebuild it as a 3D model, and then I'm going to 3D scan this whole area with this drone. We're going to combine that into an immersive experience so that they can get to see what their new home looks like. So let's try it out.

Alright, so let's get this thing in the air. There are a lot of automated apps that you just draw a perimeter and then it will automate the flight of the drone. For now, I'm going to keep doing all of this manually, just so I can make sure that we get what we need. Also, there's a lot of trees, so we've got to be careful.

So our flight plan today: we are going to do criss-crossing patterns over about a block and a half, and then we're going to do a bunch of 90-degree angle shots back and forth, and then we're going to do a bunch of downward looking shots, and a couple orbits in there. Hopefully this will give us the kind of coverage we need to ensure a good photogrammetric reconstruction. Droning is pretty tough work. :)

All right, now that we've got everything shot, I'm gonna take all that footage we shot in DNG RAW on on this Mavic, and since the Mavic doesn't have a huge dynamic range -- the shadows may be a bit too dark, the highlights may be a bit blown out -- I'm going to take the shadows and boost them up, take the highlights and pull them down. What that's going to do is push more of those details into the mid-tones so that the photogrammetry software will have more details that it can make a much smoother surface with.

Alright, so now that we have our processed images, we're going to throw them on this big desktop machine. I built it to do machine learning research, but it also happens to do VR really well and also photogrammetry kind of processing really well. So I'm going to take the images and put them into [3D photogrammetry software] Reality Capture.

Okay, so let's bring the images into a new project. We're going to bring the folder in, and as you can see, there's roughly 650 images. They have attached EXIF data that has GPS and focal length. All we've got to do is hit the start button.

So now that we gave the computer some time to think on that, it's produced a really high-quality mesh, and we are going to try to use that in two different places: one is a room scale VR experience that the homeowners can walk around in. And then also we're going to publish a 3D model on SketchFab and then embed that into a Tourbuzz tour.

Both Unity and Sketchfab have different graphics capabilities because of the different platforms they're on, so they're going to require two different workflows. But we're going to start by cropping the original model into just the surrounding block or so, maybe even less - it depends on what you want to show. And as you can see here, this is where each one of the drone shots were taken from and you can get a sense of the flight path.

So even though we've cropped our scene, we're still sitting at something like 30 million triangles, so let's simplify down to three million triangles. That's something that Unity can actually render real-time in virtual reality, and then let's texture it. So you only see the colored vertices right here, but once it's brought into Unity, it will be able to render the complete mesh, fully textured.

We're going to be importing that mesh into an existing unity project. Previously, we had taken the floor plan of the homeowners new house and converted into a 3D model using Sketchup, imported it into Unity, and lit it with the global illumination system so it looks really natural. But let's take it that one step further by dropping the scanned environment in there. As you can see, it's a rough sketch so far and we still have some tweaking to do.

But as cool as I think this view is, the real power of this came when I put the homeowner into their home for the first time in VR. They were able to walk around and experience not just the look of the house, but the feel of it. They could walk to each window and actually see what their view was going to be. But my favorite part is how he was able to use this experience to bring those insights to his architect, to get his house ever closer to the home he was imagining for his family.

So the VR stuff is fantastic. It's a whole new world, and I'll be covering a lot in the future. But we're looking to create as much engagement as we can now. I think this drone scanning technique adds a powerful new point of engagement to virtual tours. It's a new kind of moment for home buyers to interact with the house and see how it's situated within the neighborhood.

For this demo, I'm going to upload our model to Sketchfab. It's simple and robust, and we can embed it directly into a Tourbuzz tour using our custom tour links. The easiest way to upload to Sketchfab is to use Reality Capture's integrated Sketchfab uploader. You're going to use this if you don't want to add any additional 3D content to your scene. So just add your Sketchfab API token; this will connect your account, and if you don't have one you can create one here. So add a name for the model and any other info you want to include. I'd suggest simplifying the model to fit Sketchfab's limits; that way it'll show as best as it can over the internet. Just to show you what's possible here, I'm using the free edition of Sketchfab, but they do have pro pricing tiers for higher end needs as well.

So let's tweak some settings. We're going to change our shading model from `lit` to `shadeless`. This is because our model is derived from photographs and the lighting is baked in. Taking a look at our scene, we also want to take a good thumbnail view, so save the view, and save the settings, and exit back to the model.

So to put this into a Tourbuzz tour we're going to use an embed link, but we're going to add some options to make it integrate into the tour a bit better. That will produce a URL that we're going to copy; it's in quotation marks right after the `src=`. So copy that into your clipboard, and let's go to our tour.

In Tourbuzz, in the tour editor's `info` tab, scroll down to the custom link field name. I chose to name the link `3D Neighborhood`. I thought that had a good ring to it, and then paste the link into both URL fields. It should come as a default, but make sure that `open link in a pop-up` box is selected. That'll keep the homebuyer engaged in the tour. So save the tour, and let's preview it. And as you can see our fancy new `3D Neighborhood` custom link appears prominently in the tour's info panel, and it pops up right within the tour.

That was such a fun experiment to do, not just because it's new tech, but because it gives professional real estate photographers a genuinely new way to capture the attention of home buyers, adding real value to their experience. For this demo I used Reality Capture and Sketchfab, mainly because it's the tools I know, but there's a number of cloud services coming soon to make it easier for everyday.

To make this a true workhorse I would probably upgrade to a better drone. The Mavic is super-fun, but it's basically a cell phone in the sky. It's bigger brother, the Phantom 4 Pro+ has a bigger, better camera and produces phenomenal images that stitch together more reliably. It also has better proximity sensors, so you can rely on flight automation without having to sweat the trees and other houses.

Oh and I'll mention that Sketchfab has been crushing it with their WebVR support, so you can have full room-scale VR experiences right in the browser, right in your Tourbuzz tour. Get your drone working for more than just stills and videos.

I want some feedback on who's interested in this. Who wants to add this to their bag of tricks? I want to talk to you. Get ahold of me! I'm a people person! Anyway, alright guys. Get those drones up in the sky and I'll see you next time.

-Alan