• Follow us

Gadgets

How Oculus squeezed sophisticated tracking into pipsqueak hardware – TechCrunch

Making the VR experience simple and portable was the main goal of the Oculus Quest, and it definitely accomplishes that. But going from things in the room tracking your headset to your headset tracking things in the room was a complex process. I talked with Facebook CTO Mike Schroepfer (“Schrep”) about the journey from “outside-in” to “inside-out.”

When you move your head and hands around with a VR headset and controllers, some part of the system has to track exactly where those things are at all times. There are two ways this is generally attempted.

One approach is to have sensors in the room you’re in, watching the devices and their embedded LEDs closely — looking from the outside in. The other is to have the sensors on the headset itself, which watches for signals in the room — looking from the inside out.

Both have their merits, but if you want a system to be wireless, your best bet is inside-out, since you don’t have to wirelessly send signals between the headset and the computer doing the actual position tracking, which can add hated latency to the experience.

Facebook and Oculus set a goal a few years back to achieve not just inside-out tracking, but make it as good or better than the wired systems that run on high-end PCs. And it would have to run anywhere, not just in a set scene with boundaries set by beacons or something, and do so within seconds of putting it on. The result is the impressive Quest headset, which succeeded with flying colors at this task (though it’s not much of a leap in others).

What’s impressive about it isn’t just that it can track objects around it and translate that to an accurate 3D position of itself, but that it can do so in real time on a chip with a fraction of the power of an ordinary computer.

“I’m unaware of any system that’s anywhere near this level of performance,” said Schroepfer. “In the early days there were a lot of debates about whether it would even work or not.”

Our hope is that for the long run, for most consumer applications, it’s going to all be inside-out tracking. The term for what the headset does is simultaneous localization and mapping, or SLAM. It basically means building a map of your environment in 3D while also figuring out where you are in that map. Naturally robots have been doing this for some time, but they generally use specialized hardware like lidar, and have a more powerful processor at their disposal. All the new headsets would have are ordinary cameras.

“In a warehouse, I can make sure my lighting is right, I can put fiducials on the wall, which are markers that can help reset things if I get errors — that’s like a dramatic simplification of the problem, you know?,” Schroepfer pointed out. “I’m not asking you to put fiducials up on your walls. We don’t make you put QR codes or precisely positioned GPS coordinates around your house.”

quest top

“It’s never seen your living room before, and it just has to work. And in a relatively constrained computing environment — we’ve got a mobile CPU in this thing. And most of that mobile CPU is going to the content, too. The robot isn’t playing Beat Saber at the same time it’s cruising though the warehouse.”

It’s a difficult problem in multiple dimensions, then, which is why the team has been working on it for years. Ultimately, several factors came together. One was simply that mobile chips became powerful enough that something like this is even possible. But Facebook can’t really take credit for that.

More important was the ongoing work in computer vision that Facebook’s AI division has been doing under the eye of Yann LeCun and others there. Machine learning models frontload a lot of the processing necessary for computer vision problems, and the resulting inference engines are lighter weight, if not necessarily well understood. Putting efficient, edge-oriented machine learning to work inched this problem closer to having a possible solution.

Most of the labor, however, went into the complex interactions of the multiple systems that interact in real time to do the SLAM work.

“I wish I could tell you it’s just this really clever formula, but there’s lots of bits to get this to work,” Schroepfer said. “For example, you have an IMU on the system, an inertial measurement unit, and that runs at a very high frequency, maybe 1000 Hz, much higher than the rest of the system [i.e. the sensors, not the processor]. But it has a lot of error. And then we run the tracker and mapper on separate threads. And actually we multi-threaded the mapper, because it’s the most expensive part [i.e. computationally]. Multi-threaded programming is a pain to begin with, but you do it across these three, and then they share data in interesting ways to make it quick.”

Schroepfer caught himself here; “I’d have to spend like three hours to take you through all the grungy bits.”

Part of the process of creating Insight was also extensive testing, for which they used a commercial motion tracking rig as ground truth. They’d track a user playing with the headset and controllers, and using the OptiTrack setup measure the precise motions made.

optitrack

Testing with the OptiTrack system

To see how the algorithms and sensing system performed, they’d basically play back the data from that session to a simulated version of it: video of what the camera saw, data from the IMU and any other relevant metrics. If the simulation was close to the ground truth they’d collected externally, good. If it wasn’t, the engineers would adjust the system’s parameters and they’d run the simulation again. Over time, the smaller, more efficient system drew closer and closer to producing the same tracking data the OptiTrack rig had recorded.

Ultimately it needed to be as good or better than the standard Rift headset. Years after the original, no one would buy a headset that was a step down in any way, no matter how much cheaper it was.

“It’s one thing to say, well my error rate compared to ground truth is whatever, but how does it actually manifest in terms of the whole experience?” said Schroepfer. “As we got towards the end of development, we actually had a couple passionate Beat Saber players on the team, and they would play on the Rift and on the Quest. And the goal was, the same person should be able to get the same high score or better. That was a good way to reset our micro-metrics and say, well this is what we actually need to achieve the end experience that people want.”

The computer vision team here, they’re pretty bullish on cameras with really powerful algorithms behind them being the solution to many problems. It doesn’t hurt that it’s cheaper, too. Lidar is expensive enough that even auto manufacturers are careful how they implement it, and time-of-flight or structured-light approaches like Kinect also bring the cost up. Yet they massively simplify the problem, being 3D sensing tools to begin with.

“What we said was, can we get just as good without that? Because it will dramatically reduce the long-term cost of this product,” he said. “When you’re talking to the computer vision team here, they’re pretty bullish on cameras with really powerful algorithms behind them being the solution to many problems. So our hope is that for the long run, for most consumer applications, it’s going to all be inside-out tracking.”

I pointed out that VR is not considered by all to be a healthy industry, and that technological solutions may not do much to solve a more multi-layered problem.

Schroepfer replied that there are basically three problems facing VR adoption: cost, friction and content. Cost is self-explanatory, but it would be wrong to say it’s gotten a lot cheaper over the years. PlayStation VR established a low-cost entry early on, but “real” VR has remained expensive. Friction is how difficult it is to get from “open the box” to “play a game,” and historically has been a sticking point for VR. Oculus Quest addresses both these issues quite well, being at $400 and, as our review noted, very easy to just pick up and use. All that computer vision work wasn’t for nothing.

Content is still thin on the ground, though. There have been some hits, like Superhot and Beat Saber, but nothing to really draw crowds to the platform (if it can be called that).

“What we’re seeing is, as we get these headsets out and in developers hands, that people come up with all sorts of creative ideas. I think we’re in the early stages — these platforms take some time to marinate,” Schroepfer admitted. “I think everyone should be patient, it’s going to take a while. But this is the way we’re approaching it, we’re just going to keep plugging away, building better content, better experiences, better headsets as fast as we can.”

Read More



Leave A Comment

More News

Digital Trends

Here’s how you can watch UFC fights live 2019-09-14 13:41:06Cord cutting is a great way to save money, but watching sports online can be tricky due to restrictions. If you want to stream UFC, the world’s

How tough is the iPhone 11 Pro? New 2019-09-14 13:36:12Apple has released a pair of ads that literally throws the book, a refrigerator vegetable drawer, some kid's toy box, your secret rubber duckie colle

Hubble scientists find new way to measure the 2019-09-14 13:30:11A new Hubble image shows a type of star cluster called a globular cluster, and reveals clues about how star clusters form and grow by looking at objec

The best 4K TVs under $500 2019-09-14 13:17:13These days, you can land a pretty epic 4K TV for under $500. Still, sorting through the litany of options online can be both time-consuming and overwh

Google will promote original reporting in its search 2019-09-14 13:01:56Google has announced it will elevate original reporting in search and news results, promoting in-depth news coverage which includes elements of invest

The best new podcasts 2019-09-14 13:00:50Feel like you’re drowning in podcasts? In this weekly series, we’ll help you pick out the best of the new and returning shows. This week,

Spin me right round: Mars 2020 rover gets 2019-09-14 12:01:59NASA's Mars 2020 has been subjected to a battery of tests to ensure it's ready to touch down. The latest test was a spin on a rotating table to chec

Windows 10 update to fix Cortana bug breaks 2019-09-14 11:02:46A fix rolled out for a Cortana bug in Windows 10 has caused further issues of its own, by breaking the start menu of some users.

How to control your iOS app permissions 2019-09-14 10:00:18Your iOS apps have a lot of information about you, but that doesn't mean they should share it with the world. Take a few minutes to find out what you

How to create a playlist on YouTube 2019-09-14 09:00:48Want to make a playlist of YouTube videos? It's easier than you think. Whether you're just grouping together your favorite videos or you just need a

Fitbit Versa 2 review: The subscription smartwatch 2019-09-14 09:00:00The Fitbit Versa 2's multi-day battery, smartwatch features, and $200 price should set it apart. But it falls short in an unexpected way -- fitness.

The best new music this week: Sandy (Alex 2019-09-14 08:00:41Looking for the best new music? Each week, we scour the internet to find the most compelling new releases just for you.

Engadget RSS Feed

Mercedes' latest high-tech concept car is a throwback 2019-09-14 14:15:00Many of Mercedes' concept cars are focused squarely on the future, but its newest example goes back in time even as it aims to go forward. The autom

Recommended Reading: The redesigned WWE Network 2019-09-14 13:00:00 WWE Network 2.0: How WWE rebuilt its streaming service after a split with Disney Chris Welch, The Verge After a flashy reveal at CES a few years ag

Hitting the Books: 'Dirty bomb' fears spawned America's 2019-09-14 12:00:00Welcome to Hitting the Books. With less than one in five Americans reading just for fun these days, we've done the hard work for you by scouring the

This week in tech history: Apple pulls the 2019-09-14 11:00:00At Engadget, we spend every day looking at how technology will shape the future. But it's also important to look back at how far we've come. That's

The Morning After: MoviePass shuts down 2019-09-14 10:45:00Hey, good morning! You look fabulous. Apple's big iPhone 11 event revealed the biggest news of the week, but that's not all that happened. Tesla we

Amazon's 'Undone' takes rotoscope animation to new heights 2019-09-14 10:00:00Undone, Amazon Prime Video's first animated series by Bojack Horseman writers Kate Purdy and Raphael Bob-Waksberg, is a staggering work of art. It st

Galaxy Tab S6 review: Good notepad, bad notebook 2019-09-14 09:00:00While the rest of the industry may have given up on making premium Android tablets, Samsung isn't quitting just yet. It still believes Google's soft

Sling TV now works in Apple's Safari browser 2019-09-14 00:00:00You can now stream both live and on-demand Sling TV content within Apple's Safari browser for desktop. The Dish-owned over-the-top internet TV servic

The creators of 'Silicon Valley' are making a 2019-09-13 22:10:00The co-creators of Silicon Valley are developing a series on the origins of Match.com, according to Variety. John Altschuler and David Krinsky are rep

Decades-old missing person case solved with Google Earth 2019-09-13 18:52:00The body of a man who had been missing for nearly 22 years was found this week thanks to Google Maps, according to the Sun Sentinel. Remains of 40-yea

Disney CEO Bob Iger resigns from Apple board 2019-09-13 17:39:00Bob Iger, the long-time Disney CEO, has resigned from Apple's board of directors. Apple made the announcement on Friday in an SEC filing, where it re

Amazon-owned Whole Foods cuts medical benefits for part-time 2019-09-13 17:28:00Not all is well with Whole Foods in the Amazon era. The grocery chain has confirmed to Business Insider that it's cutting medical benefits for part-

WIRED

A Super-Powered iPhone Chip, a Potential HIV Treatment, 2019-09-12 18:03:57Catch up on the most important news from today in two minutes or less.

How to Watch Tonight's Democratic Primary Debate 2019-09-12 12:36:52Joe Biden and Elizabeth Warren will be onstage together at last.

Need Some Fashion Advice? Just Ask Stitch Fix's 2019-09-12 09:00:00Stitch Fix is launching a new service, driven by machine learning, that builds an outfit to suit your personal style.

The Best iPhone 11 Preorder Deals (and Which 2019-09-12 09:00:00Are you planning to buy an iPhone 11, iPhone 11 Pro, or iPhone 11 Pro Max? Here's the WIRED guide to choosing between them, how to preorder, and the

Best iPads (2019): Which New iPad Should You 2019-09-12 08:00:00Choosing an iPad is more complicated than it needs to be, but we're here to help with our complete guide to all of Apple's tablets, from the new 201

The Biggest iPhone News Is the Apple U1 2019-09-12 07:00:00By embracing ultra-wideband location tech, Apple has a chance to reshape experiences way beyond AirDrop.

Adam Savage’s New VR App Lets You Visit 2019-09-11 12:00:00Strap on an Oculus headset and approach the workbench as the world's best makers show you how they craft their creations.

5 Great Ways to Trade In or Sell 2019-09-11 08:00:00So you’re planning to buy a new iPhone. Don’t forget to cash in on your old one through sites like Decluttr and Gazelle, or Apple’s

Apple iPhone Event 2019: Everything That Was Announced 2019-09-10 16:04:37New iPhones, of course, plus quite a bit more.

iPhone 11 and iPhone 11 Pro: Price, Specs, 2019-09-10 15:26:05The company showed off three new models of iPhone on Tuesday. The new flagship, called iPhone 11 Pro, has a camera that looks truly nutty.

Apple Event Breakdown, a Honkin’-Huge Car Screen, and 2019-09-10 15:22:18Catch up on the most important news from today in two minutes or less.

Apple Watch Series 5: Price, Specs, Release Date 2019-09-10 14:42:27The new Apple Watch is here, with new cases and a face that never turns off.

Gadgets | Popular Science

With thousands missing and displaced after Dorian, the 2019-09-12 18:47:37Recovery efforts on these northern Bahamian islands have only just begun, and many small villages have not yet received aid.

A ban on flavored vaping products could help 2019-09-12 18:00:00The White House and the Department of Health and Human Services announced this week that they’re readying a ban on flavored e-cigarettes, which

This distant world is a lot like Earth, 2019-09-12 16:05:53“It’s the closest we have come to detecting some kind of environment similar to the Earth.”

Stir it up with these next-level woks 2019-09-12 15:04:18Woks to make tasty stir-fry meals in your home kitchen.

Why some experts are trying to redefine suicide 2019-09-12 14:10:45There's no established method of knowing patients might self harm. Some researchers are trying to change this.

Calming accessories that help you sleep better 2019-09-12 12:40:47Improve your sleep and feel more rested with these innovative pieces of tech.

How to check if you've been hacked 2019-09-12 12:30:00If you think someone might have gained access to one of your online accounts, the earlier you take action, the better. Every minute you don’t, i

How to safely find out what's on a 2019-09-12 09:30:00Companies like to hand out USB drives like candy. At media functions, for example, these flash storage devices contain product photos, press informati

Male honeybees might blind queens to keep them 2019-09-12 08:48:13A newly-discovered toxin that blinds queens marks a step up in the sexual arms race of bee mating season.

Comets could all come from the same place 2019-09-11 18:06:56Uncovering the objects’ possible shared origin required melding decades of comet observations with new theories of solar system formation.

Four master tools for your bug repellent arsenal 2019-09-11 16:52:00Effective products to keep the bugs away, whether you're indoors or out playing.

The best wildlife photos of the year remind 2019-09-11 16:40:00A selection of highly commended images from the London Natural History Museum's annual exhibit.

Ars Technica » Gear

Preventing GPS spoofing is hard—but you can at 2019-09-12 06:45:45This GPS spoofing defense software looks promising, but it's short on detail.

AMD improves real-world boost and idle on Ryzen 2019-09-11 15:24:37The extra 25-50MHz boost speed probably means less than the lower idle, really.

iFixit tears down the newest Fairphone—how repairable is 2019-09-11 13:14:04The Fairphone 3 gets the full iFixit treatment and a 10/10 score.

Apple iPad 2019 hands-on: A 6th-generation iPad in 2019-09-10 21:23:02An older CPU meets a (sort of) newer design, but it's still a notable improvement.

Apple reveals iOS 13 and iPadOS release dates—but 2019-09-10 21:10:40watchOS and tvOS updates are also just around the corner.

Apple Watch series 5 hands-on: Software is king 2019-09-10 20:10:10New materials and sensors headline this annual update. And we have photos.

iPhone 11, iPhone 11 Pro, and iPhone 11 2019-09-10 18:46:01Apple's new iPhones bring greatly improved cameras and new colors, among other things.

Apple continues health push with three new medical 2019-09-10 16:49:16Research partners are enthusiastic, but they face hurdles with this kind of data.

Apple’s iPhone 11 Pro is official, with three 2019-09-10 14:21:24$999 5.8-inch Pro and $1,099 6.5-inch Pro Max arrive on September 20.

Apple introduces the new $699 iPhone 11—and it’s 2019-09-10 13:54:19The follow-up to the iPhone XR adds a new dual-camera system.

The $399 Apple Watch Series 5 is coming 2019-09-10 13:46:28New materials and customization options abound, plus an always-on display.

Surprise! Apple unveils its 7th-gen iPad with a 2019-09-10 13:46:19It will still cost $329 and ship on September 30.

TechCrunch » Gadgets

Nintendo shows off exercise-powered RPG for Switch, Ring 2019-09-12 13:28:43Nintendo has been at the crossroads of video games and fitness since the famous Power Pad for the NES, and the Switch is the latest to receive a game

ckbk pulls a ‘Spotify for recipes’ out of 2019-09-12 07:48:12Cooking may be under sustained attack by a wave of on-demand food delivery startups, with names that can double as gluttonous calls to action (oh hey

Why does the new iPhone 11 Pro have 2019-09-10 14:51:29On the back of the iPhone 11 Pro can be found three cameras. Why? Because the more light you collect, the better your picture can be. And we pretty mu

iOS 13 will be available on September 19 2019-09-10 14:48:12Apple announced in a press release that iOS 13 will be available on September 19. Even if you don’t plan to buy a new iPhone, you’ll be ab

Apple introduces the A13 Bionic for the iPhone 2019-09-10 14:12:13Apple is announcing new iPhone models today. The iPhone 11 uses an Apple A13 Bionic system-on-a-chip. It is a nice improvement over the A12 Bionic in

Apple introduces the Apple Watch Series 5 with 2019-09-10 13:45:55Apple today introduced some new Apple Watch models at a press conference. The Apple Watch Series 5 has an always-on display. It seems to look just lik

Watch Apple unveil the new iPhone live right 2019-09-10 13:01:57Apple is set to announce new iPhone models today. The company is holding a keynote on its campus at 10 AM PT (1 PM in New York, 6 PM in London, 7 PM i

The Polaroid Lab uses the light from your 2019-09-10 08:00:29When all of us are carrying phones that can snap a thousand photos a minute and are connected to cloud systems that can store millions, there’s

SimShine raises $8 million for home security cameras 2019-09-10 03:19:29SimShine, a computer vision startup based in Shenzhen, has raised $8 million in pre-Series A funding for SimCam, its line of home security cameras tha

FDA says Juul ‘ignored the law’ and warns 2019-09-09 17:53:38The Food and Drug Administration has put vaping giant Juul on notice with a pair of letters calling the company out for misleading statements about it

Apple could add in-screen fingerprint reader to 2020 2019-09-05 10:55:11According to a new report from Bloomberg, Apple has been working on in-screen fingerprint readers. But that feature won’t be ready for the new i

Sonos took the mic out of its smart 2019-09-05 09:00:42Sonos has a new entry-level connected speaker that will give you all of its multi-room, high-quality sound — without the onboard microphones and


Disclaimer and Notice:WorldProNews.com is not responsible of these news or any information published on this website.