3D computational camera technology could bring stereoscopic magic to mobile photography.
Smartphone and tablet cameras let us snap and share photos from almost anywhere, but what if our experiences could be captured with immersive, stereoscopic depth so they looked more like real life?
What if each photo you captured could be refocused, give you measurements, animate, and discover things that you didn’t know existed when you snapped the scene?
New Intel RealSense snapshot technology, a combination of cameras arrays and image processing software, is making it possible for mobile devices to shoot this kind of depth photography.
“We live in a 3D world, but our photos capture it as if it were flat,” said Manuel Monroy, a cloud and ecosystem architect for Intel RealSense snapshot.
“We need to allow photos to have depth.”
Monroy sees this as the next big step for mobile photography. It’s like having a mobile device with more eyes, each capturing the same scene from slightly different angles then layering all of that image data into one spatially enhanced photo, he said.
Listening to stereophonic sound makes us feel submerged or surrounded; seeing a stereoscopic photo can make us feel like we’re witnessing real life. “The effect of what we see has a lot to do with math — how distance and angles hit the back of the eye — but a lot has to do with perception — how our brain rapidly fills in the gaps by interpreting light and textures,” said Dr. Gloria Surh, optometrist at Ideal Eyes in Burlingame, California.
When we look at a standard photograph of train tracks laid straight ahead into the horizon, our mind can understand there’s distance, but with a depth-mapped image we can more clearly measure the distance.
By adding depth map data to high-quality images, RealSense snapshot technology allows the user to manipulate their photos in new ways. RealSense snapshot technology is available in the Dell Venue 8 7000, which, at 6 mm thick, is the world’s thinnest tablet.
The first working prototype demoed last year at the Intel Developer Forum, and, in January, the sleek 8.4-inch tablet won the CES Innovation Award.
During his keynote at the International Consumer Electronics show earlier this year, Intel CEO Brian Krzanich showed how using RealSense snapshot in the Venue 8 7000 allowed him to bring different layers of a photo into focus and change lighting and color on specific objects with the touch of a finger.
There are different ways to capture stereoscopic images. The first uses two or more strategically placed cameras that each snaps a shot then overlays them into one photo. The second uses a single camera that snaps the same object or scene from slightly different but overlapping angles.
RealSense snapshot technology in the Dell Venue 8 7000 uses three cameras — an 8 MP camera at the top and two 780p cameras fitted below, spread apart almost at the distance of the human eye. The first prototype required nine cameras and components, which cost around $50. That’s expensive, and it used too much processing power to be practical.
The RealSense snapshot development team came up with the current three-camera model, which simplifies the design and significantly lowers manufacturing costs.
To write the software, the team built a test board using six GoPro cameras with a variety of layouts. This helped find the best way to configure the final three-camera set. The three cameras shoot simultaneously, then combine the overlaid information to create an image file that has many more capabilities than a standard JPEG.
Once an image is captured, it is processed into a high-quality, depth-mapped file.
“It creates a fat JPEG,” said Erhhung Yuan, system architect and lead developer for Intel RealSense snapshot. “It’s essentially a JPEG with more metadata fields, including the computed depth map.”
These depth maps use gradients of grey to depict the distance from each pixel in the photo. This allow apps to generate and manipulate individual objects — depth layers — within the captured scene.
About 30 percent larger than a file capture only by the tablet’s 8 Megapixel camera, the fat JPEGs averages between 2 to 2.5 MB file.
The position, resolution and synchronization of the cameras are essential. “The cameras need to be properly calibrated,” said Monroy.
“Once captured, the image moves through an algorithm pipeline consisting of a number of steps, including dynamic calibration, rectification, and disparity calculation,” he said.
“The end result is a fat JPEG which is then used by app developers, to power such use cases as depth filters, refocus and parallax.”
From there it goes to apps — such as measurement, refocus, background highlighting, filtering, motion effects, saving and sharing.
What started out as low-cost, computational photography to create immersive visual experiences for Android tablets is now being updated to work on forthcoming Windows 10 tablets, according to Monroy.
“Inside there’s complicated depth mapping and algorithm processing, but what you see is really magic,” said Monroy. The Dell Venue 8 7000 is now available on Dell.com and at Best Buy for $399. ZDNet called it “insanely thin” and Gizmodo dubbed it, “The most interesting Android tablet … in ages.”
While most early reviews saw RealSense snapshot imaging technology as something new and interesting, some pointed out that the new cameras and apps require some improvement.
“It’s the first time out in the wild, so we have teams working on improvements and new apps,” said Monroy.
He also said that his team is deploying a cloud platform to enable immersive visual experiences that require higher compute power than what’s available on most mobile devices. He sees it as a backbone supporting multi-user data driven apps.
“Imaging of what we can do with 100 depth photos of the statue of David in Florence, or with 1,000 taken at a rock concert.”
Editor’s note: In his second series of TV ads for Intel, actor Jim Parsons uses a tablet with Intel RealSense snapshot technology to capture a photo and then measure the distance between two objects. Learn more about this and other Intel innovations that are bringing new experiences to personal computing devices.