NVIDIA Introduces AI Interactive Graphics Research: 3D from Real-World Video

Subject: General Tech | December 3, 2018 - 08:00 AM |
Tagged: ue4, nvidia, NeurIPS, deep learning, ai, 3D rendering

NVIDIA has introduced new research at the NeurIPS AI conference in Montreal that allows rendering of 3D environments from models trained on real-world videos. It's a complex topic that does have potential beyond scientific research with possible application for game developers, though this is not to the "product" stage just yet. A video accompanying the press release today shows how the researchers have implemented this technology so far:

View Full Size

"Company researchers used a neural network to apply visual elements from existing videos to new 3D environments. Currently, every object in a virtual world needs to be modeled. The NVIDIA research uses models trained from video to render buildings, trees, vehicles and objects."

The AI-generated city of a simple driving game demo shown at the NeurIPS AI conference gives us an early look at the sort of 3D environment that can be rendered by the neural network, as "the generative neural network learned to model the appearance of the world, including lighting, materials and their dynamics" from video footage, and this was rendered as the game environment using Unreal Engine 4.

"The technology offers the potential to quickly create virtual worlds for gaming, automotive, architecture, robotics or virtual reality. The network can, for example, generate interactive scenes based on real-world locations or show consumers dancing like their favorite pop stars."

Beyond video-to-video this research can also be applied to still images, with models providing the basis for what is eventually rendered movement (the video embedded above includes a demonstration of this aspect of the research - and yes, dancing is involved). And while all of this might be a year or two away from appearing in a new game release, but the possibilities are fascinating to contemplate, to say the least.

Source: NVIDIA

December 3, 2018 | 11:15 AM - Posted by remc86007

Cool tech. I'm sure it's a relatively easy fix, but it's strange how bad the color mapping is. I would think that would be the easy part.

December 3, 2018 | 01:39 PM - Posted by Power (not verified)

Judging by the output quality Nvidia gave up on both decent framerate and raytracing.

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Lines and paragraphs break automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <blockquote><p><br>
  • Web page addresses and e-mail addresses turn into links automatically.

More information about formatting options

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.