Review Index:
Feedback

Visual Computing Still Decades from Computational Apex

Author: Ryan Shrout
Manufacturer: Epic Games

Calculus, bitches!

As a CS major, calculus is all too familiar to me.  Hopefully, not all of you are so fortunate...

View Full Size

The rest of the graphical computation story is brought to life by calculus, the math that allows us to approximate everything from areas to light rays.  A first order approximation is the most basic form that is the easiest to compute the least accurate in its result.  For each successive iteration of the approximation, quality improves but the mathematical load placed on the processor increases dramatically. 

View Full Size

Doom - 1993

Doom was the first game to really attempt to simulate reality and used the most basic of first order approximations to generate the graphics with a single bounce of light from the texture or sprite to the 3D camera being manipulated.

View Full Size

Unreal - 1998

Sweeney's own Unreal was an early adopter of the second-order approximation, allowing light rays to bounce off of two things before converging on the virtual camera.  According to Sweeney 99% of today's games on the console and PC are still using engines based on this type of simulation method.

View Full Size

Samaritan Demo - 2011

Third-order approximations are much more complex but allow light to bounce through many items and as a result you are seeing more realistic reflections, skin coloring and highlights.  The Samaritan demo from Epic Games shown last year is the company's investment in this type of computing and requires several of today's fastest GPUs to be rendered with even minimal interaction and lower than desired frame rates. 

View Full Size

As an interesting demonstration Sweeney gives us the approximate computing power of these titles.  The original Doom required 10 MFLOPS (millions of floating point operations) to run at 320x200 in 1993 while Unreal required 1 GFLOPS (billions) to run at 1024x768 in 1998; both at 30 Hz. 

The Samaritan Demo that Epic unveiled in 2011 was running at 1920x1080, still at only 30 frames per second, with more than 40,000 operations per pixel.  The total required GPU computing power was 2.5 TFLOPS (take a guess?) and it was running on three GTX 580 cards if my memory serves.  Of note is that the latest Radeon HD 7970 GPU is capable of a theoretical computing power of 3.79 TFLOPS though how much of that an engine like Samaritan could utilize has yet to be determined fully.

Also interesting to see for those "consoles are good enough" users, take a look at the compute power of the Xbox 360 - only 250 GFLOPS, a tenth of the required power to run Samaritan. 

View Full Size

And while the visual quality we are seeing in games like Battlefield 3 and the Samaritan demo are impressive, Sweeney was quick to point out that to truly get to the level of movie-quality lighting we will need to progress even beyond the third-order approximations we are seeing as the limit today.  It is likely we will need to see another order of magnitude increase in computing power in order to reach that 4th level - PetaFLOPS are in our future.

View Full Size

Because we know (well, someone knows) and completely understands how lighting, shadows, skin, smoke and other complex visual challenges worked in the scientific sense, they can be approximated accurately.  And with enough orders of approximation, as we have seen in the slides above, we can get very close to perfection.  Sweeney estimates that we will need around 5000 TFLOPS of performance, or 5 PFLOPS, to reach that goal.  That is a factor of 2000x today's best GPU hardware and leaves a lot of room for development from NVIDIA, AMD and even Intel until we reach it.

Even Sweeney wouldn't put a time frame on the goal of hitting that 5000 TFLOPS mark but depending on the advancement of both process technology and the drive of designers at these technology firms, it could be as soon as 5 years or as long as 10.  If we go over that, I feel we have wasted the potential of the hardware available to us. 

View Full Size

What may hinder the game development community is not the graphics and lighting challenges but rather the need to approximate things that we just don't have algorithms for yet.  Show me a textbook that details human thought (for AI), movement (for animation) or personality (for believable characters) in a mathematical formula and I'll show you Josh's natural hair growing back in.  Sweeney states in his talk that we simply lack the fundamental understanding of these things and even with an infinite amount of computing horsepower our developers would be lost. 

 

There is quite a bit more that you can learn from listening to the remainder of Tim Sweeney's talk (found over at Gamespot) including his thoughts on Moore's Law, stacked transistors and the future of computer interaction.  For me though, the primary point to be made was that the idea of the hardware races being over because of a lack of NEED for additional computing horsepower is simply false.  The hardware races might slow down due to a lack of competition or without the profitable discrete add-in card market to drive it, but those are tangential reasons at best. 

With one of our current top gaming software developers telling us we need something on the edge of 2000x performance increases compared to current GPU hardware to reach our peak of creativity, I have much more faith that hardware companies like AMD, Intel, NVIDIA and others we might not have heard of yet, are busy working on the future.  

March 7, 2012 | 05:44 PM - Posted by Nilbog

This article just made my day. Thanks Ryan! It makes me happy to see developers that care about innovating for the entire industry, not just for themselves. It does seem strange that the gaming industry doesn't have some kind of universal algorithm for AI, ect.
I also can't wait to see this engine! That screenshot looks gorgeous.

March 8, 2012 | 12:04 AM - Posted by Devil 2 U

Agreed, great article. The slide have all the details and Ryan effectively breaks it down and explains it in his analysis. I am going to share this on Facebook.

March 8, 2012 | 04:28 AM - Posted by Humanitarian

Wow, this made quite an interesting read. Nice to see things in perspective and ponder what future generations hold.

More of this! Moooaaaar.

March 8, 2012 | 06:43 AM - Posted by KungFu_Toe (not verified)

So, if I'm getting this right, the video card is basically going down the same path as the sound card did eight to ten years ago.

March 8, 2012 | 07:57 AM - Posted by Ryan Shrout

Not sure I understand the analogy...?

March 8, 2012 | 01:04 PM - Posted by Anonymous (not verified)

16 years from now any IGP will have the power to drive this specs... So no need for video cards.

March 9, 2012 | 10:16 AM - Posted by Josh Walrath

Not necessarily. We don't know what other visual technologies will come out in the next 16 years that will require much more horsepower from graphics cards (or whatever they may be at that time). Think of the past 12 years what all we have seen. Anti-aliasing, Anisotropic filtering, higher resolutions, multi-monitor, stereo 3D, etc. etc. Think of a high end monitor in 1996 that was 17" CRT/Trinitron and cost upwards of $1000 that was limited to 1280x1024 @ 60Hz. Now even the most basic monitor can do 1080P for under $200.

I'm pretty sure we are going to see some neat stuff by then that will continue to push the computational needs of cutting edge games and applications.

March 8, 2012 | 07:57 AM - Posted by Ryan Shrout

Thanks for the comments everyone!

March 8, 2012 | 12:27 PM - Posted by Nardella (not verified)

Do these statements consider that particle based effects are becoming more prevelant? We also need to consider that emulating/simulating reality is not the only goal, we are also interested in some unreal things that may require more graphics horse power.

Also, video games generally have a limited number of characters and creatures running around doing stuff. I have not seen any games or demos with hundereds of characters running around doing stuff in the same scene. That and most games do not have insects and birds flying around in realistic numbers.

It is one thing to render a photorealistic scene, but it is another to render a photorealistic scene with extremly high numbers of extremly complex elements.

And what about physics simulations. We may realize improvements well beyond a 2000x increase in GPU power.

Also, what if someone wants to make a game like minecraft with cubes that are 1x1x1cm? How much power would that take? Add in complex physics simlutations and photorealistic graphics.

Please note that I am not an expert. I would like to hear responses to this comments.

March 8, 2012 | 03:52 PM - Posted by Rob R. (not verified)

Particle effects will increase with the increase in computing power, assuming nothing else (like graphics, more creatures, physics, etc.) took the extra bandwidth instead.

The reason you don't see lots of people in a scene and the scene being photorealistic is they had to one or the other, because otherwise 95%(made up stat, but A LOT of people) wouldn't be able to run it because they don't have the crazy extreme hardware(expensive multi-GPUs, expensive 6-Core+ CPUs - like the i7 used in the tests)

And I guess a game where you have lots of people, but not the best graphics would be almost any MMORPG, and any game you have great graphics and few people in a scene is almost any FPS.

The power needed for your new version of Minecraft is probably A LOT, and you may be better with a complete particle system since the "cubes" are so small, just a guess though I don't know how much horsepower you need for Minecraft in the first place because I have never played it. Although I did look up that on the PC it is using Java, so that would need to change for your situation because it probably is no where near fast enough

March 8, 2012 | 10:00 PM - Posted by Anonymous (not verified)

Particle simulations will be trivial before graphics are trivial. I don't know of any large simulations even in feature films which take longer than an 8k render. Many simulations are still done on single beefy machines. And GPUs are getting better and better at crunching fluid sims which are the most intensive particle simulations to date. Newtonian hard surfaces and even cloth simulations are already nearly real-time at respectable detail levels on a GPU.

As to hundreds of characters, again full photo-realistic rigs are nearly real-time today. I've seen some GPU skin algorithms that are realtime.

As to there being hundreds of thousands of characters, that shouldn't be a problem. That will be a limitation of RAM and RAM we can assume will follow processor speed in parallel. By the time we get to 8k true life rendering we'll almost certainly be raytracing and raytracing doesn't particularly care about triangle count so much as resolution as long as you can fit it into memory. With fast disk arrays and smart caching you can load and unload everything not in view. And LOD systems aren't going anywhere anytime soon.

March 8, 2012 | 12:55 PM - Posted by inputexpert (not verified)

For the pc gaming machine, the gamer is limited by the old standard stand alone keyboard and mouse, old technology.

With an advanced/next generation integrated optical pointing keyboard new game interfaces can be developed giving the player a new user experience.

All fingers on the home row are usable instead on the left hand only.

A major Bellevue Washington gaming company will be exploring the possibilities of using advanced/next generation integrated optical pointing keyboard.

Very cool stuff.

March 8, 2012 | 01:42 PM - Posted by Anonymous (not verified)

Mr. Shrout,

You have made a factual mistake. At the bottom of page 1, you state that the best GPUs are capable of drawing 2.8 _million_ triangles per second, yet this is obviously wrong as the original crysis often had scenes displaying that many triangles (and the 8800 series gpus performed that about 30 times per second!). Secondly, if you refer to your link, it shows ~2800 _million_ (2.8 billion) triangles per second! Clearly we have come a long way! Although to do this at 72hz is a whole other matter ;). Keep in mind these are absolute peak numbers.

A better means of assessing how far we are to real time, truly photo real graphics, would be to examine the processing power of current rendering farms and frame render times for film CGI. I know there are such farms that are hundreds of times more powerful than the most powerful gpu, yet they can take _hours_ to render a single frame, much less 72 times per second!!

March 8, 2012 | 02:52 PM - Posted by Ryan Shrout

Thanks for the typo, it has been fixed!

March 8, 2012 | 03:56 PM - Posted by Anonymous (not verified)

That's a terrible methodology for determining how far we've come in real time... On one hand, CPU's and GPU's are built around a completely different architecture and compute problem sets extremely differently.

On the other, we don't utilize an ENTIRE farm on a single frame of a movie because the network overhead would be immense as there's literally several gigs of light/geometry/image data derived/cached/read from disk and used to draw the final 2d pixel representation. (This is assuming a 3d render and not a composite which is even worse since now we're reading 50 passes of 20-50 megabyte/image files that we're applying/caching complex 2d and 3d operations on but I digress.) Honestly we generally render 1 frame on one machine... With this nugget alone you are now comparing render times of a 16-24 (32 now with sandy bridge!) core xeon machine against the thousands of cuda cores in strung up Teslas which is a moot comparison anyway since their cores are not analogous.

There's a reason why my industry is slowly building more gpu-centric tool sets.

Also, most of our calculations are unoptimized because we're lazy and don't read/apply the latest siggraph whitepapers on computer graphics as quickly as we should.

March 8, 2012 | 06:45 PM - Posted by yammerpickle2 (not verified)

I think there is a lot more power needed! First off I want larger than 90 degree FOV. So let’s start off at least 24,000 x 4000 for three screens wide. Then to prevent micro stuttering you need to be able to consistently hit 90 FPS. Okay what about 3D. You need to either double the FPS or double the resolution of the monitors. Let’s talk about environments instead of a tree swaying in the breeze to be realistic each branch and unique leaf has to be fluttering in the wind with a forest of unique trees with all the unique ground clutter. Let’s not forget the artificial limit we have for on screen NPC’s and other characters. Let’s have an army of thousands of unique characters running, jumping, and fighting in that forest. Let’s start to introduce AI to each of those unique NPC’s Okay what about particle effects like smoke and dust swirling around that army in the forest and realistic damage for all of it. Now try and model all the noise and echo’s from all those interactions in 9.3 surround sound at 96khz or better. Let’s not forget to incorporate head, eye, and other motion capture controls without the lag and inaccuracy of things like Kinect. Okay NVidia, AMD, Intel, and Epic make it happen. Seriously I want a user experience that is like The Matrix where reality and gaming are indistinguishable from each other.

March 8, 2012 | 09:24 PM - Posted by Anonymous (not verified)

Let's face it. Gaming/Graphics computing won't reach its peak until we have an equivalent of the matrix. Complete, totally immersive virtual reality almost indistinguishable from reality. Then I think we could say we're done. :)

March 8, 2012 | 11:09 PM - Posted by Anonymous (not verified)

I'm not sure where these people get their information, but the 72FPS figure quoted is nonsense. The Primary Visual Cortex in the rear of the brain (casually known as V1) perceives visual information at 120Hz/FPS; anything slower than this will be perceived as 'not real', though one may not be aware of how or why. This kind of divide between knowing something instinctively (because your brain knows) and not being able to explain it (because your brain doesn't bother passing the information to the conscious mind) is well-established in all 5 visual cortices. Look up 'blindsight', for example, which is the ability for people who are blind because of a brain lesion (injury) to navigate around obstacles they can't consciously see: they 'know' where they are and what's around them, because their eyes and the non-lesioned parts of their brain still work, but they can't consciously see because the part of their brain that is lesioned is the one that would pass that information to the conscious mind. All this is information any neuroscience student could give you; it's a shame it's been misrepresented to so many here.

March 9, 2012 | 10:34 AM - Posted by Josh Walrath

I had a good laugh at the whole 72 fps thing. When I wrote my 30 vs 60 fps article back in 1999, I used 72 fps as a basis for "good enough for most everyone" because back then 72 Hz on a CRT monitor was considered mostly flicker free. My thinking was wrong there, but that rumor of 72 fps has unfortunately persisted.

Considering the somewhat dual digital/analog nature of our visual system, it is also not terribly accurate to say, "Our upper limit is 120 Hz" because that is a purely mechanical definition that does not really reflect the reality of light and sight. With some components of the eye able to detect even a single photon of light, it can quickly detect changes in light reaching it. So saying, "We can only perceive up to 120 Hz/FPS" is again false. The limit is higher, and some research has shown that even upwards of 240+ fps gives a greater feeling of reality than 120 fps.

The long and short of it is this; the real limitation is most likely the visual cortex, but through training and immersion in higher fps/quality visual simulations the cortex will learn to recognize the differences in speeds and input quality.

March 9, 2012 | 02:09 AM - Posted by SweenyIsWrong (not verified)

LOL no one should take anything sweeny says seriously. He thought 3D addin cards would be gone by 2005, it's 2012 and 3D add in hardware is still going strong.

All these discussions don't talk about memory bandwidth, and memory bandwidth is one of the key problems computing has to solve for anything sweeny says to have any kind of merit.

One only has to chart 3D hardware performance of the last 4 years, the gains been getting smaller and smaller because the memory bandwidth isn't there.

Sweeny is a hack who doesn't grasp memory bandwidth put's fundamental limits on how fast anything can go.

March 9, 2012 | 10:45 AM - Posted by Josh Walrath

Eh, yes and no to the bandwidth issue. The performance gains that we have seen through the years scale really nicely, and in fact have surpassed even the loosest interpretation of Moore's Law due to better advances in not just process technology, but the ability of the major graphics guys to radically change their architectures. If we compare/contrast GPU development over the past 10 years vs. that of CPUs, I think we can see that the change is much more radical on the GPU side rather than being stuck with the x86 CPU architecture.

Memory has also moved forward, but one area that you are discarding totally is that of the rise of large caches within the GPU. Yes, we still need as much bandwidth as possible, but now that we have started to really use mathematics based shading rather than pure texture based rendering, memory bandwidth is not as huge a concern as it once was. Not saying that it is trivial or unimportant, as people still expect high quality texturing in modern games. So, while GDDR-5 with a 384 bit bus will give a pretty impressive 250+ GB/sec of bandwidth, the much larger and more complex caches in the GPUs are doing a lot of the work... so it helps to minimize the impact that slower/higher latency off chip memory has on overall performance.

As time passes, we will see much larger and complex cache structures on GPUs, which will allow better performance even though external memory will always be behind GPU development.

March 9, 2012 | 12:17 PM - Posted by Anonymous (not verified)

is there a video of this talk anywhere?

March 9, 2012 | 02:39 PM - Posted by Ryan Shrout

Yes, there is a link to Gamespot in the story.

March 12, 2012 | 02:31 PM - Posted by Goofus Maximus (not verified)

I think we've already reached the point of dimishing returns with graphics. We just don't NEED photo-realism for games.

All those extra little touches to add realism will mostly just jack up the number of man/hours needed to develop a game, and we're already getting shorter gameplay experiences and longer development cycles, just so they can add extra bits of realism to the vast array of terrain and items that go into a game.

It hasn't escaped my notice that the most fun I've had with new games comes from titles that don't even come close to photo-realism, such as Recettear, Minecraft, and Torchlight.

March 31, 2012 | 10:56 AM - Posted by Anonymous (not verified)

"What Humans have created can be approximated very close to perfection; but what God has created cannot be approximated close to perfection even with infinite computing power".

April 7, 2012 | 08:15 AM - Posted by raghav

that simply shows how perfectly designed our biological parts are ... and now its understoodable,why it took millions of years of evolution to come to us...

thats still fast...

August 3, 2012 | 08:33 PM - Posted by figure competitor blog (not verified)

Well you have mention a really valuable points. Game development is a very interesting job, keep going dude. You done great job.

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Lines and paragraphs break automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <blockquote><p><br>
  • Web page addresses and e-mail addresses turn into links automatically.

More information about formatting options

By submitting this form, you accept the Mollom privacy policy.