GDC 2017: The Khronos Group Releases Vulkan

Linked Multi-GPU Arrives... for Developers

The Khronos Group has released the Vulkan specification, which includes experimental (more on that in a couple of paragraphs) support for VR enhancements, sharing resources between processes, and linking similar GPUs. This spec was released alongside a LunarG SDK and NVIDIA drivers, which are intended for developers, not gamers, that fully implement these extensions.

I would expect that the most interesting feature is experimental support for linking similar GPUs together, similar to DirectX 12’s Explicit Linked Multiadapter, which Vulkan calls a “Device Group”. The idea is that the physical GPUs hidden behind this layer can do things like share resources, such as rendering a texture on one GPU and consuming it in another, without the host code being involved. I’m guessing that some studios, like maybe Oxide Games, will decide to not use this feature. While it’s not explicitly stated, I cannot see how this (or DirectX 12’s Explicit Linked mode) would be compatible in cross-vendor modes. Unless I’m mistaken, that would require AMD, NVIDIA, and/or Intel restructuring their drivers to inter-operate at this level. Still, the assumptions that could be made with grouped devices are apparently popular with enough developers for both the Khronos Group and Microsoft to bother.

View Full Size

A slide from Microsoft's DirectX 12 reveal, long ago.

As for the “experimental” comment that I made in the introduction... I was expecting to see this news around SIGGRAPH, which occurs in late-July / early-August, alongside a minor version bump (to Vulkan 1.1).

I might still be right, though.

The major new features of Vulkan are implemented as a new classification of extensions: KHX. In the past, vendors, like NVIDIA and AMD, would add new features as vendor-prefixed extensions. Games could query the graphics driver for these abilities, and enable them if available. If these features became popular enough for multiple vendors to have their own implementation of it, a committee would consider an EXT extension. This would behave the same across all implementations (give or take) but not be officially adopted by the Khronos Group. If they did take it under their wing, it would be given a KHR extension (or added as a required feature).

The Khronos Group has added a new layer: KHX. This level of extension sits below KHR, and is not intended for production code. You might see where this is headed. The VR multiview, multi-GPU, and cross-process extensions are not supposed to be used in released video games until they leave KHX status. Unlike a vendor extension, the Khronos Group wants old KHX standards to drop out of existence at some point after they graduate to full KHR status. It’s not something that NVIDIA owns and will keep it around for 20 years after its usable lifespan just so old games can behave expectedly.

View Full Size

How long will that take? No idea. I’ve already mentioned my logical but uneducated guess a few paragraphs ago, but I’m not going to repeat it; I have literally zero facts to base it on, and I don’t want our readers to think that I do. I don’t. It’s just based on what the Khronos Group typically announces at certain trade shows, and the length of time since their first announcement.

The benefit that KHX does bring us is that, whenever these features make it to public release, developers will have already been using it... internally... since around now. When it hits KHR, it’s done, and anyone can theoretically be ready for it when that time comes.

March 1, 2017 | 08:04 AM - Posted by derz

Might make multiGPU more appealing. Will it affect transcode time..

March 1, 2017 | 08:55 AM - Posted by Anonymous (not verified)

So now that Khronos has its Explicit Multi-Adaptor ducks in order where are the Vulkan/windows 7 and Vulkan/windows 8.1 gaming benchmarks. Better yet it's time for more Vulkan Linux focus as Valve's VR games/gaming engines use by Valve for its Steam OS/Linux gaming will be built targeting the Vulkan graphics API.

"I’m guessing that some studios, like maybe Oxide Games, will decide to not use this feature. While it’s not explicitly stated, I cannot see how this (or DirectX 12’s Explicit Linked mode) would be compatible in cross-vendor modes."

No Nvidia and AMD only have to provide their respective Vulkan and DX12 light weight and to the metal drivers with those drivers abstracting the GPU's hardware to the OS/New Graphics APIs. So It's not up to the GPU’s drivers it's up to the DX12/Vulkan support that the GPU makers provide with their GPU hardware SKUs. Any vendor specific DX12/Vulkan extensions can be added to the Vulkan/DX12 APIs as the Graphics APIs are extensible and can have any vendor specific GPU hardware features added by the Vendor to the DX12/Vulkan APIs. So it’s up to the games/gaming engine makers to provide the cross vendor support in their games via DX12/Vulkan extended API calls for any difference in the respective GPU maker’s products for the games that can call on any vendor specific DX12/Vulkan extensions registered with the new graphics APIs.

So the games makers have the option of creating games that only call on only the common DX12/Vulkan graphics API features that are supported by AMD's and Nvidia’s/others GPU hardware for at least the ability to use any maker's GPUs plugged into a PC or they can also support in their games/gaming engines any vendor specific features to allow their games to even make use of some vendor specific GPU feature sets should the games maker want to.

Also with Khronos any vendor specific Vulkan extended feature added that gets adopted by Khronos as part of the regular Vulkan standard will by being adopted by Khronos be part of the Vulkan standard and can be, at some point in time, used on any other vendors product. Some extended vendor features not implemented in all vendor hardware can also be implemented in emulation/software for any legacy/other hardware that does not have the adopted extension implemented in any vendor’s specific GPU hardware, at the usual cost of emulation to performance.

Look for the gaming engine makers to make support for any vendor specific GPU hardware extended support via Vulkan/DX12 extended vendor specific Vulkan/DX12 extensions an easy process in the gaming engines’ SDKs. The games industry, and the development community, has more resources to develop the software IP for the better multi-GPU load balancing than the GPU makers alone.

March 1, 2017 | 04:34 PM - Posted by Scott Michaud

You're a bit mistaken.

Linked explicit multi-GPU isn't designed to be out of the hands of the graphics vendors. (It's referred to as "Explicit Homogeneous mGPU" in this MSDN post.) The point is that the GPUs need to be basically identical, which allows the game developer to make some assumptions about them. Performance will be roughly identical between physical sub-devices, the memory management will be the same, etc.

Explicit mGPU comes in two flavors: homogeneous, and heterogeneous.

Homogeneous mGPU refers to a hardware configuration in which you have multiple GPUs that are identical (and linked). Currently, this is what most people think of when ‘MultiGPU’ is mentioned. Right now, this is effectively direct DX12 control over Crossfire/SLI systems. This type of mGPU is also the main focus of this post.

Linked explicit multi-GPU is designed to be up to the driver. Using the assumption that GPUs are identical, it allows the driver to do some of the extra work that a game developer would need to do, like copying a render target to the host before ingesting it in another GPU's render pipeline (rather than having the developer do all these instructions by hand). This feature, which adds driver-level control for the purpose of simplifying game development, would be a huge burden for vendors to agree upon for a product-by-product basis.

On the other hand, unlinked multi-GPU can be cross-vendor. The game developer can do those driver-level tasks themselves. Those titles can easily be cross-vendor if the game developer wants to bother with that hassle (and Oxide, for one, already did).

March 1, 2017 | 07:13 PM - Posted by Anonymous (not verified)

Yes M$ breaks it down further but I'm talking about the form that uses the simplified drivers and that does not use any legacy CF/SLI driver bloat and that allows for heterogeneous GPU usage and that is called/managed from the DX12 or Vulkan APIs.

So take the common Vulkan API and its to the metal driver access, or DX12 and its to the metal drivers access(if using DX12), feature set for example that both Nvidia's and AMD's GPUs support and use that common Vulkan, or common DX12(For dx12 based games), calls and getting games that at the minimum support that can be used with AMD's and Nvidia's GPUs plugged into a PC at the same time.

This is games/gaming engine managed[Via the new graphics API calls] minimum cross vendor multi-GPU API support using the common/standard Vulkan, or common/standard DX12 for DX12 games, based calls where the drivers just provide to the metal access and nothing more.

"On the other hand, unlinked multi-GPU can be cross-vendor. The game developer can do those driver-level tasks themselves. Those titles can easily be cross-vendor if the game developer wants to bother with that hassle (and Oxide, for one, already did)."

Yes this is correct for DX12, and Vulkan is similar with its unlinked multi-GPU type of support(Vulkan has no "Explicit Homogeneous" support as far as I know). But the game's developer is most likely using a gaming engine SDK for their respective game/s and it’s the gaming engine developer/s that is/are the real systems software engineers and not the games developers. So all that too hard for the game developer work is the job of the real systems software engineers at the gaming engine company and the gaming engine SDK is there to assist the games developers to take advantage of the new graphics API's feature sets.

So I’m talking about doing away with CF/SLI and the GPU maker’s poor history of developing their proprietary multi-GPU load balancing “solutions” and going full on with using that graphics API managed way of multi-GPU load balancing with the very simple GPU drivers that only make the very primitive/to the metal GPU hardware feature sets available via the new graphics APIs and simplified drivers. And it’s not too difficult to do under Vulkan for the various GPU makers to also include in the Vulkan API their vendor specific GPU hardware extensions to allow those non common GPU hardware features sets to be accessible on their specific brands of GPUs even if that hardware specific feature is not available on any other vendor’s GPUs that may also be plugged into the PC at the same time. So as long as there is enough common Vulkan features sets support by any vendor’s GPUs, heterogeneous GPU usage should be easy and even the support the non common features for game usage also can be managed.

The OS/APIs and Graphics APIs should have been what managed any homogenous or heterogeneous GPU processor load balancing. I do not understand why the computing industry ever allowed the GPU hardware makers to ever be that much in control of any traditionally OS/APIs hardware management responsibility. It’s been the responsibility of the OS/APIs to manage CPUs under the OS/APIs from day one, so what makes the GPU so special. The real software brain power does not exist solely inside the GPU makers’ facilities and the GPU makers only responsibility should have been relegated to only providing the minimal driver functionality to allow for the GPU's hardware to be accessed at the lowest level possible.

This initial mistake on the part of the computing industry is what has lead to all of the Multi-GPU usage problems that have existed and its because the GPU makers never had the resources and brain power on their own to tackle a problem that should have been the responsibility of the OS/API makers and the computing/software industry as a whole.

March 1, 2017 | 09:15 PM - Posted by Scott Michaud

The device pool added with is Vulkan's "Explicit Homogeneous" support.

"Explicit Heterogeneous" should just be managing multiple vkDevices directly, which I believe was feature-complete in Vulkan 1.0, although very few studios would want to use it. I haven't had time to actually implement something in Vulkan, though. At least, not yet.

March 1, 2017 | 09:53 PM - Posted by Anonymous (not verified)

They need to get the full level of support hopefully by Vulkan/Next's release. And these "very few studios" may just use it if the support and hand holding is there from the gaming engine SDKs gaming engine software engineers. Damn, some of those gaming companies have budgets bigger than Hollywood on some gaming titles and they spend billions leasing/licensing gaming engines/gaming engine SDKs and consultant fees so they can afford the systems software engineers. They also need to tell any "Programmers" to get up to speed or find a new line of work if they can not make the adjustment.

Most gaming script writers are not programmers in the well trained sense of the programmer title and do not have to be if the actual staff programmers and gaming engine SDKs are there to help. I have never worked at any programming job that did not have top level software engineers to help the bog standard programmers with getting the hard parts done and even at college computing facilities they employ high level software engineers to help the lower level programmers, teachers/researches, and students with the hard parts of interfacing with the changing OS/API ecosystems.

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Lines and paragraphs break automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <blockquote><p><br>
  • Web page addresses and e-mail addresses turn into links automatically.

More information about formatting options

By submitting this form, you accept the Mollom privacy policy.