Windows ML adds experimental GGUF support. Should your team switch its local inference stack?
GGUF support in the Windows ML Runtime API remains experimental. Check its supported hardware and NPU limitation, use Microsoft’s C++ sample for an isolated trial, and distinguish loading a model from running inference successfully.
What to consider
Do not move an entire working local inference application over yet. Microsoft now documents GGUF support in the Windows ML Runtime API, but the API is experimental, is not supported for production, and should not be used in apps published to the Microsoft Store. GGUF runs through llama.cpp, with CPU and NVIDIA CUDA validated for this release. Windows ML’s NPU options do not mean GGUF supports NPUs. Start by checking your model in an isolated sample.
Check GGUF and ONNX hardware support separately
Windows ML selects a backend by the model file extension: ONNX and ORT files use ONNX Runtime, while GGUF files use an app-local llama.cpp backend. The table below follows Microsoft’s Runtime concepts guide. Hardware options cannot simply be transferred between backends. An NPU-equipped device does not by itself establish NPU acceleration for a GGUF language model.
For a GPU trial, check that the CUDA module matches the llama.cpp revision used by the package. Microsoft documents that an explicit GGUF GPU target causes Build to return an unsupported error if the CUDA module is absent; insufficient memory can also cause a failure. Record these cases separately so results that did not use the intended GPU are not mixed into a performance comparison.
| Model file | Execution backend | Scope listed in the current documentation |
|---|---|---|
| .onnx / .ort | ONNX Runtime | CPU, GPU, or NPU through a compatible execution provider |
| .gguf | llama.cpp | CPU baseline; NVIDIA GPU requires the additional CUDA module |
| GGUF with an NPU target | llama.cpp | Unsupported; do not reuse ONNX NPU settings |
Start with a separate C++ sample project
The C++ setup guide requires Visual Studio 2022 or later with the Desktop development with C++ workload. Create a native console project, configure NuGet as described, and install Microsoft.Windows.AI.MachineLearning version 2.7.2021-experimental. The documentation samples also use Microsoft.Windows.CppWinRT and C++20. Those are sample requirements, not a requirement that every Runtime application use C++20.
Run the introductory sample that only creates IWinMLRuntime before loading your model, so build errors and DLL dependencies are resolved first. Pin the NuGet version in the test project and record the model file checksum. Rerun the test separately after changing the package or the model to identify what caused a behavior change. Check the linked deployment requirements for Windows compatibility instead of guessing minimum requirements from model size.
Separate model loading from a complete generation run
A successful LoadModelFromFile call only confirms that the file is recognized and readable. Hardware compatibility is established when Build succeeds. Follow the GGUF example in the concepts guide, beginning with a CPU target, adding a model stage, setting the context length, and building the pipeline. Completing this baseline makes it easier to distinguish a damaged file from missing components or an incompatible device.
For a text application, continue with the documented Text Generation API, which handles tokenization and token-by-token generation and provides streaming and cancellation. Inspect the completed result’s finish reason, such as a generation limit or cancellation. When comparing two independent prompts, also check that state is reset so the first prompt does not contaminate the next. We did not run these Windows samples; the steps are based on Microsoft’s documentation.
Base a migration decision on usable results from the same inputs
Choose inputs that the existing application already handles correctly. Keep the model version and generation settings the same in the old stack and the test project. Record cold-start and steady-state times separately, then check whether the outputs still serve the original task. Time to the first text does not replace the wait for the complete answer. If an answer is truncated, inspect its finish reason before calling the new stack faster.
As of October 20, 2026, the Runtime overview lists C++ and Python support and says C# is not yet supported. Applications with existing cross-platform ONNX Runtime session code can continue using the ONNX Runtime APIs included in Windows ML; format-support news alone is not a reason to rewrite them. We checked the documented API scope and examples, but did not measure throughput, memory use, or device compatibility.
Checked sources
The following sources support the project descriptions and reported facts. The analysis is by the site editors; claims from these sources are not presented as results from our own tests.