Simplify Your SR-IOV Setup: A Guide to NixOS Modules and specialisations

Defining system images in a declarative manner unlocks huge efficiency gains in software engineering. Done right, the same definitions can serve product development, deployment, and shipping, as well as testing - both manually and using automation. Different components of the resulting system are encapsulated in clearly defined modules. Mixing, matching, and tweaking those modules creates your final system image. Or test run. Or work environment. How is all that so easy? Enter NixOS modules and specialisations - the ultimate productivity machine that powers all our engineering.

The power of declarative configuration in NixOS

Setting up a complex system can be a daunting task of completing many steps in a defined sequence. Take the SR-IOV graphics acceleration, for example. Preparing the host consists of building a custom kernel, changing boot parameters, setting special configuration options, and executing a list of commands everytime the system boots up. Wouldn’t it be nice to just define what the system should look like and then have a separate boot option that starts up exactly that system? And if you don’t want it anymore, all you have to do is reboot into the standard variant?

NixOS is a powerful and unique Linux distribution that focuses on declarative configuration and reproducibility. Unlike traditional systems, where you manually install and configure software, NixOS allows you to define your entire system setup in a single configuration file. This means you can specify what your system should look like, and NixOS will automatically build it for you, ensuring consistency across deployments. With features like rollback capabilities and isolated environments, NixOS is particularly suited for complex setups, making it an ideal choice for tasks like setting up SR-IOV graphics acceleration in a reliable and reversible way.

With the help of NixOS’ declarative module system, we are going to enable a separate boot option for the SR-IOV graphics acceleration feature together with our KVM Backend for VirtualBox.

Ease of use is key

The procedure to enable your host system to run VMs with graphics acceleration involves significant changes to the operating system. Not all of them are desirable for permanent use. Plus, some of the components might come with limitations. When the graphics performance is the focus, these limitations might be acceptable, but outside of that use case, you might want to stick to your default system configuration. It is therefore very useful to contain the changes to the “graphics world” and make them easily reversible.

At the same time, it should be as easy as possible for someone to try this out. We have noticed this in our internal Dogfood programme, where we encourage team members to test-drive our own software in daily use. It’s one thing for an engineer actively working on the KVM backend for VirtualBox to try it out on their own machine. It’s another story for colleagues working on other projects or don’t even have any programming background at all. That’s why we invested quite a bit in a (close to) 1-click solution to deploy our development versions.

What’s important is that trying out development versions should never impact the overall system. If something doesn’t work, simply going back to what worked should be a reboot at worst.

Two key mechanisms of NixOS enabled us to achieve the goals outlined above: NixOS modules and specialisations.

The Power of NixOS Modules: Simplifying Complex Configurations

NixOS modules are building blocks that allow you to configure and extend your system in a modular and reusable way. Each module encapsulates a specific functionality—like enabling a service or setting up a system component—behind a simple, declarative interface. Instead of manually adjusting settings across different files, you can activate features by importing the relevant modules and setting their options in your configuration file. This approach simplifies complex configurations and ensures that your system remains consistent, reproducible, and easy to manage, even as it grows in complexity.

Modules are a great tool to hide complex definitions behind a well-defined interface. Turning features on or off, as well as potentially tweaking some parameters, is all users need in the end. Turning on SR-IOV graphics support and our KVM backend for VirtualBox boils down to the following NixOS configuration snippet:

let
  sriov-modules = builtins.fetchGit {
    url = "https://github.com/cyberus-technology/nixos-sriov";
    ref = "main";
  };
in
{
  # ...
  imports = [
    "${sriov-modules}/sriov.nix"
  ];

  virtualisation.cyberus.intel-graphics-sriov.enable = true;
  virtualisation.virtualbox.host = {
    enable = true;
    enableKvm = true;
    enableHardening = false;
    addNetworkInterface = false;
  };
  # ...
}

Line 10 imports the file sriov.nix, containing the modules defining all the different aspects of the SR-IOV graphics acceleration feature. A simple enable = true; turns it on.

Similarly, NixOS already provides a module for VirtualBox. We extended this to offer our KVM backend as an alternative to the standard Oracle hypervisor backend. The KVM version also features the necessary graphics adapter needed for SR-IOV. Again, simply setting enable = true; makes the system ready to run VirtualBox VMs. The other parameters select the KVM backend and disable features that are currently not supported, or in the case of hardening not needed, by that variant.

Safely experimenting with NixOS specialisations

The limitations of our KVM backend and the necessary parameter tweaks in the VirtualBox host module are a great motivator to keep this environment separate. You might have workloads that depend on those other features. Similarly, the Linux kernel with SR-IOV graphics virtualization support might not work for all your applications. Ideally, the SR-IOV world should just be a dedicated boot option. If we select this option during boot, all SR-IOV settings and components should be applied and ready to go. If we select the default boot option, the system should behave as if SR-IOV was never set up.

Fortunately, NixOS provides another mechanism that solves this task in an incredibly elegant way: specialisations. Essentially, you can add a section of settings to your global NixOS configuration file that defines options that differ from the standard configuration. NixOS will encapsulate these into a completely separate environment that can be selected when needed. This selection can be done at runtime (for simple things like different application versions in the default path) or at boot time (for lower level things like the Linux kernel).

A separate boot entry for our SR-IOV environment therefore needs only one simple change to the configuration snippet above:

let
  sriov-modules = builtins.fetchGit {
    url = "https://github.com/cyberus-technology/nixos-sriov";
    ref = "main";
  };
in
{
  # ...
  imports = [
    "${sriov-modules}/sriov.nix"
  ];

  # Wrap the SR-IOV settings into a specialisation called "vbox-kvm-sriov"
  specialisation.vbox-kvm-sriov.configuration = {
    virtualisation.cyberus.intel-graphics-sriov.enable = true;
    virtualisation.virtualbox.host = {
      enable = true;
      enableKvm = true;
      enableHardening = false;
      addNetworkInterface = false;
    };
  };
  # ...
}

With this configuration, the SR-IOV flavor of the system will be built and ready to be selected. But unless you actually boot into it, your system is untouched. Once booted, you can proceed straight to the VM setup section of our tutorial, after verifying that everything works correctly.

Summary

Using NixOS, we turned the daunting task of experimenting with SR-IOV graphics acceleration into a piece of cake. Simply add some NixOS configuration, try it out, and switch back when you’re done. Everything is fully reproducible and non-destructive. Adaptations to the system (such as updating the kernel version) can be done as well and only require a few extra lines of configuration.

The best part: NixOS modules and specialisations are not just great for individual use, but bear huge potential in testing and shipping products. These mechanisms can be found all over our engineering and provide an efficiency boost we definitely can’t afford to miss.

If you’re ready to simplify your own system configurations, we encourage you to dive in! Check out our example configurations on GitHub to get started. If you have questions or need assistance, don’t hesitate to reach out to our team — we’re here to help you every step of the way.