The Value of Nix/NixOS in our Engineering

Why we use Nix at Cyberus

At Cyberus, Nix and NixOS are very important technologies for our engineering. Many of the advantages that Nix and NixOS bring over other package managers or distributions are a central part of our every-day engineering.

We like reproducible builds. Our CI pipeline is only used as a check for correctness of a certain change, and sometimes to benefit from better build performance, but it is no longer the single source of truth. There is no “but it builds on my machine” in our daily lives.

While CI caching can be enabled with other technologies as well, using Nix’ Binary Caching allows engineers to quickly obtain CI artifacts for local use. Reproducible builds make this possible in the first place.

New team members can develop on day one, because they don’t have to spend their time setting up required environments on their developer machine. The same can be accomplished by providing docker containers, or centralized build machines, but these come with certain disadvantages that Nix doesn’t have. In particular, builds with Nix are, again, reproducible, and developers don’t depend on a particular infrastructure environment for their daily work. We have seen other organizations where the only way to build software was via SSH access to a build machine on the other side of the globe. This is an unacceptable work environment for us.

Nix opens the door for an approach to dependency management that is incredibly hard to get right with other technologies. Using niv or Nix Flakes, one project can simply point to another one and consume their outputs, which, again, are usually cached. Gone are the days of adding a dependency as a git submodule and cautiously inserting it into the existing build process of your project.

Use Case: Declaratively defined system images

Developing hypervisors and other systems software implies that you sometimes need to manually test your software on certain hardware that your software doesn’t behave as expected on. While we’d like to automate all the things, sometimes it’s more reasonable to quickly manually test something than to spend days of automating a test case.

In these cases, it can be valuable to have an ISO image that contains exactly the software you need for your investigation. Using NixOS, it is easily possible to define a system, including, e.g, the linux kernel, and some services, from a short configuration file, and generate a NixOS image from that.

Use Case: Direnv + nix shell

While nix-build is what the CI runs when verifying the correctness of submitted code, that may not be what you want to do during development, since nix-build will always build your project from scratch (but not its dependencies).

The recommended way to enable incremental builds and fast feedback cycles is to enter a nix shell, and build your project manually from there. This can get cumbersome and error prone, e.g., if you forget to exit the nix shell when switching projects, or when you change dependencies of your nix shell.

Direnv has a great nix integration, with a .envrc containing only use nix or use flake , entering a directory on the shell automatically prepares your environment for the respective project. Depending on your IDE and the technologies you are working with you may not need any system or user-wide installed dependencies anymore.

At Cyberus, each of our projects provides a nix shell and a corresponding .envrc, which allows even quicker onboarding into new projects.

Use Case: Dogfooding

Cyberus’ software is actively used by our own employees in their daily work. This kind of dogfooding allows us to catch bugs or performance issues which aren’t yet covered by our test systems. We are therefore customers of our own products.

While it is possible to use other Linux distributions (or even macOS) and still enjoy many of the benifts of Nix, most members of our team use NixOS as their daily driver. NixOS’ modules provide a handy way to enable dogfooding with a single setting such as cyberus.dogfood.enable = true. Enabling this setting installs Cyberus software on your computer. You may use the most recent version, a development branch, or go back in time to analyze an older state. The simplicity of this approach leaves no excuses for not participating in our dogfood program.

Use Case: Configuration Sharing

In addition to dogfooding, our employees share other NixOS configuration settings with custom modules. Do you need to configure the company VPN, install printer drivers or want to use the company fonts? Our employees frequently share their configurations and turn them into official Cyberus NixOS modules when they see a common need.

Are you a vim/emacs/… power user? Chances are high that others share your enthusiasm. NixOS configurations empower you to share your settings with other team members so that you can learn from the experts or onboard and guide less experienced users.

Use Case: Reproducible Benchmarks

If you want to assess the end user’s performance of your software, there are hundreds of tools to choose from. Given that you found one that matches your use case, you can run that benchmark manually to compare different versions of your software, e.g., to analyze the effects of a new performance optimization. When you start the next improvement a couple of weeks later, chances are high that you don’t recall all your benchmark’s configuration options and run the new experiments with completely different settings. Wouldn’t it be nice to have a reliable, reproducible way to compare the different software versions?

At Cyberus, we have a benchmark system that runs performance evaluations on a nightly base. This benchmark system is based on Nix. Because we package our software using Nix, we can root cause performance degradation or improvements that come from upgrading dependencies.

We also automate benchmark invocations with the help of Nix so that there are no manual interactions in our experiments. These invocations may accumulate to a complex series of steps:

1. Build a Linux image with all software installed
2. Boot the image on a test machine
3. Bring up a virtual machine
4. Run a benchmark in the virtual machine
5. Transfer benchmark scores to an evaluation system
6. Power off the test machine

We’re here to help!

If you are seeking a career option with Nix/NixOS, want to learn more about our use cases or face a transition to Nix/NixOS and need support: Feel free to reach out via our contact form to get in touch!