New release of sbuild-qemu Utilities

I just released a new version of the sbuild-qemu utilities as part of sbuild. Notable changes are support for new architectures, and a new sbuild-qemu-boot utility to start a VM and interact with its console directly through the terminal.

As a reminder, these utilities leverage functionality in vmdb2, autopkgtest, and sbuild to use QEMU VM images for building and testing packages in strongly isolated environments and/or on foreign architectures and/or that might break the system. For example, the following would create an image for sid on the ppc64el architecture, and then use that image to build a package:

# Creating an image still requires root, unfortunately
$ sudo sbuild-qemu-create --arch ppc64el -o simple.img unstable http://deb.debian.org/debian
$ sudo chown <user>:<group> simple.img

# Building a package doesn't
$ sbuild-qemu --arch ppc64el --image simple.img -d unstable [<other sbuild options>] FOO.dsc

# Neither does updating the image (runs apt-get update && dist-upgrade in the VM)
$ sbuild-qemu-update --arch ppc64el simple.img

Since the sbuild-qemu utilities build on top of the autopkgtest utilities, the created images can also be used for running autopkgtests:

$ autopkgtest <test params> -- qemu --dpkg-architecture ppc64el simple.img

sbuild-qemu-boot

This new release now ships with sbuild-qemu-boot which boots a VM directly to a console, where it can be used as a porterbox, or to evaluate changes that might break the system, etc. By default, the VM is started in read-only mode, so changes are not saved when the VM is shut down. This can be changed with the --read-write option. With the --shared-dir option, a directory can be shared with the host. It will be mounted at /shared within the VM.

$ sbuild-qemu-boot --arch ppc64el --shared-dir /path/to/folder simple.img

Note that the console in the VM is 24x80 by default. This can be changed within the VM with stty to match your host, e.g.:

$ stty rows M cols N

I'm still pondering whether to automate this in sbuild-qemu-create.

New Architectures

Thanks to updates to vmdb2 and autopkgtest, the sbuild-qemu utilities now also support new architectures. Currently, amd64, i386, arm64, armhf, and ppc64el are supported, so most of the official architectures.

However, there's an autopkgtest issue (#10031002) with the host-guest communication bridge on armhf and on multi-core arm64 (single-core seems to work fine) that currently makes images unusable for sbuild and autopkgtest. I haven't yet figured out what the problem seems to be, but for the time being, the images can still be used with sbuild-qemu-boot.

qemu-sbuild-utils merged into sbuild

qemu-sbuild-utils have been merged into sbuild and are now shipped as package sbuild-qemu. The executables have been renamed from qemu-sbuild-* to sbuild-qemu-*, to be consisent with the other utilities provided by sbuild.

I may or may not have botched the transitional dummy package, but as the original package never migrated to testing (this was deliberate) and popcon was low, I'm confident that people will manage.

sbuild-qemu depends on the recently uploaded vdbm2, which added support for arm64, armhf, and ppc64el images. This is really exciting, as this means that sbuild-qemu and autopkgtest will soon be able to build for and test on most of the officially supported architectures, all from one host machine. MRs to enable these new features in autopkgtest have already been filed by Ryutaro Matsumoto. Support for the armel architecture is being discussed; support for the MIPS architectures is a more complicated issue, as they don't use GRUB.

I'd like to thank Johannes Schauer for reaching out, initiating discussion, and collaborating on this merge!

Keeping your Workstation Silent

I require moderately powerful boxes for my work. Moderately powerful boxes generate a non-negligible amount of heat, and therefore require appropriate cooling. Most cooling solutions involve a certain degree of noise, which I dislike.

Water-cooling is supposed to be the most effective solution for noise reduction, but it always felt like too much of a hassle to try to me.

Considering the various solutions I've tried in the past years, two observations I've made really stick out:

  1. Noctua's reputation for CPU cooling is well-earned.

  2. A "silent" (meaning sound-proofed) case, isn't. At least, not anymore.

I've tried numerous coolers in the past, some of monstrous proportions (always thinking that more mass must be better, and reputable brands are equally good), but I was never really satisfied; hence, I was doubtful that trying yet another cooler would make a difference. I'm glad I tried the Noctua NH-D15 anyway. With some tweaking to the fan profile in the BIOS, it's totally inaudible at normal to medium workloads, and just a very gentle hum at full load—subtle enough to disappear in the background.

For the past decade, I've also regularly purchased sound-proofed cases, but this habit appears anachronistic now. Years ago, sound-proofed cases helped contain the noise of a few HDDs. However, all of my boxes now contain NVMe drives (which, to me, are the biggest improvement to computing since CPUs going multi-core).

On the other hand, some of my boxes now contain powerful GPUs used for GPGPU computing, and with the recent higher-end Nvidia and AMD cards all pulling in over 300W, there is a lot of heat to manage. The best way to quickly dump heat is with good airflow. Sound-proofing works against that. Its insulation restricts airflow, which ultimately causes even more noise, as the GPU's fans need to spin at very high RPMs. This is, of course, totally obvious in hindsight.

I removed the front cover from a new GPU box, and the noise went down from almost-hair-dryer to acceptable-whoosh levels. I'm wondering whether a completely open box might not be even more effective.

qemu-sbuild-utils 0.1: sbuild with QEMU

qemu-sbuild-utils, which were recently accepted into unstable, are a collection of scripts that wrap standard features of sbuild, autopkgtest-build-qemu and vmdb2 to provide a trivial interface for creating and using QEMU-based environments for package building:

  • qemu-sbuild-create creates VM images

  • qemu-sbuild-update is to VM images what sbuild-update is to chroots

  • qemu-sbuild wraps sbuild, adding all necessary (and some useful) options for autopkgtest mode.

Here's a simple two-line example for creating an image, and then using it to build a package:

$ sudo qemu-sbuild-create -o simple.img unstable http://deb.debian.org/debian

$ qemu-sbuild --image simple.img -d unstable [sbuild-options] FOO.dsc

That's it.

Both qemu-sbuild-create and qemu-sbuild automate certain things, but also accept a number of options. For example, qemu-sbuild-create can install additional packages either from one of the APT sources, or .deb packages from the local filesystem.

qemu-sbuild will pass on every option it does not consume itself to sbuild, so it should mostly work as a drop-in replacement for it (see the Limitations section below for where it doesn't).

The created images can also be used for running autopkgtest itself, of course.

Advantages

Excellent isolation. One can go nuts in an environment, change or even break things, and the VM can always simply be reset, or rolled back to an earlier state. Snapshots are just terrific for so many reasons.

With KVM acceleration and a fast local APT cache, builds are really fast. There's an overhead of a few seconds for booting the VM on my end, but that overhead is negligible in comparison to the overall build time. On the upside, with everything being memory-backed, even massive dependency installations are lightning fast.

With the parameters of the target environment being configurable, it's possible to test builds in various settings (for example: single-core vs. multi-core, or with memory constraints).

Technically, it should be possible to emulate, on one host, any other guest architecture (even if emulation might be slow because of missing hardware acceleration). This would present an attractive alternative to (possibly distant and/or slow) porter boxes. However, support for that in qemu-sbuild-utils is not quite there yet.

Limitations

The utilities are currently only available on the amd64 architecture, for building packages in amd64 and i386 VMs. There are plans to support arm64 in the near future.

qemu-sbuild-create needs root, for the debootstrap stage. I'm looking into ways around this (by extending vmdb2). In any case, image updating and package building do not need privileges.

autopkgtest mode does not yet support interactivity, so one cannot drop into a shell with --build-failed-commands, for example. The easy workaround is to connect to the VM with SSH. For this, the image must contain the openssh-server package.

Alternatives

I looked at qemubuilder, but had troubles getting it to work. In any case, the autopkgtest chroot mode of sbuild seemed far more powerful and useful to me.

vectis looks incredibly promising, but I had already written qemu-sbuild-utils by the time I stumbled over it, and as my current setup works well for me for now and is simple enough to maintain, I decided to polish and publish it.

I'm also looking into Docker- and other namespace-based isolation solutions (of which there are many), which I think are the way forward for the majority of packages (those that aren't too close to the kernel and/or hardware).

Rather than relying on the kernel for isolation, KVM-based solutions like Amazon's Firecracker and QEMU's microvm machine type provide minimal VMs with almost no boot overhead. Firecracker, for example, claims less than 125ms from launch to /sbin/init. Investigating these is a medium-term project for me.

Why not schroot?

I have a strong aversion towards chroot-based build environments. The concept is archaic. Far superior software- and/or hardware-based technologies for process isolation have emerged in the past two decades, and I think it is high time to leave chroot-based solutions behind.

Acknowledgments

These utilities are just high-level abstractions. All the heavy lifting is done by sbuild, autopkgtest, and vmdb2.

RStudio is a refreshingly intuitive IDE

I currently need to dabble with R for a smallish thing. I have previously dabbled with R only once, for an afternoon, and that was about a decade ago, so I had no prior experience to speak of regarding the language and its surrounding ecosystem.

Somebody recommended that I try out RStudio, a popular IDE for R. I was happy to see that an open-source community edition exists, in the form of a .deb package no less, so I installed it and gave it a try.

It's remarkable how intuitive this IDE is. My first guess at doing something has so far been correct every. single. time. I didn't have to open the help, or search the web, for any solutions, either -- they just seem to offer themselves up.

And it's not just my inputs; it's the output, too. The RStudio window has multiple tiles, and each tile has multiple tabs. I found this quite confusing and intimidating on first impression, but once I started doing some work, I was surprised to see that whenever I did something that produced output in one or more of the tabs, it was (again) always in an intuitive manner. There's a fine line between informing with relevant context and distracting with irrelevant context, but RStudio seems to have placed itself on the right side of it.

This, and many other features that pop up here and there, like the live-rendering of LaTeX equations, contributed to what has to be one of the most positive experiences with an IDE that I've had so far.

My new favorite utility: autojump

Like any developer, I have amassed an impressive collection of directory trees both broad and deep. Navigating these trees became increasingly cumbersome, and setting CDPATH, using auto-completion, and working with the readline history search alleviated this only somewhat.

Enter autojump, from the package of the same name.

Whatever magic it uses is unbelievably effective. I estimate that in at least 95% of my cases, typing j <name-fragment> changes to the directory I was actually thinking of.

Say I'm working on package scikit-learn. My clone of the Salsa repo is in ~/code/pkg-scikit-learn/scikit-learn. Changing to that directory is trivial, I only need to specify a name fragment:

$ j sci
/home/christian/code/pkg-scikit-learn/scikit-learn
christian@workstation:~/code/pkg-scikit-learn/scikit-learn

But what if I want to work on scikit-learn upstream, to prepare a patch, for example? That repo has been cloned to ~/code/github/scikit-learn. No problem at all, just add another name fragment:

$ j gi sci
/home/christian/code/github/scikit-learn
christian@workstation:~/code/github/scikit-learn

The magic, however, is most evident with directory trees I rarely enter. As in: I have a good idea of the directory name I wish to change to, but I don't really recall its exact name, nor where (in the tree) it is located. I used to rely on autocomplete to somehow get there which can involve hitting the [TAB] key far too many times, and falling back to find in the worst case, but now, autojump always seems gets me there on first try.

I can't believe that this has been available in Debian for 10 years and I only discovered it now.

Curved Monitor

It's been two weeks since I purchased my first curved monitor. Switching away from a flat panel proved to be a novel and unusual experience — so much, in fact, that within the first five minutes, I already wanted to return it. Nevertheless, I gave it a try, and I'm glad I did, because not only did I eventually get over the initially perceived issues, I'm now extremely satisfied with it.

Shifted Perspective

My sole motivation for the switch was that I had become irritated (to a probably irrational degree) by reading and writing text in whatever window tile was on the left side of my desktop. Even though my previous monitor wasn't a particularly large one with 24", the shift in perspective on the far side of that window always made me feel as if I were reading something to the side of me, rather than in front of me — even if I turned to face it directly.

It was time to try out a curved monitor.

Process

Purchasing something like a monitor is always a pain; there's just so much choice. I would have preferred something with an IPS panel, 4K resolution, and either a 27" or 32" size, and would compromise for a VA panel and WQHD resolution. On geizhals.at, an Austrian price comparison site, ~50 monitors satisfied those criteria. Further limiting the list to reputable brands and reasonable prices still left me with more than two dozen options.

Without going into the details why (I was just glad to be done with it), I eventually settled for an MSI Optix MAG271CQR, a 27" WQHD monitor with a VA panel.

Once the new monitor arrived, I removed the old monitor from my VESA desk mount, installed the new one, booted, and gave it a try.

Within the first five minutes of use, I made three key observations:

  1. My shifted perspective issue on the sides was solved (great!), and

  2. I had gained quite a bit of screen real estate (great!), but …

  3. Because of the curvature, the bottom task bar now looked bent (Oh Noes).

Now, point (3) might not sound like that big of an issue, but when you're willing to change your monitor just because vim looks kind of weird to you when it's window is on the left side of the desktop, then a bent-looking task bar is a deal-breaker. I decided that I had to return it.

However, that meant: removing it, re-boxing it, shipping it back, etc. Tedious work. As it was already mounted and connected, a friend encouraged me to give it a day or two anyway, just in case.

That turned out to be great advice. I would never have expected this, but I got over the bent-looking task bar issue pretty fast. The pleasure of a corrected perspective on either side (everything just looks "right" now) more than makes up for the bent-looking tar bar at the bottom; I don't even notice it anymore. And the added screen real estate is a bonus I hadn't planned for.

The MAG271CQR targets the gaming demographic, and thus comes loaded with various features. My new favorite is "Reader Mode", which has an effect quite similar to "Night Mode" on mobile devices (reduced brightness, blue light filter). My eyes barely tire anymore, even after a long day's use. It also has a Picture-in-Picture mode for a second input which I haven't tried yet, but should come in handy for SBCs and the like.

hello, world

It slipped my mind that the recent March 17, 2020, marked the 10th anniversary of my first upload to the official Debian archive. How time flies!

Although I was never much of a blogging person myself, I do enjoy reading other people's contributions, and on some occasions felt that the Planet Debian feed might have been a more appropriate medium for sharing a particular tidbit, instead of sharing it with a specific mailing list, or not sharing at all.

So, to commemorate my anniversary, I venture into this new space.

first upload: https://lists.debian.org/debian-devel-changes/2010/03/msg01807.html